RU2785950C2 - Maintenance of contextual information between user interactions with voice assistant - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the field of computer technology for control of user equipment, using speech input. Contextual information related to interaction between a user and user equipment is received; speech input is received from the user through a conversational interface; speech input is processed for determination of a user’s intention associated with speech input, while the user’s intention is associated with a task parameter; and, upon determining that the user’s intention associated with speech input is related to activation of a software application installed in user equipment, the task parameter is changed based on the software application, the software application is activated in user equipment in an external way relatively to a graphic user interface including the conversational interface, while the software application is activated, using the changed task parameter, and a response is issued based on the user’s intention and contextual information.

EFFECT: increase in the accuracy of control of user equipment, using speech input.

24 cl, 5 tbl, 49 dwg

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application Serial Number 61/295774 entitled "Intelligent Automated Assistant", Attorney's Statement Number SIRIP003P, filed January 18, 2010, which is incorporated herein by reference.

[0002] This application further claims the priority of U.S. Patent Application 12/987982 entitled "Intelligent Automated Assistant", Attorney's Statement Number P10575US1, filed January 10, 2011, which is incorporated herein by reference.

[0003] This application is further related to U.S. Patent Application Serial Number 11/518292 entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006, which is incorporated herein by reference.

[0004] This application is further related to U.S. Provisional Application Serial Number 61/186414 entitled "System and Method for Semantic Auto-Completion", filed June 12, 2009, which is incorporated herein by reference.

The technical field to which the invention belongs

[0005] The present invention relates to intelligent systems, and more specifically, to classes of applications for intelligent automated assistants.

PRIOR ART

[0006] Today's electronic devices have the ability to access a significant, growing and diverse number of functions, services (services) and information, both via the Internet and from other sources. Functionality for such devices is expanding rapidly as many consumer devices, smartphones, tablet computers, etc. have the ability to run software applications to perform various tasks and provide various types of information. Often, each application, feature, website, or feature has its own user interface and its own practical paradigms, many of which can be difficult to learn or difficult for users to understand. In addition, many users may experience difficulty even discovering what functionality and/or information is available on their electronic devices or on various web sites; thus, these users may be frustrated or confused, or may simply be unable to make efficient use of the resources available to them.

[0007] In particular, inexperienced users or people who have any physical impairment or disability, and/or are elderly, fussy, inattentive, and/or drive a vehicle, may have difficulty interacting effectively with their electronic devices and /or effective use of online services. Such users in particular are likely to be hampered by the large number of diverse and inconsistent features, applications and websites that may be available for their use.

[0008] Accordingly, existing systems are often complex to use and navigate, and often present users with inconsistent and difficult to understand interfaces that often interfere with users' effective use of the technology.

The essence of the invention

[0009] According to various embodiments of the present invention, an intelligent automated assistant is implemented on an electronic device to facilitate user interaction with the device and to help the user make better use of local and/or remote services. In various embodiments, the intelligent automated assistant interacts with the user in a uniform conversational manner using natural language dialogue and activates external services as needed to obtain information or perform various actions.

[0010] According to various embodiments of the present invention, an intelligent automated assistant integrates a variety of features provided by various software components (e.g., to support natural language recognition and natural language dialogue, multimodal input, personal information management, task flow management, distributed service orchestration, and etc.). In addition, in order to offer intelligent interfaces and useful functionality to users, the intelligent automated assistant of the present invention, in at least some embodiments, can coordinate these components and services. The conversational interface and the ability to receive information and perform a further task are implemented, in at least some embodiments, by coordinating various components, such as language components, dialog components, task control components, information control components, and/or a plurality of external services.

[0011] According to various embodiments of the present invention, intelligent automated assistant systems can be configured to provide various special types of operations, functionality and/or features and/or combine multiple features, operations and applications of the electronic device on which they are installed. In some embodiments, systems based on the intelligent automated assistant of the present invention may perform any or all of the following: actively extract input from the user, interpret the user's intent, resolve ambiguities between competing interpretations, request and receive clarification information when necessary, and perform (or initiation) of actions based on discernible intent. Actions may be performed, for example, by activating and/or interacting with any applications or services that may be available on the electronic device, as well as services that are available over an electronic network such as the Internet. In various embodiments, such activation of external services may be done through an API or through any other suitable mechanism. Thus, systems based on the intelligent automated assistant of various embodiments of the present invention can unify, simplify, and improve user interaction with many different applications and functions of an electronic device, and with respect to services that can be accessed over the Internet. The user can thereby be relieved of the burden of learning what functionality may be available on the device and web-based services, how to interact with such services to get what they want, and how to interpret the output received from such services. services; on the contrary, the assistant according to the present invention can act as an intermediary between the user and such various services.

[0012] In addition, in various embodiments, the assistant of the present invention provides a conversational interface that a user may find more intuitive and less burdensome than traditional graphical user interfaces. The user may interact in the form of a conversational dialogue with the assistant using any number of available input and output mechanisms, such as, for example, speech, graphical user interfaces (buttons and links), text input, and the like. The system may be implemented using any number of different platforms such as device APIs, web technologies, email, and the like. or any combination of the above. Requests for additional input may be presented to the user in the context of such a conversation. Short-term storage and long-term storage can be applied so that user input can be interpreted in the proper context, taking into account previous events and communications in the session, as well as statistical and profile information about the user.

[0013] In addition, in various embodiments, contextual information derived from user interaction with a feature, operation, or application on a device may be used to facilitate other features, operations, or applications on the device or other devices. For example, an intelligent automated assistant may use the context of a telephone call (eg, a call with a person) to facilitate the initiation of a text message (eg, to determine that the text message should be sent to the same person without the user having to explicitly specify the recipient of the text message). The intelligent automated assistant of the present invention can thus interpret instructions, for example, "send him a text message", with "him" being interpreted according to contextual information derived from the current phone call and/or from any feature, operation or application on the device. In various embodiments, the intelligent automated assistant takes into account the various types of available context data to determine which address book contact to use, which contact information to use, which phone number to use for the contact, and the like, so that the user does not have to re-enter this information manually.

[0014] In various embodiments, the assistant may also take external events into account and respond appropriately, for example, to initiate an action, initiate communication with a user, provide alerts, and/or modify a previously initiated action based on external events. If user input is required, the conversational interface can again be used.

[0015] In one embodiment, the system is based on sets of interrelated subject areas and tasks and uses additional functionality supported through external services with which the system can interact. In various embodiments, these external services include web-based services as well as functionality associated with the hardware device itself. For example, in an embodiment in which the intelligent automated assistant is implemented on a smartphone, personal digital assistant, tablet computer, or other device, the assistant can control many device operations and functions, such as dialing a phone number, sending a text message, setting reminders, adding events. to the calendar, etc.

[0016] In various embodiments, the system of the present invention may be implemented to provide assistance in any of a number of different subject areas. Examples include:

- local services (including location and time dependent services such as restaurants, movies, ATMs, events and meeting places);

- personal and social storage services (including scheduled tasks, notes, calendar events, shared links, etc.);

- e-commerce (including online purchases of goods such as books, DVDs, music, etc.);

- travel services (including flights, hotels, attractions, etc.).

[0017] Those skilled in the art will recognize that the above list of subject areas is merely exemplary. In addition, the system of the present invention can be implemented in any combination of subject areas.

[0018] In various embodiments, the intelligent automated assistant systems disclosed herein may be configured or designed to include functionality to automate the use of data and services available over the Internet to discover, find, select, purchase , reserve or order products and services. In addition to automating the process of using these data and services, at least one embodiment of the intelligent automated assistant system disclosed herein can also provide simultaneous combined use of multiple data sources and services. For example, it can combine product information from multiple review websites, check prices and availability from multiple distributors and check their locations and time limits, and help the user find a personalized solution to their problem. Additionally, at least one embodiment of the intelligent automated assistant system disclosed herein may be configured to include functionality to automate the use of data and services available over the Internet to discover, collect information, select, reserve and otherwise learn about activities (including, but not limited to, films, events, performances, exhibits, shows, and attractions); places to go (including, but not limited to, travel destinations, hotels and other places to stay, sights and other places of interest, etc.); places to eat and drink (such as restaurants and bars), times and places to meet other people, and any other sources of entertainment or social interaction that can be found on the Internet. Additionally, at least one embodiment of the intelligent automated assistant system disclosed herein may be configured to include functionality to enable applications and services to operate through natural language dialogue, which may otherwise be provided through dedicated applications with graphical user interfaces, including search (including location-based search); navigation (maps and route directions); search in the database (for example, finding organizations or people by name (name) or other properties); obtaining weather data and weather forecasts, checking the price of commercial items or the status of financial transactions; monitoring traffic or flight status; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; communication via email or other messaging platforms; and local or remote operation of devices (eg, dialing telephone numbers, controlling lighting and temperature, controlling home security devices, playing music or video, etc.). Additionally, at least one embodiment of the intelligent automated assistant system disclosed herein may be configured to include functionality to identify, generate, and/or provide personalized recommendations for activities, products, services, entertainment source, time management or any other type of recommendation service that benefits from interactive natural language dialogue and automated access to data and services.

[0019] In various embodiments, the intelligent automated assistant of the present invention can control many features and operations of an electronic device. For example, an intelligent automated assistant may call services that interact with functionality and applications on the device through APIs or other means to perform functions and operations that might otherwise be initiated using a traditional user interface on the device. Such functions and operations may include, for example, setting an alarm, making a phone call, sending a text message or mail message, adding a calendar event, and the like. Such functions and operations may be performed as additional functions in the context of a conversational dialogue between the user and the assistant. Such functions and operations may be specified by the user in the context of such a dialog, or they may be automatically performed based on the context of the dialog. Those skilled in the art will recognize that the assistant can thus be used as a control mechanism to initiate and control various operations on an electronic device, and can be used as an alternative to traditional mechanisms such as buttons or graphical user interfaces.

Brief description of the drawings

[0020] The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention according to the embodiments. Those skilled in the art will recognize that the specific embodiments illustrated in the drawings are merely exemplary and are not intended to limit the scope of the present invention.

[0021] FIG. 1 is a block diagram illustrating an example of one embodiment of an intelligent automated assistant system.

[0022] Figure 2 illustrates an example of an interaction between a user and an intelligent automated assistant according to at least one embodiment.

[0023] FIG. 3 is a block diagram illustrating a computing device suitable for implementing at least a portion of an intelligent automated assistant according to at least one embodiment.

[0024] FIG. 4 is a block diagram illustrating an architecture for implementing at least a portion of an intelligent automated assistant in an autonomous computing system, according to at least one embodiment.

[0025] FIG. 5 is a block diagram illustrating an architecture for implementing at least a portion of an intelligent automated assistant in a distributed computing network, according to at least one embodiment.

[0026] FIG. 6 is a block diagram showing a system architecture illustrating several different types of clients and modes of operation.

[0027] FIG. 7 is a block diagram illustrating a client and a server that communicate with each other to implement the present invention according to one embodiment.

[0028] FIG. 8 is a block diagram illustrating a fragment of an active ontology according to one embodiment.

[0029] FIG. 9 is a block diagram illustrating an example of an alternative implementation of an intelligent automated assistant system.

[0030] FIG. 10 is a flowchart illustrating a method of operation for active input extraction component(s) according to one embodiment.

[0031] FIG. 11 is a flowchart illustrating a method for actively retrieving typed input according to one embodiment.

[0032] FIGS. 12-21 are screen shots illustrating some portions of some procedures for actively retrieving typed input, according to one embodiment.

[0033] FIG. 22 is a flowchart illustrating a method for actively extracting input for voice or speech input, according to one embodiment.

[0034] FIG. 23 is a flowchart illustrating a method for actively retrieving input for GUI-based input according to one embodiment.

[0035] FIG. 24 is a flowchart illustrating a method for actively retrieving input at a dialog flow level, according to one embodiment.

[0036] FIG. 25 is a flowchart illustrating a method for actively monitoring relevant events according to one embodiment.

[0037] FIG. 26 is a flowchart illustrating a method for actively retrieving a multimodal input according to one embodiment.

[0038] FIG. 27 is a set of screenshots illustrating an example of various types of functions, operations, actions, and/or other features that can be provided by domain model and service orchestration component(s) according to one embodiment.

[0039] FIG. 28 is a flowchart illustrating an example of a method for natural language processing according to one embodiment.

[0040] FIG. 29 is a screenshot illustrating natural language processing according to one embodiment.

[0041] FIGS. 30 and 31 are screen shots illustrating an example of various types of functions, operations, actions, and/or other features that may be provided by the dialog flow processor component(s) according to one embodiment.

[0042] FIG. 32 is a flowchart illustrating a method of operation for the dialog flow processor component(s) according to one embodiment.

[0043] FIG. 33 is a flowchart illustrating an automatic call and response procedure according to one embodiment.

[0044] FIG. 34 is a flowchart illustrating an example task flow for a constrained selection task according to one embodiment.

[0045] FIGS. 35 and 36 are screen shots illustrating an example operation of a constrained selection task according to one embodiment.

[0046] FIG. 37 is a flowchart illustrating an example procedure for performing a service orchestration procedure according to one embodiment.

[0047] FIG. 38 is a flowchart illustrating an example of a service activation procedure according to one embodiment.

[0048] FIG. 39 is a flowchart illustrating an example of a multi-phase output procedure according to one embodiment.

[0049] FIGS. 40 and 41 are screen shots illustrating examples of output processing according to one embodiment.

[0050] FIG. 42 is a flowchart illustrating an example of multimodal output processing according to one embodiment.

[0051] FIGS. 43A and 43B are screen shots illustrating an example of using short-term personal storage component(s) to maintain conversational context when changing location, according to one embodiment.

[0052] FIGS. 44A-44C are screen shots illustrating an example of using the long-term personal storage component(s), according to one embodiment.

[0053] FIG. 45 illustrates an example of an abstract model for a constrained selection problem.

[0054] FIG. 46 illustrates an example of a dialog flow model to help guide the user through the search process.

[0055] FIG. 47 is a flowchart illustrating a restricted selection method according to one embodiment.

Detailed description of embodiments

[0056] Various technologies are now described in detail with respect to several exemplary embodiments, as illustrated in the accompanying drawings. In the following description, many specific details are set forth in order to provide a thorough understanding of one or more aspects and/or features described or referenced herein. However, those skilled in the art will appreciate that one or more of the aspects and/or features described or referenced herein may be practiced without some or all of these specific details. In other instances, known process steps and/or structures are not described in detail so as not to obscure certain aspects and/or features described or referenced herein.

[0057] One or more different inventions may be described in this application. Additionally, for one or more of the invention(s), many of the embodiments described herein may be described in this patent application and are provided by way of illustration only. The described embodiments are not intended to be limiting in any sense. One or more invention(s) may be broadly applicable to a variety of embodiments, as should be apparent from the disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the invention(s), and it should be understood that other embodiments may be used and that structural, logical, software, electrical, and other changes may be carried out without departing from the scope of one or more of the invention(s). Accordingly, those skilled in the art will recognize that one or more of the invention(s) may be practiced with various modifications and variations. Specific features of one or more invention(s) may be described in relation to one or more specific embodiments or drawings that are part of the present disclosure, and which show, by way of illustration, specific embodiments of one or more invention(s). However, it should be understood that such features are not limited to use in one or more specific embodiments or in the drawings with respect to which they are described. The present disclosure is neither a literal description of all embodiments of one or more invention(s), nor a list of features of one or more invention(s) that must be present in all embodiments.

[0058] The section headings provided in this patent application and the title of this patent application are provided for convenience only and should not be construed as any limitation on the disclosure.

[0059] Devices that communicate with each other do not need to communicate continuously with each other, unless expressly stated otherwise. In addition, devices that communicate with each other may communicate directly or indirectly through one or more intermediaries.

[0060] The description of an embodiment with multiple components communicating with each other does not imply that all such components are required. Rather, a variety of optional components are described to illustrate a wide variety of possible embodiments of one or more of the invention(s).

[0061] Additionally, although process steps, method steps, algorithms, and the like may be described in sequential order, such processes, methods, and algorithms may be configured to operate in alternate orders. In other words, any sequence or order of steps that may be described in this patent application does not by itself indicate that the steps must be performed in that order. The steps of the described processes may be performed in any practical order. Additionally, some steps may be performed concurrently, despite being described or implied to be non-simultaneous (eg, because one step is described after another step). Furthermore, the illustration of a process by means of a drawing does not imply that the illustrated process excludes other changes and modifications therein, does not imply that the illustrated process or any of its steps are indispensable to one or more of the invention(s), and does not imply, that the illustrated process is preferred.

[0062] When describing a single device or article, it should be obvious that several devices/articles (whether they interact or not) can be used instead of a single device/article. Similarly, if multiple devices or articles are described (whether they interact or not), it should be obvious that a single device/article can be used in place of multiple devices or articles.

[0063] The functionality and/or features of the device may alternatively be implemented by one or more other devices that are not explicitly described as having such functionality/features. Thus, other embodiments of one or more of the invention(s) need not include the device itself.

[0064] The technologies and mechanisms described or referenced herein are sometimes described in the singular for clarity. However, it should be noted that specific embodiments include multiple technology iterations or mechanism instantiations unless otherwise noted.

[0065] While described in the context of intelligent automated assistant technology, it can be understood that various aspects and technologies described herein (such as, for example, those associated with active ontologies) may also be deployed and/or applied in other technical fields. involving human and/or computerized interaction with software.

[0066] Other aspects related to AI-based technology (for example, which may be used by, provided by, and/or implemented in one or more of the AI-assisted system embodiments described herein) are disclosed in one or more of the following opposed materials:

- U.S. provisional patent application serial number 61/295774, entitled "Intelligent Automated Assistant", attorney's statement number SIRIP003P, filed January 18, 2010, the disclosure of which is contained herein by reference;

- US patent application serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006, the disclosure of which is incorporated herein by reference; and

- provisional patent application (US) serial number 61/186414, entitled "System and Method for Semantic Auto-Completion", filed on June 12, 2009, the disclosure of which is contained herein by reference.

Hardware architecture

[0067] In general, the intelligent automated assistant technologies disclosed herein may be implemented in hardware or in a combination of software and hardware. For example, they may be implemented in the kernel of an operating system, in a separate user process, in a library package tied to network applications, in a specially built machine, or in a network interface card. In a particular embodiment, the technologies disclosed herein may be implemented in software, such as an operating system or an application running on an operating system.

[0068] The software/hardware hybrid implementation(s) of at least part of the embodiment(s) of the intelligent automated assistant disclosed herein may be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in a storage device . These network devices may have multiple network interfaces, which may be configured to use various types of network communication protocols. The general architecture for some of these machines may be apparent from the descriptions disclosed in this document. According to particular embodiments, at least some of the features and/or functionality of various embodiments of the intelligent automated assistant disclosed herein may be implemented in one or more general purpose networked host machines such as an end user computer system, a computer, a network server or server system, a mobile computing device (such as a personal digital assistant, a mobile phone, a smartphone, a laptop computer, a tablet computer, etc.), a consumer electronic device, a music player, or any other suitable electronic device, a router, a switch, and etc. or any combination of the above. In at least some embodiments, at least some of the features and/or functionality of various embodiments of the intelligent automated assistant disclosed herein may be implemented in one or more virtualized computing environments (e.g., networked computing clouds, etc.). P.).

[0069] Referring now to FIG. 3, a block diagram is shown illustrating a computing device 60 suitable for implementing at least a portion of the features and/or functionality of an intelligent automated assistant disclosed herein. The computing device 60 may be, for example, an end user computer system, a network server or server system, a mobile computing device (e.g., a personal digital device, a mobile phone, a smartphone, a laptop computer, a tablet computer, etc.), a consumer electronic device, music player or any other suitable electronic device, or any combination or part thereof. Computing device 60 may be configured to communicate with other computing devices, such as clients and/or servers, over a communication network, such as the Internet, using known protocols for this communication (wireless or wired).

[0070] In one embodiment, computing device 60 includes a central processing unit (CPU) 62, interfaces 68, and a bus 67 (eg, a Peripheral Component Interconnect (PCI) bus). When running the appropriate software or firmware, the CPU 62 may be responsible for implementing specific functions associated with those of a specially configured computing device or machine. For example, in at least one embodiment, a user's personal digital assistant (PDA) may be configured to act as an intelligent automated assistant system using a CPU 62, storage device 61, 65, and interface(s) 68. in at least one embodiment, the CPU 62 may be instructed to perform one or more of various types of intelligent automated assistant functions and/or operations under the control of software modules/components, which, for example, may include an operating system and any appropriate application software. , drivers, etc.

[0071] CPU 62 may include one or more processors 63, such as, for example, a processor from the Motorola or Intel microprocessor family or the MIPS microprocessor family. In some embodiments, processor(s) 63 may include custom designed hardware (e.g., application specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field programmable gate arrays (FPGAs), and the like. ) to control the operation of the computing device 60. In a specific embodiment, the storage device 61 (for example, non-volatile random access memory (RAM) and/or read only memory (ROM)) is also part of the CPU 62. However, there are many different the ways in which the storage device can be connected to the system. The storage unit 61 may be used for a variety of purposes such as caching and/or storing data, programming instructions, and the like, for example.

[0072] As used herein, the term "processor" is not limited to simply these integrated circuits referred to in the art as a processor, but broadly means a microcontroller, microcomputer, programmable logic controller, ASIC, and any other programmable circuit. .

[0073] In one embodiment, the interfaces 68 are provided as interface cards (sometimes referred to as "line cards"). In general, they control the sending and receiving of data packets over a computer network and sometimes support other peripherals used with computing device 60. Of the interfaces that may be provided, Ethernet interfaces, frame relay protocol interfaces, cable interfaces, DSL interfaces are provided. , token ring protocol interfaces, etc. In addition, various types of interfaces can be provided, such as, for example, Universal Serial Bus (USB), Serial, Ethernet, FireWire, PCI, Parallel, Radio Frequency (RF), Bluetooth™, Near Field Communication (e.g. using magnetism Near Field), 802.11 (Wi-Fi), Frame Relay, TCP/IP, ISDN, Fast Ethernet interfaces, Gigabit Ethernet interfaces, ATM based interfaces, High Speed Serial Interfaces (HSSI), point of sale terminal interfaces (POS), distributed data transmission interfaces over fiber optic channels (FDDI), etc. In general, such interfaces 68 may include ports suitable for communication with appropriate media. In some cases, they may also include an independent processor and, in some cases, volatile and/or non-volatile storage (eg, RAM).

[0074] Although the system shown in FIG. 3 illustrates one particular architecture for computing device 60 for implementing the technologies of the invention described herein, it is by no means the only device architecture in which at least a portion of features and technologies described in this document. For example, architectures having one or any number of processors 63 may be used, and such processors 63 may be present in one device or distributed among any number of devices. In one embodiment, one processor 63 handles communications as well as routing calculations. In various embodiments, different types of intelligent automated assistant features and/or functionalities may be implemented in an intelligent automated assistant based system that includes a client device (e.g., a personal digital assistant or smartphone running client software) and a server system. (s) (for example, a server system, described in more detail below).

[0075] Regardless of the configuration of the network device, the system of the present invention may use one or more storage devices or storage modules (such as, for example, storage unit 65) configured to store data, program instructions for operating a general purpose network, and /or other information related to the functionality of the intelligent automated assistant technologies described in this document. Program instructions may control, for example, the operation of an operating system and/or one or more applications. The storage device or storage devices may also be configured to store data structures, keyword taxonomy information, alert information, user click and tap information, and/or other specific non-program information described herein.

[0076] Because this information and program instructions may be used to implement the systems/methods described herein, at least some embodiments of a network device may include non-volatile computer-readable storage media, which, for example, may be configured to store program instructions, state information, and the like. to perform the various operations described in this document. Examples of such non-volatile computer-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs; magneto-optical media such as floppy disks; and hardware devices that are specifically configured to store and execute program instructions, such as read-only memory (ROM), flash memory, memristor storage, random access memory (RAM), etc. P. Examples of program instructions include both machine code, such as generated by a compiler, and files containing top-level code that can be executed by a computer using an interpreter.

[0077] In one embodiment, the system of the present invention is implemented on a standalone computing system. Referring now to FIG. 4, there is shown a block diagram illustrating an architecture for implementing at least a portion of an intelligent automated assistant in an autonomous computing system, according to at least one embodiment. Computing device 60 includes processor(s) 63 that execute software to implement intelligent automated assistant 1002. Input device 1206 may be of any type suitable for receiving user input, including, for example, a keyboard, touch screen, microphone ( for example, for voice input), mouse, touch pad, trackball, five-button switch, joystick, and/or any combination of the foregoing. Output device 1207 may be a screen, speaker, printer, and/or any combination of the above. The memory device 1210 may be random access memory having a structure and architecture known in the art for use by the processor(s) 63 during software execution. Storage device 1208 may be any magnetic, optical, and/or electrical storage device for storing data in digital form; examples include flash memory, magnetic hard disk, CD-ROM, and the like.

[0078] In another embodiment, the system of the present invention is implemented in a distributed computer network, such as a network having any number of clients and/or servers. Referring now to FIG. 5, there is shown a block diagram illustrating an architecture for implementing at least a portion of an intelligent automated assistant in a distributed computing network, according to at least one embodiment.

[0079] In the arrangement shown in Fig. 5, any number of clients 1304 are provided; each client 1304 may run software to implement the client portions of the present invention. In addition, any number of servers 1340 may be provided to process requests received from clients 1304. Clients 1304 and servers 1340 may communicate with each other over an electronic network 1361, such as the Internet. Network 1361 may be implemented using any known network protocols, including, for example, wired and/or wireless protocols.

[0080] In addition, in one embodiment, servers 1340 may call external services 1360 as needed to obtain additional information or store data regarding previous interactions with specific users. Communication with external services 1360 may be via network 1361, for example. In various embodiments, external services 1360 include web-based services and/or functionality associated with or installed on the hardware device itself. For example, in an embodiment where assistant 1002 is implemented on a smartphone or other electronic device, assistant 1002 can obtain information stored in a calendar application ("app"), contacts, and/or other sources.

[0081] In various embodiments, assistant 1002 can control many of the features and operations of the electronic device on which it is installed. For example, assistant 1002 may call external services 1360 that interact with functionality and applications on the device via an API or other means to perform functions and operations that might otherwise be initiated using a traditional user interface on the device. Such functions and operations may include, for example, setting an alarm, making a phone call, sending a text message or mail message, adding a calendar event, and the like. Such functions and operations may be performed as additional functions in the context of a conversational dialogue between the user and assistant 1002. Such functions and operations may be specified by the user in the context of such a dialogue, or they may be automatically performed based on the context of the dialogue. Those skilled in the art will recognize that assistant 1002 can thus be used as a control mechanism for initiating and controlling various operations on an electronic device, which can be used as an alternative to traditional mechanisms such as buttons or graphical user interfaces.

[0082] For example, the user may provide input to assistant 1002, such as "I need to wake up at 8:00 tomorrow." After assistant 1002 determines the user's intent using the technologies described herein, assistant 1002 can call external services 1340 to interact with an alarm clock feature or application on the device. Helper 1002 sets an alarm on behalf of the user. Thus, the user can use assistant 1002 as a replacement for traditional mechanisms for setting an alarm or performing other functions on the device. If user requests are ambiguous or require further clarification, helper 1002 may use various techniques described herein, including active retrieval, paraphrasing, recommendations, and the like, to obtain the necessary information so that the correct services 1340 are invoked, and the intended action is taken. In one embodiment, assistant 1002 may prompt the user for confirmation prior to calling service 1340 to perform a function. In one embodiment, the user may selectively deactivate the ability of assistant 1002 to invoke specific services 1340, or may deactivate all such invocation of services as needed.

[0083] The system of the present invention can be implemented for many different types of clients 1304 and modes of operation. 6 is a block diagram showing the system architecture illustrating several different types of clients and modes of operation. Those skilled in the art will recognize that the various types of clients 1304 and modes of operation shown in FIG. 6 are merely exemplary, and that the system of the present invention may be implemented using clients 1304 and/or modes of operation other than those illustrated. Additionally, the system may include any or all of such clients 1304 and/or modes of operation, one at a time or in any combination. Illustrated examples include:

- Computer devices with input/output devices and/or sensors 1402. The client component may be deployed on any such computing device 1402. At least one embodiment may be implemented using a web browser 1304A or other application to enable communication with servers 1340 via network 1361. The input and output channels may be of any type, including, for example, visual and/or acoustic channels. For example, in one embodiment, the system of the invention may be implemented using voice communication techniques providing an embodiment of an assistant for the blind in which the equivalent of a web browser is speech-driven and uses speech for output.

- Mobile devices with I / O modules and sensors 1406, for which the client can be implemented as an application on a mobile device 1304B. These include, but are not limited to, mobile phones, smart phones, personal digital devices, tablet devices, network game consoles, and the like.

- Home appliances with I/O modules and 1410 sensors, for which the client can be implemented as an embedded application in the 1304C appliance.

- Cars and other vehicles with dashboard interfaces and sensors 1414, for which the client can be implemented as an embedded system application 1304D. These include, but are not limited to, car navigation systems, voice control systems, in-car electronic multimedia systems, and the like.

- Networked computing devices, such as routers 1418 or any other device that resides on or interacts with the network, for which the client can be implemented as a resident application 1304E.

- Mail clients 1424, with which the implementation of the assistant connects through the server 1426 modality by e-mail. The email modality server 1426 acts as a communication bridge, for example, accepting input from the user as emails sent to the helper and sending the output from the helper to the user as responses.

- Clients 1428 instant messaging, with which the embodiment of the assistant connects through the server 1430 modality protocol messaging. The messaging protocol modality server 1430 acts as a communication bridge, accepting input from the user as messages sent to the helper, and sending output from the helper to the user as messages in response.

- Voice telephones 1432 to which the assistant's embodiment connects via a voice-over-IP (VoIP) modality server 1430. The VoIP modality server 1430 acts as a communication bridge, receiving input from the user as a voice spoken to the assistant, and sending output from the assistant to the user, such as synthesized speech, in response.

[0084] For messaging platforms, including but not limited to email, instant messaging, discussion forums, group chat sessions, online help or technical support sessions, and the like, assistant 1002 may act as a participant in conversations . Assistant 1002 may track the conversation and response of people or a group using one or more of the technologies and methods described herein for one-to-one interactions.

[0085] In various embodiments, the functionality for implementing the technologies of the present invention may be distributed across any number of client and/or server components. For example, different software modules may be implemented to perform different functions in connection with the present invention, and such modules may be implemented differently to run on server and/or client components. Referring now to FIG. 7, an example of a client 1304 and a server 1340 are shown that communicate with each other to implement the present invention according to one embodiment. 7 illustrates one possible arrangement by which program modules may be distributed between client 1304 and server 1340. Those skilled in the art should recognize that the illustrated arrangement is merely exemplary, and that such modules may be distributed in many different ways. In addition, any number of clients 1304 and/or servers 1340 may be provided, and modules may be distributed among these clients 1304 and/or servers 1340 in any of a number of different ways.

[0086] In the example of FIG. 7, the input extraction functionality and the output processing functionality are distributed between the client 1304 and the server 1340, with the input extraction client side 1094a and the output processing client side 1092a located in the client 1304, and the input extraction server side 1094b and the output processing backend 1092b is located on the server 1340. The following components are located on the server 1340:

- complete dictionary 1058b;

- a complete library of modules 1060b recognition of language constructs;

- main (master) version 1052b short-term personal storage device;

- master version 1054b of the long-term personal storage device.

[0087] In one embodiment, client 1304 stores subsets and/or portions of these components locally to increase responsiveness and reduce reliance on network communication. Such subsets and/or portions may be maintained and updated according to known cache management techniques. Such subsets and/or parts include, for example:

- vocabulary subset 1058a;

- a subset of the library of modules 1060a recognition of language constructs;

- cache 1052a short-term personal storage device;

- cache 1054a long-term personal storage device.

[0088] Additional components may be implemented as part of the server 1340, including, for example:

- language interpreter 1070;

- processor 1080 flow of dialogues;

- output processor 1090;

- database 1072 data objects of the subject area;

- task flow models 1086;

- orchestration of 1082 services;

- service characteristics models 1088.

[0089] Each of these components is described in more detail below. Server 1340 obtains additional information through interaction with external services 1360 as needed.

Conceptual architecture

[0090] Referring now to FIG. 1, a simplified block diagram of a specific exemplary embodiment of an intelligent automated assistant 1002 is shown. As described in more detail herein, various embodiments of intelligent automated assistant systems can be configured to provide various special types of operations. , functionalities and/or features generally associated with technology based intelligent automated assistant. Additionally, as described in more detail herein, the many different operations, functionalities, and/or features of the intelligent automated assistant system disclosed herein may provide or provide different types of advantages and/or benefits to various entities interacting with the system(s). ) based on an intelligent automated assistant. The embodiment shown in FIG. 1 may be implemented using any of the hardware architectures described above, or using another type of hardware architecture.

[0091] For example, according to various embodiments, at least some intelligent automated assistant system(s) may be configured to provide various special types of operations, functionality, and/or features, such as, for example, one or more of the following (or a combination of the above):

- automating the use of data and services available on the Internet to discover, find, select, buy, reserve or order products and services. In addition to automating the process of using these data and services, the intelligent automated assistant 1002 can also provide simultaneous combined use of multiple data sources and services. For example, it can combine product information from multiple review websites, check prices and availability from multiple distributors and check their locations and time limits, and help the user find a personalized solution to their problem.

- automating the use of data and services available over the Internet to discover, collect information, select, reserve and otherwise learn about activities (including, but not limited to, films, events, performances, exhibitions, shows and attractions); places to go (including, but not limited to, travel destinations, hotels and other places to stay, sights and other points of interest, etc.); places to eat and drink (such as restaurants and bars), times and places to meet other people, and any other sources of entertainment or social interaction that can be found on the Internet.

- providing applications and services through natural language dialogue, which are otherwise provided through special applications with graphical user interfaces, including search (including location-based search); navigation (maps and route directions); search in the database (for example, finding organizations or people by name (name) or other properties); obtaining weather data and weather forecasts, checking the prices of commercial items or the status of financial transactions; monitoring traffic or flight status; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; communication via email or other messaging platforms; and local or remote operation of devices (eg, dialing telephone numbers, controlling lighting and temperature, controlling home security devices, playing music or video, etc.). In one embodiment, assistant 1002 may be used to initiate, operate, and manage many of the features and applications available on the device.

- offering personal recommendations for activities, products, services, source of entertainment, time management or any other type of recommendation service that benefits from interactive natural language dialogue and automated access to data and services.

[0092] According to various embodiments, at least a portion of the various types of functions, operations, actions, and/or other features provided by the intelligent automated assistant 1002 may be implemented on one or more client systems, on one or more server systems, and /or combinations of the above.

[0093] According to various embodiments, at least a portion of the various types of functions, operations, actions, and/or other features provided by assistant 1002 may be implemented by at least one embodiment of an automated call and response procedure, to example, illustrated and described, for example, with respect to Fig.33.

[0094] Additionally, various embodiments of assistant 1002 described herein may include or provide a number of different advantages and/or benefits over currently existing intelligent automated assistant technology, such as, for example, one or more of the following (or combinations thereof):

- Integration of natural language understanding and speech-to-text technology, which is limited by a set of explicit domain models, tasks, services and dialogs. Unlike assistant-based technology that attempts to implement a general purpose artificial intelligence system, the embodiments described herein can apply multiple sources of constraints to reduce the number of decisions to a more manageable size. This results in fewer ambiguous language interpretations, fewer relevant domains or tasks, and fewer ways to operationalize intent in services. Focusing on specific subject areas, tasks, and conversations also allows you to achieve scope and task coverage with a human-driven vocabulary and transformations from intent to service parameters.

- The ability to resolve user problems by activating services on their behalf over the Internet, using the API. Unlike search engines, which only return links and content, some embodiments of the automated assistants 1002 described herein can automate troubleshooting and problem solving activities. The ability to activate multiple services for a given request also provides more functionality to the user than is achieved by visiting a single Web site, for example, to create a product or service or find an activity.

- The use of personal information and personal interaction history when interpreting and fulfilling user requests. Unlike traditional search engines or question answering services, the embodiments described herein use information from personal interaction history (e.g., history of conversations, previous choices from results, etc.), personal physical context (e.g., location user and time) and personal information collected in the context of the interaction (for example, name, email addresses, physical addresses, telephone numbers, personal account numbers, preferences, etc.). The use of these sources of information provides, for example:

-- better interpretation of user input (eg, using personal history and physical context when interpreting the language);

-- more personalized results (for example, focused on preferences or recent elections);

-- increased efficiency for the user (for example, by automating steps that involve signing up for services or filling out forms).

- Use of dialog background in natural language interpretation of user inputs. Because embodiments can store personal history and apply natural language understanding to user inputs, they can also use conversational context such as current location, time, subject area, task stage, and task parameters to interpret new inputs. Traditional search engines and shells interpret at least one query regardless of the history of the dialogs. The ability to use dialogue backstory can allow for more natural interaction that resembles normal human conversation.

- Active input retrieval, in which assistant 1002 actively directs and restricts input from the user based on identical patterns and information used to interpret their input. For example, assistant 1002 can apply dialog models to recommend next steps in a dialog with the user in which they refine the query; suggest padding for partially typed input based on domain- and context-specific capabilities; or use semantic interpretation to choose from ambiguous interpretations of speech as text or text as intent.

- Explicit modeling and dynamic management of services with dynamic and reliable orchestration of services. The architecture of the described embodiments allows the helper 1002 to interact with many external services, dynamically determine which services can provide information for a particular user request, map user request parameters to different service APIs, invoke multiple services at once, integrate results from multiple services, recover gracefully after errors during the operation of the services and/or effectively support the implementation of the services as their API and characteristics improve.

- The use of active ontologies as a method and hardware system for assembling assistants 1002, which simplifies software development and data maintenance of automated assistant systems. Active ontologies are an integration of data modeling and execution environments for helpers. They provide the infrastructure to link different sources of models and data (domain concepts, task flows, vocabulary, language construct recognizers, conversational context, user personal information, and transformations from domain and task queries to external services). The active ontologies and other architectural innovations described in this document provide the practicality of building deep functionality across domains, unifying multiple sources of information and services, and doing so across a set of domains.

[0095] In at least one embodiment, the intelligent automated assistant 1002 may be configured to use and/or generate various special types of data and/or other types of information when performing specific tasks and/or operations. They may include, for example, input data/information and/or output data/information. For example, in at least one embodiment, the intelligent automated assistant 1002 may be configured to access, process, and/or otherwise use information from one or more different types of sources, such as, for example, one or more local and/or or remote storage devices, devices and/or systems. Additionally, in at least one embodiment, the intelligent automated assistant 1002 may be configured to generate one or more different types of output data/information, which, for example, may be stored in the memory of one or more local and/or remote devices. and/or systems.

[0096] Examples of various types of input/information that may be accessed and/or used by intelligent automated assistant 1002 may include, but are not limited to, one or more of the following (or a combination of the foregoing):

- Voice input: from mobile devices such as mobile phones and tablets, computers with microphones, Bluetooth headsets, in-vehicle voice control systems, via phone system, recordings in recording and messaging services, voice mail in integrated messaging services, household devices with voice input, such as clock radios, personal telephones, home electronic multimedia equipment control systems, and game consoles.

- Entering text from keyboards on computers or mobile devices, keypads on remote controls or other consumer electronic devices, mail messages sent to the assistant, instant messages or similar short messages sent to the assistant, text received from players in multiplayer gaming environments , and text passed in message flows.

- Location information coming from sensors or location-based systems. Examples include global positioning system (GPS) and GPS assisted technology (A-GPS) on mobile phones. In one embodiment, location information is combined with explicit user input. In one embodiment, the system of the present invention is able to detect when a user is at home based on known address information and current location determination. Thus, certain inferences can be drawn about the type of information a user might be interested in when at home as opposed to being away from home, and the type of services and actions that should be activated on behalf of a user depending on whether they are at home. or not.

- Time information from clocks in client devices. It may include, for example, the time from phones or other client devices indicating the local time and time zone. In addition, time can be used in the context of user queries, for example, to interpret phrases such as "in an hour" and "tonight".

- Compass, accelerometer, gyroscope and/or driving speed data and other sensor data from mobile or handheld devices or embedded systems such as automotive control systems. They may also include device position determination data from remote controls for devices and game consoles.

- Click (click) and menu item selection and other events from the graphical user interface (GUI) on any device that has a GUI. Additional examples include touch screen touches.

- Events from sensors and other data-driven triggers such as alarms, calendar alerts, price change triggers, location triggers, push notifications to the device from servers, etc.

[0097] Input in the embodiments described herein also includes a user interaction history context, including a history of dialogs and prompts.

[0098] Examples of various types of input/information that may be accessed and/or used by intelligent automated assistant 1002 may include, but are not limited to, one or more of the following (or a combination of the foregoing):

- Text output sent directly to the output device and/or device user interface.

- Text and graphics sent to the user via email.

- Text and graphics sent to the user via the messaging service.

- Speech output may include one or more of the following (or combinations of the above):

- Synthesized speech.

- Sampled speech.

-- Recorded messages.

- Graphical information layout with photos, lots of text, videos, sounds and hyperlinks. For example, content rendered in a web browser.

- An actuator pin to control physical actions in the device, such as instructing it to turn on or off, make a sound, change color, vibrate, control lights, etc.

- Activate other applications on the device, such as calling a mapping application, voice dialing a phone number, sending a mail or instant message, playing media, making entries in calendars, task managers and note-taking applications, and other applications.

- Actuator pin to control physical actions for devices connected to or controlled by the device, such as remote camera operation, wheelchair control, music playback on remote speakers, video playback on remote displays, etc.

[0099] It may be appreciated that the intelligent automated assistant 1002 of FIG. 1 is only one example of a wide range of intelligent automated assistant system embodiments that may be implemented. Other embodiments of the Intelligent Automated Assistant system (not shown) may include additional, fewer, and/or different components/features compared to what is illustrated, for example, in the exemplary embodiment of the Intelligent Automated Assistant system of FIG. one.

User interaction

[0100] Referring now to FIG. 2, an example of an interaction between a user and at least one embodiment of an intelligent automated assistant 1002 is shown. The example of FIG. , which may be a speech input engine, and the output is a graphical layout to an output device 1207, which may be a scrollable screen. Conversation screen 101A shows a conversational user interface showing what user 101B said ("I'd like a romantic Italian food place near my office") and helper 1002's response, which is a summary of his findings 101C ("OK, I found Italian restaurants , which, judging by the reviews, are romantic, not far from your work:") and the result set 101D (the first three from the list of restaurants are shown). In this example, the user clicks on the first result in the list and the result automatically opens to reveal more information about the restaurant shown on the information screen 101E. The information screen 101E and the conversation screen 101A may be displayed on the same output device such as a touch screen or other display device; the examples illustrated in FIG. 2 are two different output states for an identical output device.

[0101] In one embodiment, information screen 101E shows information collected and combined from a variety of services, including, for example, any or all of the following:

- addresses and geographic locations of organizations;

- distance from the user's current location;

- Reviews from many sources.

[0102] In one embodiment, information screen 101E also includes some examples of services that assistant 1002 can offer on behalf of a user, including:

- dialing a telephone number to call the organization ("call");

- remembering this restaurant for later use ("save");

- sending an email message to someone with directions on how to get there and information about this restaurant ("share");

- showing the location and directions on how to get to this restaurant on the map ("show on the map");

- saving personal notes about this restaurant ("my notes").

[0103] As shown in the example of Figure 2, in one embodiment, assistant 1002 includes intelligence beyond simple database management applications, such as, for example:

- processing the statement of intent in natural language 101B, and not just keywords;

- the inference of semantic intent from this language input, for example, the interpretation of "Italian food places" as "Italian restaurants";

- operationalizing a semantic intent into a strategy for using online services and executing that strategy on behalf of the user (eg, operationalizing a desire for a romantic place into a strategy for checking online review websites for reviews that describe the place as "romantic").

Intelligent Automated Assistant Components

[0104] According to various embodiments, intelligent automated assistant 1002 may include many different types of components, devices, modules, processes, systems, and the like, which, for example, may be implemented and/or specified through the use of hardware and /or combinations of hardware and software. For example, as illustrated in the exemplary embodiment of FIG. 1, assistant 1002 may include one or more of the following types of systems, components, devices, processes, and the like. (or combinations of the above):

- one or more active ontologies 1050;

- component(s) 1094 active input extraction (may include a client part 1094a and a server part 1094b);

- component(s) 1052 short-term personal storage device (may include a master version 1052b and cache 1052a);

- long-term personal storage component(s) 1054 (may include a master version 1052b and a cache 1052a);

- component(s) 1056 domain models;

- dictionary component(s) 1058 (may include a complete dictionary 1058b and a subset 1058a);

- component(s) 1060 of the language construct recognition module(s) (may include a complete library 1060b and a subset 1560a);

- component(s) 1070 language interpreter;

- base(s) 1072 data objects of the subject area;

- component(s) 1080 of the processor flow of dialogs;

- component(s) 1082 orchestration services;

- component(s) 1084 services;

- component(s) 1086 task flow models;

- component(s) 1087 dialog flow models;

- component(s) 1088 service models;

- component(s) 1090 output processor.

[0105] As described in connection with FIG. 7, in certain client/server embodiments, some or all of these components may be distributed between client 1304 and server 1340.

[0106] For purposes of illustration, at least a portion of the various types of components of a particular exemplary embodiment of an intelligent automated assistant 1002 will now be described in more detail with respect to the exemplary embodiment of the intelligent automated assistant 1002 of FIG.

Active ontologies 1050

[0107] Active ontologies 1050 serve as a unifying infrastructure that integrates models, components, and/or data from other parts of embodiments of intelligent automated assistants 1002. In the field of computer science engineering, ontologies provide structures for representing data and knowledge, such as classes/types , relationships, attributes/properties and their instance instantiation. Ontologies are used, for example, to assemble data and knowledge models. In some embodiments of the intelligent automation system 1002, the ontologies are part of a modeling framework in which models, such as domain models, can be assembled.

[0108] In the context of the present invention, the "active ontology" 1050 can also serve as a runtime environment in which various processing elements are hosted in an ontological manner (eg, having different attributes and relationships with other processing elements). These processing elements perform at least some of the tasks of the intelligent automated assistant 1002. Any number of active ontologies 1050 may be provided.

[0109] In at least one embodiment, active ontologies 1050 can be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or combinations of the above):

• Performing the functions of a modeling and development environment that integrates models and data from various model and data components, including, but not limited to:

- models of 1056 subject areas

- dictionary 1058

- database 1072 data objects of the subject area

- 1086 task flow models

- 1087 Dialog Flow Models

- service characteristics models 1088

• Performing the functions of a data modeling environment in which ontology-based editing tools can run to develop new models, data structures, schemas, and database views.

• Live execution of runtime functions specifying values for elements of domain models 1056, tasks 1086 and/or dialogs 1087, language construct recognizers and/or vocabulary 1058, and user-specific information, such as stored in short-term personal storage 1052 , long-term personal storage device 1054 and/or in the results of orchestration 1182 services. For example, some active ontology nodes may correspond to domain concepts such as a restaurant and its "restaurant name" property. During live execution, these active ontology nodes can be fleshed out with the identity of the particular restaurant object and its name, and how its name matches words when the natural language input is spoken. Thus, in this embodiment, the active ontology serves both as a modeling environment, specifying the concept that restaurants are objects with identities that have names, and for maintaining the dynamic links of these modeling nodes with data from object databases and parsing. natural language.

• Enabling communication and coordination between components and processing elements of an intelligent automated assistant, such as, for example, one or more of the following (or combinations of the above):

- component(s) 1094 active input extraction

- component(s) 1070 language interpreter

- component(s) 1080 of the dialog flow processor

- service orchestration component(s) 1082

- component(s) 1084 services

[0110] In one embodiment, at least a portion of the functions, operations, actions, and/or other features of the active ontologies 1050 described herein may be implemented, at least in part, using various methods and hardware systems, described in U.S. Patent Application Serial No. 11/518292 entitled "Method and Apparatus for Building the Intelligent Automated Assistant" filed September 8, 2006.

[0111] In at least one embodiment, a given active ontology instance 1050 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by active ontologies 1050 may include, but are not limited to, one or more of the following (or combinations of the above):

- static data that is available from one or more components of the intelligent automated assistant 1002;

- data that is dynamically instantiated per user session, such as, but not limited to, maintaining the state of user-specific inputs and outputs passed between components of the intelligent automated assistant 1002, contents of a short-term personal storage device, inferences drawn from previous user session states , etc.

[0112] Thus, active ontologies 1050 are used to unify elements of various components in an intelligent automated assistant 1002. Active ontology 1050 allows an author, designer, or system developer to integrate components such that elements of one component are identified by elements of other components. The author, designer or system developer can thus combine and integrate components in a simpler manner.

[0113] Referring now to FIG. 8, an example of a fragment of an active ontology 1050 according to one embodiment is shown. This example is intended to help illustrate some of the various types of functions, operations, actions, and/or other features that can be provided by active ontologies 1050.

[0114] The active ontology 1050 in FIG. 8 includes restaurant views and mealtime events. In this example, a restaurant is a concept 1610 with properties such as its name 1612, cuisines offered 1615, and its location 1613, which in turn can be modeled as a building block with properties for an actual address 1614. A meal event concept can be modeled as a node 1616 including a dinner party 1617 at a restaurant (which has 1619 attendees) and a time period 1618.

- Active ontologies may include and/or refer to domain models 1056 . For example, FIG. 8 illustrates a restaurant dining domain model 1622 associated with a restaurant concept 1610 and a meal event concept 1616. In this case, the active ontology 1050 includes a restaurant dining domain model 1622; in particular, at least two active ontology nodes 1050, namely restaurant 1610 and meal event 1616, are also included and/or referenced in restaurant dining domain model 1622. This domain model represents, among other things, the idea that a dinner party at a restaurant encompasses a meal event that takes place in restaurants. The active ontology nodes restaurant 1610 and meal event 1616 are also included and/or referenced in other intelligent automated assistant components, shown by dashed lines in FIG.

- Active ontologies may include and/or refer to task flow models 1086 . For example, FIG. 8 illustrates an event management task flow model 1630 that models event organization regardless of the subject areas applied to a domain-specific event type: a meal event 1616. Here, the active ontology 1050 includes a generic event management taskflow model 1630 that contains nodes representing events and other concepts involved in organizing them. The active ontology 1050 also includes a meal event node 1616, which is a specific kind of event. In this example, meal event 1616 is included or referenced in domain model 1622 and task flow model 1630, and both of these models are included and/or referenced in active ontology 1050. On the other hand, meal event 1616 is an example of how active ontologies can unify the elements of the various components included and/or referenced in other intelligent automated assistant components, shown by the dotted lines in FIG.

- Active ontologies may include and/or refer to dialog flow models 1087 . For example, FIG. 8 illustrates a conversation flow model 1642 for obtaining constraint values required for a transaction specified in the number of party members as a constraint, as represented in concept 1619. On the other hand, active ontology 1050 provides a framework for linking and unifying various components , for example, the dialog flow model 1087. In this case, the conversation flow model 1642 has the general concept of a constraint, which is specified in this particular example as the number of party members 1619 as an active ontology node. This particular dialog flow model 1642 operates under the abstraction of domain-independent constraints. The active ontology 1050 represents the party number property 1619 of the party node 1617 that is associated with the meal event node 1616 . In this embodiment, the intelligent automated assistant 1002 uses the active ontology 1050 to unify the constraint concept in the conversation flow model 1642 with the party party number property 1619 as part of a cluster of nodes representing the meal event concept 1616 that is part of the subject matter model 1622. areas for dining in restaurants.

- Active ontologies may include and/or refer to service models 1088 . For example, FIG. 8 illustrates a model of a restaurant reservation service 1672 associated with a conversation flow step to obtain the values required for that service to complete a transaction. In this case, the service model 1672 for the restaurant reservation service indicates that the reservation requires a value for the number of party attendees 1619 (the number of people seated at the table to be reserved). The number of party members 1619 as a concept that is part of the active ontology 1050 is also connected or linked to a common conversation flow model 1642 to query the user for restrictions on the transaction; in this case, the number of party members is the required limit for the conversation flow model 1642.

- Active ontologies may include and/or refer to databases 1072 of domain object data. For example, FIG. 8 illustrates a database of restaurant domain objects 1652 associated with a restaurant node 1610 in an active ontology 1050. The active ontology 1050 represents a general concept 1610 of a restaurant that can be used by various components of an intelligent automated assistant 1002 and is instantiated by data. about specific restaurants in the database of 1652 restaurants.

- Active ontologies may include and/or refer to vocabulary databases 1058. For example, FIG. 8 illustrates a vocabulary database 1662 of cuisines, e.g., Italian, French, etc., and words associated with each cuisine, e.g., "French", "continental", "provincial", etc. . The active ontology 1050 includes a restaurant node 1610 that is associated with a cuisine offering node 1615 that is associated with a representation of cuisines in the cuisine database 1662 . A particular entry in the database 1662 for a kitchen, for example, "French", is thus linked through the active ontology 1050 as an instance of the suggested cuisines concept 1615.

- Active ontologies can include and/or refer to any database that can be mapped to concepts or other representations in the ontology 1050. Domain object databases 1072 and vocabulary databases 1058 are just two examples of how an active ontology 1050 can integrate databases with each other and with other components of the automated assistant 1002. Active ontologies allow an author, designer, or system developer to specify a non-trivial mapping between the representations in the database and the representations in the ontology 1050. For example, the database schema for the restaurant database 1652 may represent the restaurant as a table of strings and numbers, or as a projection from a larger organization database, or any other representation suitable for database 1652. In this exemplary active ontology 1050, the restaurant 1610 is a concept node with properties and relationships organized differently from database tables. In this example, ontology nodes 1050 are associated with database schema elements. Database and ontology integration 1050 provides a unified view for interpreting and influencing specific data records in databases in terms of large sets of models and data in active ontology 1050. For example, the word "French" could be an entry in database 1662 of cuisines. Since, in this example, the database 1662 is being integrated into the active ontology 1050, the same word "French" is also interpreted as a possible cuisine in a restaurant that is included in the organization of meals, and this cuisine serves as a constraint to be used when using restaurant reservation services, etc. Active ontologies can thus integrate databases into the modeling and execution environment in order to interact with other components of the automated assistant 1002.

[0115] As described above, the active ontology 1050 allows an author, designer, or system developer to integrate components; thus, in the example of FIG. 8, component elements, such as the constraint in the conversation flow model 1642, can be identified by elements of other components, such as the required parameter of the restaurant reservation service 1672.

[0116] Active ontologies 1050 may be implemented as, for example, configurations of models, databases, and components, in which the relationships between models, databases, and components are any of the following:

- inclusion relation and/or implication;

- relationship with links and/or pointers;

- interaction via API, both internal relative to the program, and between programs.

[0117] For example, with reference now to FIG. 9, an example of an alternative implementation of an intelligent automated assistant system 1002 is shown in which the components of domain models 1056, vocabulary 1058, language construct recognition modules 1060, short-term personal memory 1052, and long-term personal The storage device 1054 is organized within a common container associated with the active ontology 1050, and other components, such as the active input retrieval component 1094, the language interpreter 1070, and the dialog flow processor 1080, are associated with the active ontology 1050 via API relationships.

Component(s) 1094 active input extraction

[0118] In at least one embodiment, active input extraction component(s) 1094 (which, as described above, may be implemented in a stand-alone configuration or in a configuration that includes both server and client components) may be configured to perform and/or implement various types of functions, operations, actions and/or other features, such as, for example, one or more of the following (or combinations of the above):

• Extraction, simplification and/or processing of input from the user or from the environment of the user and/or information regarding his need(s) or request(s). For example, if the user is searching for a restaurant, the input retrieval module may obtain information regarding the user's restrictions or preferences for location, time, cuisine, price, and so on.

• Facilitation of various types of input from various sources, such as, for example, one or more of the following (or combinations of the above):

- input from keyboards or from any other input device that forms text,

- keyboard input in user interfaces that offer dynamic recommended partial input completions,

- input from voice or speech input systems,

- input from graphical user interfaces (GUIs) in which users click, select or otherwise directly manipulate graphical objects to indicate selection options,

- input from other applications that generate text and send it to an automated assistant, including email, text messaging, or other text communication platforms.

[0119] By performing active input extraction, assistant 1002 is able to resolve intent ambiguity early in the input processing phase. For example, in an embodiment in which input is provided via speech, a signal may be sent to a server 1340 where words are extracted and semantic interpretation is performed. The results of this semantic interpretation can then be used to drive active input retrieval, which can offer the user alternative word choices from which to choose based on their degree of semantic match as well as phonetic match.

[0120] In at least one embodiment, active input retrieval component(s) 1094 actively, automatically, and dynamically directs the user towards inputs that can be influenced by one or more services offered by assistant 1002 embodiments. Referring now to 10 is a flowchart illustrating a method of operation for active input extraction components 1094 according to one embodiment.

[0121] The procedure begins at step 20. At step 21, assistant 1002 may propose interfaces on one or more input channels. For example, the user interface may offer the user options to speak or type or touch at any stage of the conversational interaction. At step 22, the user selects an input channel by initiating input in one modality, such as pressing a button, to start recording speech or invoke an interface for keyboard input.

[0122] In at least one embodiment, assistant 1002 suggests default recommendations for the selected modality 23. In other words, it suggests options 24 that are relevant in the current context before the user makes any input in that modality. For example, in a text entry modality, assistant 1002 may suggest a list of common words that should start text queries or commands, such as, for example, one or more of the following (or a combination of the above): imperative verbs (e.g., find, buy, reserve, get, call, check, schedule, etc.), nouns (e.g. restaurants, movies, events, organizations, etc.) or menu item style variants with names of subject areas of discourse (e.g. weather, sports , news, etc.)

[0123] If the user selects one of the default options at step 25, and a preference for autocomplete 30 is set, the procedure may immediately return. This is similar to the traditional menu item selection operation.

[0124] However, the initial option may be received as a partial input, or the user may begin the partial input 26. At any point in the input, in at least one embodiment, the user may choose to indicate that the partial input has ended 22, which causes the procedure to return.

[0125] In step 28, the last input selected or entered is added to the cumulative input.

[0126] At step 29, the system recommends the next possible inputs that are relevant, given the current input and other sources of constraints on what constitutes a relevant and/or meaningful input.

[0127] In at least one embodiment, the sources of user input constraints (eg, those used in steps 23 and 29) are one or more of various models and data sources that may be included in assistant 1002, which may include, but are not limited to, one or more of the following (or combinations of the foregoing):

- Dictionary 1058. For example, words or phrases that match the current input may be recommended. In at least one embodiment, a dictionary may be associated with any or one or more nodes of active ontologies, domain models, task models, dialog models, and/or service models.

- Domain models 1056, which may restrict inputs that may be instantiated or otherwise consistent with the domain model. For example, in at least one embodiment, the domain models 1056 can be used to recommend concepts, relationships, properties, and/or instances that are expected to match the current input.

- Language construct recognition modules 1060, which can be used to recognize idioms, phrases, grammatical constructs, or other constructs in the current input, and can be used to recommend completions that complete the construct.

- Domain object databases 1072 that can be used to recommend possible domain objects that match the input (eg, organization names, movie titles, event titles, etc.).

- Short-term memory 1052, which can be used to reconcile any previous input or part of the previous input and/or any other property or fact about the user interaction history. For example, partial input can be mapped to cities that the user encounters in the session, whether hypothetically (eg, mentioned in queries) and/or physically (eg, as determined from location sensors).

- In at least one embodiment, semantic paraphrases of recent inputs, query, or results can be matched against the current input. For example, if the user previously ran a "live music" query and obtained a list of concerts and then typed "music" in the active input retrieval environment, the recommendations could include "live music" and/or "concerts".

- Long-term personal storage device 1054, which can be used to recommend matching items from the long-term storage device. Such matched items may include, for example, one or more or any combination of the following: domain objects that are saved (e.g., "favorite" restaurants, movies, theaters, event venues, etc.), activity list items, list items, calendar entries, names of people in contact lists/address books, street or city names mentioned in contact lists/address books, etc.

- Task flow models 1086 that can be used to recommend inputs based on the next possible steps in the task flow.

- Dialog flow models 1087 that can be used to recommend inputs based on the next possible steps in the dialog flow.

- Service characteristic models 1088, which can be used to recommend possible services for use by name, category, characteristic, or any other property in the model. For example, the user may type part of the name of the preferred review web site, and the assistant 1002 may recommend a fill command to query that browse web site for a review.

[0128] In at least one embodiment, active input retrieval component(s) 1094 presents a conversational interface to the user, such as an interface in which the user and assistant communicate by alternately speaking phrases in a conversational manner. Active input extraction component(s) 1094 may be configured to implement and/or implement various types of conversational interfaces.

[0129] In at least one embodiment, active input retrieval component(s) 1094 may be configured to implement and/or implement various types of conversational interfaces in which assistant 1002 uses conversation taps to request, as a prompt, information from user according to dialog models. Dialog models may represent a procedure for conducting a dialog, such as, for example, a sequence of steps required to retrieve information needed to execute a service.

[0130] In at least one embodiment, active input retrieval component(s) 1094 offers real-time constraints and guidance to the user while the user is in the process of typing, speaking, or otherwise generating input. For example, active retrieval may direct a user to type text inputs that may be recognized by the assistant 1002 embodiment and/or that may be served by one or more of the services offered by assistant 1002 embodiments. This is an advantage over passively waiting for input without restrictions from the user, as it allows the user to focus on inputs that may be useful, and/or it allows implementations of the assistant 1002 to apply their real-time interpretations of the input as the user enters it.

[0131] At least a portion of the functions, operations, actions, and/or other features of actively extracting input described herein may be implemented, at least in part, using various methods and hardware systems described in the patent application. (USA), serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006.

[0132] According to specific embodiments, multiple instances or sub-processes of active input extraction component(s) 1094 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. .

[0133] According to various embodiments, one or more different sub-processes or instances of active input extraction component(s) 1094 may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering, by at least one instance of the active input extraction component(s) 1094 . Various examples of conditions or events that may trigger or start the implementation of one or more different sub-processes or instances of active input fetch component(s) 1094 may include, but are not limited to, one or more of the following (or combinations of the foregoing):

- Starting a user session. For example, when a user session launches an application that is an embodiment of assistant 1002, the interface may offer the ability for the user to initiate input, such as by pressing a button to initiate a speech input system or clicking a text field to initiate a text input session.

- User input detected.

- When the helper 1002 explicitly prompts the user for input, such as when it asks for an answer to a question or offers a menu of next steps from which to choose.

- When assistant 1002 assists the user in a transaction and collects data for that transaction, such as filling out a form.

[0134] In at least one embodiment, a given instance of active input retrieval component(s) 1094 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by active input fetching component(s) 1094 may include, but are not limited to, one or more of the following (or combinations of the above):

- database of possible words that should be used in text input;

- Grammar of possible phrases that should be used when pronouncing a text input;

- database of possible interpretations of speech input;

- database of previous inputs from the user or from other users;

- data from any of the various models and data sources that may be part of the helper 1002 embodiments, which may include, but is not limited to, one or more of the following (or combinations of the above):

- models of 1056 subject areas;

- dictionary 1058;

- modules 1060 recognition of language constructs;

- database 1072 data objects of the subject area;

- short-term storage device 1052;

- long-term personal storage device 1054;

- task flow models 1086;

- Dialog flow models 1087;

- service characteristics models 1088.

[0135] According to various embodiments, active input extraction component 1094 may apply active extraction procedures to, for example, one or more of the following (or combinations of the above):

- printable input;

- speech input;

- input from graphical user interfaces (GUI), including gestures;

- input from the recommendations offered in the dialog; and

- events from computational and/or sensed environments.

Active retrieval of typed input

[0136] Referring now to FIG. 11, a flowchart is shown illustrating a method for actively extracting typed input according to one embodiment.

[0137] The method begins 110. Assistant 1002 receives 111 partial text input, such as through an input device 1206. The partial text input may include, for example, the characters that have been typed up to that point in the text input field. At any time, the user may indicate that typing input is complete 112, such as by pressing the Enter key. If not completed, the recommendation generator generates 114 suggestions for recommendations 116. These recommendations may be syntactic, semantic, and/or other types of recommendations based on any of the information sources or constraints described herein. If a recommendation is selected 118, the input is converted 117 to include the selected recommendation.

[0138] In at least one embodiment, the recommendations may include extensions to the current input. For example, a recommendation for "rest" might be "restaurants".

[0139] In at least one embodiment, the recommendations may include replacing parts of the current input. For example, a recommendation for "rest" might be "places to eat".

[0140] In at least one embodiment, the recommendations may include replacing and rephrasing portions of the current input. For example, if the current input is "find restaurants in style", the recommendation could be "Italian", and when the recommendation is selected, the entire input could be overwritten as "find Italian restaurants".

[0141] In at least one embodiment, the resulting input that is returned is annotated 119 such that information about which options are selected in step 118 is stored along with the text input. This makes it possible to associate, for example, the semantic concepts or objects underlying the string with the string when it is returned, which improves the accuracy of subsequent language interpretation.

[0142] Referring now to FIGS. 12-21, screen shots are shown illustrating some parts of some procedures for actively extracting typed input, according to one embodiment. The screen shots illustrate an exemplary embodiment of assistant 1002 implemented on a smartphone, such as the iPhone, available from Apple Inc. of Cupertino, California. Input is provided to such a device via a touch screen that includes on-screen keyboard functionality. Those skilled in the art should recognize that the screenshots illustrate an embodiment that is merely exemplary, and that the techniques of the present invention may be implemented in other devices and using other layouts and arrangements.

[0143] In FIG. 12, screen 1201 includes a high-level recommendation set 1202 shown when no input is provided in field 1203. This corresponds to step 23 without input in FIG. 10 applied to step 114 in FIG. 11 when no input.

[0144] In FIG. 13, screen 1301 illustrates an example of using a dictionary to suggest recommended paddings 1303 of partial user input 1305 made in field 1203 using on-screen keyboard 1304. These recommended paddings 1303 may be part of active input retrieval function 1094. The user performs partial user input 1305 including the string "link". Dictionary component 1058 provides a mapping of this string to three different kinds of instances, which are listed as recommended populations 1303: the phrase "public and local events" is the events subject area category; "Business summary" is a local organizations search subject area category, and "Jewish community club" is the name of an instance of local organizations. Dictionary component 1058 may provide data lookup and management of namespaces such as these. The user may touch the Go button 1306 to indicate that he has completed input; this instructs assistant 1002 to continue with the filled text string as the unit of user input.

[0145] In FIG. 14, screen 1401 illustrates an example in which the recommended semantic completions 1303 for partial string "wh" 1305 include complete phrases with printable options. These kinds of recommendations may be provided through the use of one or more of the various input constraint models and sources described herein. For example, in one embodiment shown in FIG. 14, "what's going on in the city" is actively retrieving the location parameter for the local events domain; "where is the name of the organization" is the active extraction of the restriction in the form of the name of the organization for the subject area of the search for local organizations; "what is shown in the venue name" is the active extraction of the venue name constraint for the local events domain; and "what's on at the cinema" is the active retrieval of the cinema name constraint for the local events domain. These examples illustrate that the recommended populations are generated by models, and not simply retrieved from a database of previously entered queries.

[0146] In FIG. 15, screen 1501 illustrates the continuation of an identical example after the user enters additional text 1305 in field 1203. Recommended paddings 1303 are updated to match additional text 1305. In this example, data from base 1072 is used. data objects of the subject area: venues for events whose name begins with "f". It should be noted that this represents a much smaller and more semantically relevant set of recommendations than all words that begin with "f". On the other hand, recommendations are generated by applying a model, in this case a domain model, that represents local events as taking place in event venues, which are organizations with names. Recommendations actively retrieve inputs that should make up potentially meaningful records when using the Local Events service.

[0147] In FIG. 16, screen 1601 illustrates the continuation of an identical example after the user has selected one of the recommended paddings 1303. Active extraction continues by prompting the user to further indicate the type of information required, here by presenting a certain number of specifiers 1602, from which the user can to choose. In this example, these specifiers are generated by domain models, task flow, and conversation flow. The subject area is local events, which includes categories of events that take place on specified dates in specified locations and have the names of events and speakers. In this embodiment, the fact that these five options are offered to the user is generated from the dialog flow model, which indicates that users should be prompted for restrictions that they have not yet entered, and from the service model, which indicates that the five restrictions are options for services to provide a list of local events available to the assistant. Even the choice of preferred phrases to use as specifiers, such as "by category" and "participate", is generated from domain-specific vocabulary databases.

[0148] In Fig. 17, screen 1701 illustrates the continuation of an identical example after the user has selected one of the qualifiers 1602.

[0149] In Fig. 18, screen 1801 illustrates a continuation of an identical example, in which the selected specifier 1602 is added to field 1203 and additional specifiers 1602 are presented. The user can select one of the specifiers 1602 and/or provide additional text input through the keyboard 1304.

[0150] In FIG. 19, screen 1901 illustrates a continuation of the identical example, in which the selected specifier 1602 is added to field 1203, and more additional specifiers 1602 are presented. In this example, previously entered constraints are not actively redundantly extracted.

[0151] In FIG. 20, screen 2001 illustrates a continuation of an identical example in which the user touches the Go button 1306. The user's input is shown in field 2002, and a message is shown in field 2003, providing feedback to the user regarding the query being executed in response to the user's input.

[0152] In FIG. 21, screen 2101 illustrates a continuation of an identical example in which results are found. The message is displayed in field 2102. Results 2103, including input elements, allow the user to view more detailed information, save an identified event, purchase tickets, add notes, and the like.

[0153] On the same screen 2101 and other displayed screens can be scrolled, allowing the user to scroll up to see the screen 2001 or other previously presented screens, and make changes to the request if necessary.

Active extraction of speech input

[0154] Referring now to FIG. 22, a flowchart is shown illustrating a method for actively extracting input for voice or speech input, according to one embodiment.

[0155] The method begins 221. Assistant 1002 receives 121 voice or speech input in the form of an audio signal. The speech-to-text service 122 or processor generates a set of text interpretations 124 from the audio signal. In one embodiment, the speech-to-text service 122 is implemented using, for example, a nuance recognition module available from Nuance Communications, Inc., Burlington, MA.

[0156] In one embodiment, assistant 1002 uses statistical language models to generate text interpretations 124 for speech input 121.

[0157] In addition, in one embodiment, statistical language models are tuned to look for words, titles, and phrases that appear in various models of assistant 1002 shown in FIG. For example, in at least one embodiment, words, names, and phrases from some or all of the following are provided to the statistical language models: domain models 1056 (e.g., words and phrases associated with restaurant and mealtime events), models 1086 task flow (e.g., words and phrases associated with organizing an event), dialog flow models 1087 (e.g., words and phrases associated with restrictions that are needed in order to collect inputs for a restaurant reservation), a domain object database 1072 ( for example, restaurant names), vocabulary databases 1058 (for example, kitchen names), service models 1088 (for example, service provider names such as OpenTable), and/or any words, names, or phrases associated with any node of the active ontology 1050.

[0158] In one embodiment, the statistical language models are also configured to look up words, titles, and phrases from the long-term personal storage device 1054. For example, text from activity list items, list items, personal notes, calendar entries can be provided to the statistical language models. , names of people in contact lists/address books, e-mail addresses, street or city names mentioned in contact lists/address books, etc.

[0159] The ranking component analyzes the interpretation options 124 and ranks 126 them according to how well they match the syntactic and/or semantic models of the intelligent automated assistant 1002. Any sources of restrictions on user input can be used. For example, in one embodiment, helper 1002 can rank the output of the speech-to-text interpreter according to how well the parsing of the interpretations in the syntactic and/or semantic sense, the domain model, the task flow model, and/or the dialog model, etc., is performed. p.: it evaluates how well the various combinations of words in the interpretations of the text 124 correspond to the concepts, relationships, objects, and properties of the active ontology 1050 and its associated models. For example, if the speech-to-text service 122 generates two interpretations of "Italian food for lunch" and "Italian shoes for lunch", semantic relevance ranking 126 may rank "Italian food for lunch" higher if it better matches active ontology nodes. 1050 helper 1002 (for example, the words "Italian", "food" and "lunch" match nodes in the ontology 1050, and they are all connected through relationships in the ontology 1050, while the word "shoes" does not match the ontology 1050 or matches the node , which is not part of the Dining Out Domain Network).

[0160] In various embodiments, the algorithms or procedures used by the helper 1002 for interpreting text inputs, including any embodiment of the natural language processing procedure shown in FIG. 28, may be used to rank and quantify the options. 124 text interpretations generated by the speech-to-text service 122.

[0161] In one embodiment, if the ranking component 126 determines 128 that the highest ranked speech interpretation of the interpretations 124 ranks above a specified threshold, the highest ranking interpretation may be automatically selected 130. If none of the interpretations ranks above a specified threshold values, possible interpretations of speech 134 are presented 132 to the user. The user can then choose 136 between the displayed choices.

[0162] In various embodiments, user selection 136 between displayed choices may be achieved by any input mode, including, for example, any of the multi-modal input modes described in connection with FIG. Such input modes include, without limitation, actively retrieved typing input 2610, actively retrieved speech input 2620, actively presented GUI input 2640, and the like. In one embodiment, the user may select from interpretation options 134, such as by touch or by speaking. In the case of pronunciation, the possible interpretation of the new speech input is severely limited by the small set of choices 134 offered. phrase "Italian food" and not get confused with other global input interpretations.

[0163] Regardless of whether the input is selected 130 automatically or selected 136 by the user, the resulting input 138 is returned. In at least one embodiment, the returned input is annotated 138 so that information about which options are selected at step 136 is stored along with the text input. This makes it possible to associate, for example, the semantic concepts or objects underlying the string with the string when it is returned, which improves the accuracy of subsequent language interpretation. For example, if "Italian food" is offered as one of the interpretation options 134 based on the semantic interpretation Cuisine=Italian Food, then the machine-readable semantic interpretation may be sent along with the user's selection of the string "Italian food" as the annotated text input 138.

[0164] In at least one embodiment, text interpretation options 124 are generated based on speech interpretations received as output from the speech-to-text service 122.

[0165] In at least one embodiment, text interpretation options 124 are formed by paraphrasing speech interpretations in terms of their semantic meaning. In some embodiments, there may be multiple paraphrases of an identical interpretation of speech recommending different meanings of the word or homonym alternatives. For example, if the speech-to-text service 122 indicates "meeting place", the interpretations presented to the user can be paraphrased as "meeting place (local organizations)" and "meeting place (restaurants)".

[0166] In at least one embodiment, text interpretation options 124 include recommendations to correct substrings.

[0167] In at least one embodiment, text interpretation options 124 include recommendations to correct substrings of interpretation options using syntactic and semantic analysis as described herein.

[0168] In at least one embodiment, when the user selects an interpretation option, it is returned.

[0169] In at least one embodiment, the user is provided with an interface to edit the interpretation before it is returned.

[0170] In at least one embodiment, the user is prompted with an interface to continue additional voice input before the input is returned. This gives him the ability to incrementally compose the utterance of the input, obtaining syntactic and semantic corrections, recommendations, and directions in a single iteration.

[0171] In at least one embodiment, the user is prompted with an interface to navigate directly from 136 to step 111 of the active type input retrieval method (described above in connection with FIG. 11). This gives him the ability to intersperse typed and spoken input, getting syntactic and semantic corrections, recommendations, and directions in one step.

[0172] In at least one embodiment, the user is provided with an interface to proceed directly from step 111 of the Active Type Input Extraction embodiment to the Active Voice Input Extraction embodiment. This gives him the ability to intersperse typed and spoken input, getting syntactic and semantic corrections, recommendations, and directions in one step.

Active GUI based input retrieval

[0173] Referring now to FIG. 23, a flowchart is shown illustrating a method for actively extracting input for a GUI-based input according to one embodiment.

[0174] The method begins 140. Assistant 1002 presents 141 a graphical user interface (GUI) on output device 1207, which may include, for example, links and buttons. The user interacts 142 with at least one GUI element. Data 144 is received and converted 146 to a universal format. The converted data is then returned.

[0175] In at least one embodiment, some GUI elements are generated dynamically from active ontology models rather than written into a computer program. For example, assistant 1002 may suggest a set of restrictions to guide the restaurant reservation service as touch areas on the screen, with each area representing a restriction name and/or value. For example, a screen might have dynamically generated GUI layout lines with areas for cuisine, location, and price range restrictions. If the active ontology models change, then the GUI screen should automatically change without reprogramming.

Actively fetching input dialog recommendations

[0176] FIG. 24 is a flowchart illustrating a method for actively retrieving input at a dialog flow level, according to one embodiment. Assistant 1002 recommends 151 possible answers 152. The user selects 154 recommended answers. The input received is converted 154 to a universal format. The converted data is then returned.

[0177] In at least one embodiment, the recommendations provided in step 151 are offered as subsequent steps in the flow of dialogs and/or tasks.

[0178] In at least one embodiment, the recommendations offer options to refine the query, for example, using parameters from the domain model and/or tasks. For example, you may be prompted to change the allowed location or time of the request.

[0179] In at least one embodiment, the recommendations offer options to choose between ambiguous alternative interpretations provided by a language interpretation procedure or component.

[0180] In at least one embodiment, the recommendations offer options to choose between ambiguous alternative interpretations provided by a language interpretation procedure or component.

[0181] In at least one embodiment, the recommendations offer options to choose between the following steps in the flow-associated processing of the dialog flow model 1087. For example, the dialog flow model 1087 may recommend that, after collecting constraints for one subject area (eg, dinner at a restaurant), helper 1002 recommend other related subject areas (eg, a nearby movie).

Active monitoring of relevant events

[0182] In at least one embodiment, asynchronous events can be processed as inputs in a manner similar to other active fetchable input modalities. Thus, such events may be provided as inputs to helper 1002. Once interpreted, such events may be processed in a manner similar to any other input.

[0183] For example, a change in the status of a flight may trigger an alert notification to be sent to the user. If a flight is indicated as being late, assistant 1002 may continue the conversation by presenting alternative flights, generating other recommendations, and the like. based on the detected event.

[0184] Such events can be of any type. For example, assistant 1002 may detect that the user has simply returned home or deviated (from a specified route), or that a stock price is reaching a threshold, or that a TV show of interest to the user is starting, or that a musician of interest is on tour. in this area. In any of these cases, assistant 1002 may continue the conversation in much the same way as if the user himself initiates the request. In one embodiment, the events may even be based on data provided from other devices, for example, to tell the user when a colleague has returned from lunch (the colleague's device may transmit such an event to the user's device, at which point in time assistant 1002 set to user's device responds accordingly).

[0185] In one embodiment, the events may be notifications or alerts from a calendar, clock, reminder application, or activity list. For example, an alert from a calendar application regarding a dinner date may initiate a dialogue with assistant 1002 regarding a dinner appointment at a restaurant. The conversation can continue as if the user has just spoken or typed information regarding an upcoming dinner at a restaurant, such as "dinner for two in San Francisco".

[0186] In one embodiment, the context of the possible event trigger 162 may include information about people, places, times, and other data. This data may be used as part of the input to helper 1002 to be used at various stages of processing.

[0187] In one embodiment, this data from the context of the event trigger 162 can be used to disambiguate speech inputs or text inputs from the user. For example, if an alert regarding a calendar event includes the name of a person invited to the event, this information can help disambiguate input that may match multiple people with the same or similar name.

[0188] Referring now to FIG. 25, a flowchart is shown illustrating a method for actively monitoring relevant events according to one embodiment. In this example, the event trigger events are input sets 162. Helper 1002 monitors for 161 such events. Detected events may be filtered and sorted 164 for semantic relevance using models, data, and information available from other components in the intelligent automated assistant 1002. For example, an event that reports a change in flight status may be given higher relevance if the records are of short duration or long-term memory to the user indicate that the user is on that flight and/or query assistant 1002 about it. This sorting and filtering can then only present the top events for review by the user, who can then select one or more of them and act on them.

[0189] The event data is converted 166 to the universal input format and returned.

[0190] In at least one embodiment, assistant 1002 may pre-suggest services associated with events that are recommended for the user's attention. For example, if the flight status alert indicates that the flight may be cancelled, assistant 1002 may recommend a task flow to the user to re-plan an itinerary or book a hotel.

Example of input fetch component(s)

[0191] The following example is intended to help illustrate some of the various types of functions, operations, actions, and/or other features that can be provided by active input retrieval component(s) 1094.

Example: Completing a command (what can the user say to helper 1002?)

[0192] The user is presented with a text input field with a general instruction to enter "What do you want to do?" Depending on the context and user input, any of several system responses are provided. Examples are shown below.

Case: Zero input

User input System response A comment If the field is empty and there is no dialog context, typing a question or a space, or no input for a period of time.| - Find restaurants, hotels, movies
- Book a table, room, place, ...
- Order food, flowers, ...
- Buy ticket, book, DVD, ...
- Make a schedule for a meeting, event,...
- Remind about the lesson, ...
- Explain results Black text is inserted into the input field. Italic text represents further guidance after the text is pasted.

Case: Entering the first word

User input System response A comment In an empty dialog context, if the user types the first letter of any command, it is recommended. For example,
F| - Find restaurants, hotels, movies
- Black text is inserted into the input field. Italic text represents further guidance after the text is pasted.

Case: Entering a keyword

User input System response A comment In an empty dialog context, if the user types the first two letters of any keyword, it is recommended, as is the command. For example,
Re| - Remind about the lesson, ...
- Find restaurants by location, cuisine
- Black text is inserted into the input field. Italic text represents further guidance after the text is pasted.

Case: Hint for Arguments

User input System response A comment The user has entered or selected a "command" verb and nothing else.
For example,
find_ | (Find) - restaurants by location, cuisine
- hotels by location, accessibility, ...
- movies by location, genre, ...
- place name Not only is a choice class offered, but advice is given on constraint options. It should be noted that the place name is just a hint string and should not insert text.

Case: Recommendation criteria

User input System response A comment The user has already entered enough to set the task and domain, and is now prompted as a prompt for order restrictions.
Find restaurants_ | (Find restaurants) - in the city, postal code
- near home, office, place name
- the name of the restaurant with the name
- Black text is inserted into the input field. Italic text represents further guidance after the text is pasted.

Case: Adding Criteria

User input System response A comment The user said "in" and is now prompted for location hints
Find restaurants in_ | (Find restaurants in) - city, postal code
- Palo Alto, California
- Menlo Park, California Until he enters the next word, the suggestions are a hint string as to what to type. Places are cities from personal storage. The user said "about" and is now prompted as a hint for locations
Find restaurants near_ | (Find restaurants nearby) - of my house
- my work
- Palo Alto, California
- Menlo Park, California The locations are one or more locations from a personal storage device.

Case: Adding location or other restrictions

User input System response A comment Find restaurants in Palo Alto_ | (Find restaurants in Palo Alto) - offering a type of food or cuisine
- with availability tonight, tomorrow, ...
- described as romantic, with good service Prompt line for other restrictions Find restaurants in Palo Alto with availability_ | (Find restaurants in Palo Alto with availability) - tonight
- tomorrow
- at 19:00
- at 21:00
- other time or date 19:00 and 21:00 are our recommendations based on the current time

Case: Starting with a constraint, unknown task or domain

User-
input System response A comment romantic_ | (romantic) - restaurants by location, cuisine, ...
- hotels by location, accessibility, ...
- movies by location, genre, ...
- The constraint value from the ontology tells you which selection classes to use. comedy_ | (comedy) - movies by location, genre, ...
- location activities,...
- clubs by location "Comedy" is a constraint value for genre in movies, genre in events, and "comedy clubs" is a category in the local business directory

Example: Filling in the title

[0193] Here, the user has typed some text without confirming any of the commands, or he simply completes the command with the name of the object. The system may try to fill in the names depending on the context. It also resolves domain ambiguity.

Case: words without context

User input System response A comment for - Il Fornaio (restaurant)
- Ill Forgotten Gains (film) May require a search by object name. It should be noted verbose filling. Displaying the subject area as a recommendation. tom cruise (tom cruise) - movies May require a search by object name. romantic (romantic) - movies
- restaurants Ontology based only

Case: title with context

User input System response A comment Find restaurants il for (Find restaurants il for) - Il Fornaio (restaurant)
- May require a search by object name. Uses the selection class and location context from current and previous requests to limit padding. Displaying the subject area as a recommendation. tom cruise (tom cruise) - movies May require a search by object name. Find movies rom (Find movies rom) - romantic
- Roman Holiday (film) Ontology based only g-rated - movies

Example: Selecting values from a set

[0194] Here, the user responds to a system request to enter a value for a specific parameter such as location, time, cuisine, or genre. The user can either select from a list or enter a value. As it prints, matching items from the list are shown as options. Examples are shown below.

Case: valueclass selection

User input System response A comment Restaurants serving (Restaurants offering) - hamburgers
- ice cream
- hot dogs The user can type anything else and does not necessarily need to select from a menu. Restaurants serving h (Restaurants serving h) - hamburgers
- hot dogs
- spicy sauce Movies playing (Movies going) - today
- tonight
- Friday night

Example: Reusing Previous Commands

[0195] The previous requests are also options to be filled in the autocomplete interface. They may simply match as strings (when the input field is empty and there are no known restrictions), or they may be recommended as relevant in certain situations.

Case: filling previous requests

User input System response A comment Ital - Italian restaurants (regular filling)
- films starring Italian actors (recent request) Using a string match to retrieve previous requests. Lunch - eateries offering dinners at marin (recent request)
- buy the book "Naked lunch"

Example: Retrieving Personal Storage Items

[0196] Assistant 1002 may store certain events and/or objects in a personal storage device associated with the user. Autocomplete can be done based on such memorized items. Examples are shown below.

Case: filling by events and objects in a personal storage device

User input System response A comment Mary - Lunch with Mary on Saturday (from My Events)
- movies titled "Something about Mary" Lunch - Lunch with Mary on Saturday (from My Events)
- buy the book "Naked lunch" (from My Todos (My classes)) hob - Hobee Restaurant in Palo Alto (from My Restaurants)

Active retrieval of multimodal input

[0197] In at least one embodiment, active input extraction component(s) 1094 can process input from multiple input modalities. At least one modality may be implemented with an active input retrieval procedure that takes advantage of specific input types and methods to select from the options provided. As described herein, these may be embodiments of procedures for actively retrieving input for text input, speech input, GUI-based input, dialog-context input, and/or input resulting from event triggers.

[0198] In at least one embodiment, for one instance of the intelligent automated assistant 1002, there may be support for one or more (or any combination) of type input, speech input, GUI input, dialog input, and/or event input.

[0199] Referring now to FIG. 26, a flowchart is shown illustrating a method for actively retrieving a multi-modal input according to one embodiment. The method begins 100. Inputs may be received simultaneously from one or more or any combination of input modalities in any order. Thus, the method includes actively retrieving typed input 2610, speech input 2620, GUI-based input 2640, input 2650 in the context of a dialog, and/or input 2660 resulting from event triggers. Any or all of these input sources are unified into a unified input format 2690 and returned. The unified input format 2690 allows the development and operation of other components of the intelligent automated assistant 1002 regardless of the specific input modality.

[0200] Recommending active guidelines for multiple modalities and levels provides restriction and input guidelines beyond the restrictions and guidelines available for isolated modalities. For example, the kinds of recommendations offered for choosing between speech, text, and dialogue stages are independent, so that their combination is a significant improvement over adding active extraction technologies to separate modalities or levels.

[0201] A combination of multiple sources of constraints as described herein (syntactic/linguistic, vocabulary, object databases, domain models, task models, service models, etc.) and multiple places where these constraints can be active apply (speech, text, GUI, dialog, and asynchronous events) provides a new level of functionality for human-machine interaction.

Component(s) 1056 domain models

[0202] The component(s) of domain models 1056 includes representations of concepts, objects, relationships, properties, and domain instances. For example, restaurant dining domain model 1622 may include a restaurant concept as an organization with a name and address and a phone number, a meal event concept with the number of party attendees, and a date and time associated with the restaurant.

[0203] In at least one embodiment, the domain model component(s) 1056 of the assistant 1002 may be configured to implement and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- The domain model component(s) 1056 can be used by the automated assistant 1002 for several processes including: input retrieval 100, natural language interpretation 200, coordination for services 400, and output generation 600.

- The domain model component(s) 1056 may provide lists of words that may match a domain concept or object, such as restaurant names, that may be used for active input retrieval 100 and natural language processing 200.

- Component(s) 1056 of the domain model can classify the variants of words in the processes, for example, to determine that the word is the name of a restaurant.

- Component(s) 1056 of the domain model can show the relationship between partial information for natural language interpretation, e.g. that a cuisine can be associated with commercial organizations (e.g. "local Mexican food" can be interpreted as "find restaurants with style=Mexican" , and this inference is possible due to the information in the domain model 1056).

- Component(s) 1056 domain model can organize information about the services used in orchestration 1082 services, for example, that a particular web service can provide reviews of restaurants.

The domain model component(s) 1056 can provide information for generating natural language paraphrases and other output formatting, for example, by providing canonical ways of describing concepts, relationships, properties, and instances.

[0204] According to specific embodiments, multiple instances or sub-processes of domain model component(s) 1056 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software . For example, in at least some embodiments, various aspects, features, and/or functionality of domain model component(s) 1056 may be performed, implemented, and/or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes, etc. (or combinations of the above):

- Component(s) 1056 domain models can be implemented as data structures that represent concepts, relationships, properties and instances. These data structures may be stored in storage, files, or databases.

- Access to the component(s) 1056 of the domain model can be implemented through direct APIs, network APIs, database query interfaces, and the like.

- Creation and maintenance of component(s) 1056 domain models can be done, for example, through direct file editing, database transactions and/or using tools for editing domain models.

The domain model component(s) 1056 may be implemented as part of or in association with active ontologies 1050 that combine models with model instantiations for servers and users.

[0205] According to various embodiments, one or more different sub-processes or instances of domain model component(s) 1056 may be triggered in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering, by at least one instance of the component(s) 1056 domain models. For example, initiating or starting implementation of one or more different sub-processes, or instances of domain model component(s) 1056 may be triggered when domain model information is required, including during input retrieval, input interpretation, task and domain identification, natural language processing, service orchestration, and/or output formatting for users.

[0206] In at least one embodiment, a given instance of domain model component(s) 1056 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. For example, data from domain model component(s) 1056 can be associated with other model-based modeling components, including vocabulary 1058, language construct recognition modules 1060, dialog flow models 1087, task flow models 1086, service characteristics models 1088, a domain object data base 1072, and the like. For example, the organizations in the domain object databases 1072 that are classified as restaurants can be known through type identifiers that are supported in the restaurant dining domain model components.

Example component(s) of domain models

[0207] Referring now to FIG. 27, a set of screenshots is shown illustrating an example of various types of functions, operations, actions, and/or other features that can be provided by domain model component(s) 1056 according to one embodiment.

[0208] In at least one embodiment, domain model component(s) 1056 is a unifying data view that provides a view of the information shown on screens 103A and 103B about a restaurant that combines data from several different data sources and services and which includes, for example: name, address, categories of organizations, telephone number, identifier for storing in a long-term personal storage device, identifier for sharing by e-mail, overviews from multiple sources, map coordinates, personal notes, and the like.

Component(s) 1070 language interpreter

[0209] In at least one embodiment, the language interpreter component(s) 1070 of the assistant 1002 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

• Parsing user input and identifying a set of grammar analysis results.

- The user input may include any information from the user and from the context of their device that may assist in understanding the user's intent, which may include, for example, one or more of the following (or a combination of the above): word sequences, gesture identities, or GUI elements involved in retrieving input, the current dialog context, the current device application and its current data objects, and/or any other personal dynamic data obtained about the user, such as location, time, etc. For example, in one embodiment, the user input is in the form of a generic annotated input format 2690 resulting from active input retrieval 1094 .

- Parsing results are associations of data in user input with concepts, relationships, properties, instances, and/or other nodes and/or data structures in models, databases, and/or other representations of user intent and context. Parsing result associations can be complex transformations from sets and sequences of words, signals, and other user input elements into one or more of the associated concepts, relationships, properties, instances, other nodes, and/or data structures described herein.

• Parsing user input and identifying a set of parsing results, which are grammar analysis results that associate data in user input with structures that represent syntactic parts of speech, expressions, and phrases, including wordy names, sentence structure, and/or other grammatical structures. graph. The parsing results are described in element 212 of the natural language processing procedure described in connection with FIG.

• Parsing user input and identifying a set of semantic analysis results, which are grammatical analysis results that associate the data in the user input with structures that represent concepts, relationships, properties, objects, quantities, judgments, and/or other representations of the user's meaning and intent. In one embodiment, these representations of meaning and intent are represented by sets and/or elements and/or instances of models or databases and/or nodes in ontologies, as described in element 220 of the natural language processing procedure described in connection with FIG. 28.

• Disambiguation between alternative results of syntactic or semantic analysis, as described in element 230 of the natural language processing procedure described in connection with FIG.

• Determining whether or not a partially typed input is syntactically and/or semantically meaningful in an autocomplete routine, such as the routine described in connection with FIG.

• Assistance in generating recommended paddings 114 in an autocomplete routine, such as the routine described in connection with FIG.

• Determining whether or not interpretations of spoken input are syntactically and/or semantically meaningful in a speech input procedure, eg, in the procedure described in connection with FIG.

[0210] According to specific embodiments, multiple instances or subprocesses of language interpreter component(s) 1070 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software.

[0211] According to various embodiments, one or more different subprocesses or instances of language interpreter component(s) 1070 may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering at least least one instance of the component(s) 1070 language interpreter. Various examples of conditions or events that may trigger or begin implementation of one or more different subprocesses or instances of language interpreter component(s) 1070 may include, but are not limited to, one or more of the following (or combinations of the foregoing):

• when extracting input, including but not limited to:

- the recommendation of possible fillings of the printed input 114 (Fig.11);

- ranking speech interpretations 126 (FIG. 22);

- when recommending ambiguities as recommended answers in dialogue 152 (Fig.24);

• when an input retrieval result is available, including when an input is retrievable via any of the multimodal input active retrieval 100 modes.

[0212] In at least one embodiment, a given instance of language interpreter component(s) 1070 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of such database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by the language interpreter component(s) may include, but are not limited to, one or more of the following (or combinations of the above):

- models of 1056 subject areas;

- dictionary 1058;

- database 1072 data objects of the subject area;

- short-term storage device 1052;

- long-term personal storage device 1054;

- task flow models 1086;

- Dialog flow models 1087;

- service characteristics models 1088.

[0213] Also referring now to FIG. 29, a screenshot is shown illustrating natural language processing according to one embodiment. The user provides (via voice or text) a language input 2902 consisting of the phrase "who is playing filmore this weekend". This phrase is displayed to the user on the screen 2901. The language interpreter component(s) 1070 process the input 2902 component by component and generate the result of the grammatical analysis. The parsing result associates this input with a query to show local events that are scheduled for any of the upcoming weekends at any event location that has the same name as "filmore". Paraphrasing the results of the grammar analysis is shown as 2903 on screen 2901.

[0214] Also referring now to FIG. 28, a flowchart is shown illustrating an example of a method for natural language processing according to one embodiment.

[0215] The method starts 200. Language input 202 is received, such as the string "Who's playing filmore this weekend" in the example of FIG. In one embodiment, the input is expanded with current contextual information such as the user's current location and local time. When matching 210 words/phrases, language interpreter component(s) 1070 finds associations between user input and concepts. In this example, the associations are between the string "plays" and the concept of listings in venues; the string "this weekend" (together with the user's current local time) and a specification of an approximate time period that represents the coming weekend; and the string "filmore" with the name of the venue. Word/phrase mapping 210 may use data from, for example, language construct recognition modules 1060, vocabulary database 1058, active ontology 1050, short-term personal storage 1052, and long-term personal storage 1054.

[0216] The language interpreter component(s) 1070 generates parsers 212 that include the selected parse result, but may also include other parse results. For example, other grammatical analysis results may include results in which "plays" is associated with other subject areas, such as games, or with a category of event, such as a sporting event.

[0217] Short-term and/or long-term storage 1052, 1054 can also be used by language interpreter component(s) 1070 in generating parsing options 212. Thus, input that is provided earlier in an identical session and/or known information about the user can be used to improve performance, reduce ambiguity, and enhance the conversational nature of the interaction. Data from the active ontology 1050, domain models 1056, and task flow models 1086 can also be used to implement reasoning based on justification in determining valid parsing options 212.

[0218] In semantic negotiation 220, language interpreter component(s) 1070 consider combinations of possible grammatical results according to how well they match semantic models, such as domain models and databases. In this case, the parsing includes associating (1) "plays" (a word in user input) as "local event at the event site" (part of the domain model 1056 represented by the cluster of nodes in the active ontology 1050) and ( 2) "filmore" (another word in the input) as a match to the object name in the local event venue domain object database 1072, which is represented by the domain model element and the active ontology node (venue name).

[0219] Semantic negotiation 220 may use data from, for example, active ontology 1050, short-term personal storage 1052, and long-term personal storage 1054. For example, semantic negotiation 220 may use data from previous references to venues or local events in the dialog ( from short-term personal storage 1052) or favorite venues (from long-term personal storage 1054).

[0220] A set of variants of the results (or potential results) of the semantic analysis is formed 222.

[0221] At the disambiguation step 230, the language interpreter component(s) 1070 weights the justification-based intensity of the variants 222 of the semantic analysis results. In this example, the combination of parsing "plays" as "local event at the venue" and the match "filmore" as the name of the venue is a stronger match with the domain model than alternative combinations in which, for example, " plays" is associated with the sports domain model, but there is no association in the sports domain model for "filmore".

[0222] Disambiguation 230 may use data from, for example, the structure of the active ontology 1050. In at least one embodiment, connections between nodes in the active ontology provide justification-based support for disambiguating between variants 222 of semantic analysis results. For example, in one embodiment, if three active ontology nodes semantically match and all connect in the active ontology 1050, this indicates a higher justification-based intensity of semantic analysis than when these matching nodes are not connected or are connected via longer connection paths in the active ontology. 1050. For example, in one embodiment of semantic matching 220, for a grammatical parse that matches both " event location local " and "event location name ", enhanced justification-based support is provided because the combined representations of these aspects of the user's intent are connected via links and/or relationships in the active ontology 1050: in this case, the local event node is connected to the event venue node, which is connected to the venue name node, which is connected to n the name of the object in the database of event venue names.

[0223] In at least one embodiment, the connections between nodes in the active ontology that provide justification-based support for resolving ambiguities between semantic analysis results options 222 are directed arcs forming an inference lattice in which matching nodes represent justification for the nodes to which they are connected by directed arcs.

[0224] At step 232, the language interpreter component(s) 1070 sorts and selects 232 major semantic parsers as representing the user's intent 290.

Database(s) of 1072 domain object data

[0225] In at least one embodiment, the domain object database(s) 1072 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

• Saving data about the objects of the subject area. Domain objects are entities in the world or in a computing environment that can be modeled in domain models. Examples may include, but are not limited to, one or more of the following (or combinations of the above):

- organizations of any kind;

- films, videos, songs and/or other musical products and/or any other entertainment products with a title;

- products of any kind;

- activity;

- entries in the calendar;

- cities, states, countries, neighborhoods and/or other geographic, geopolitical and/or geospatial points or areas;

- places with a name, for example, attractions, airports, etc.;

• Provision of database services for these databases, including, but not limited to, simple and complex queries, transactions, trigger events, and the like.

[0226] In particular embodiments, multiple instances or subprocesses of the domain object database 1072 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, various aspects, features, and/or functionality of the domain object database 1072 may be performed, implemented, and/or initiated by database software and/or hardware residing on the client( ax) 1304 and/or on the server(s) 1340.

[0227] One example of a domain object database 1072 that can be used in connection with the present invention according to one embodiment is a database of one or more organizations storing, for example, their names and locations. The database can be used, for example, to search the words contained in the input query for matching organizations and/or to look up the location of an organization whose name is known. Those skilled in the art will recognize that many other arrangements and implementations are possible.

Dictionary component(s) 1058

[0228] In at least one embodiment, dictionary component(s) 1058 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Providing databases that associate words and strings with concepts, properties, relationships, or instances of domain models or task models;

- A dictionary of dictionary components can be used by the automated assistant 1002 for several processes including, for example: extracting input, interpreting natural language, and generating output.

[0229] According to specific embodiments, multiple instances or sub-processes of dictionary component(s) 1058 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, various aspects, features, and/or functionality of dictionary component(s) 1058 may be implemented as data structures that associate strings with names of concepts, relationships, properties, and instances. These data structures may be stored in storage, files, or databases. Access to dictionary component(s) 1058 may be through direct APIs, web APIs, and/or database query interfaces. The creation and maintenance of vocabulary component(s) 1058 can be done through direct file editing, database transactions, or domain model editing tools. Dictionary component(s) 1058 may be implemented as part of or in association with active ontologies 1050. Those skilled in the art should recognize that many other arrangements and implementations are possible.

[0230] According to various embodiments, one or more different sub-processes or instances of dictionary component(s) 1058 may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering at least , one instance of the dictionary component(s) 1058 . In one embodiment, dictionary component(s) 1058 are accessed each time dictionary information is requested, including, for example, during input retrieval, input interpretation, and output formatting for users. Those skilled in the art will recognize that other conditions or activities may trigger or begin the implementation of one or more different sub-processes or instances of dictionary component(s) 1058.

[0231] In at least one embodiment, a given dictionary component instance 1058 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. In one embodiment, dictionary component 1058 may access data from external databases, such as a data store or dictionary.

Component(s) 1060 of the language construct recognition module

[0232] In at least one embodiment, the language construct recognition module component 1060 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, searching for constructs in a language or speech input that indicate grammatical, idiomatic, and/or other compound objects of input markers. These constructs correspond, for example, to one or more of the following (or combinations of the above): words, names, phrases, data, parameters, commands, and/or speech act cues.

[0233] According to specific embodiments, multiple instances or sub-processes of component(s) 1060 of the design recognition module may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software . For example, in at least some embodiments, various aspects, features, and/or functionality of the component(s) 1060 of the language construct recognition module may be executed, implemented, and/or initiated by one or more files, databases, and/or programs, containing expressions in the language of matching with the construction. In at least one embodiment, the component(s) 1060 of the language construct recognition module are represented declaratively, and not as program code; this makes it possible to create and maintain them through editors and other tools other than programming tools. Examples of declarative representations may include, but are not limited to, one or more of the following (or combinations of the foregoing): regular expressions, construct matching rules, natural language grammars, finite automaton parsers, and/or other grammar parsing models.

[0234] Those skilled in the art will recognize that other types of systems, components, systems, devices, procedures, processes, and the like. (or combinations of the above) may be used to implement component(s) 1060 of the language construct recognition module.

[0235] According to various embodiments, one or more different sub-processes or instances of the component(s) 1060 of the language construct recognition module can be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering, at least one instance of component(s) 1060 of the language construct recognition module. Various examples of conditions or events that may trigger or begin the implementation of one or more different sub-processes or instances of the language construct recognizer component(s) 1060 may include, but are not limited to, one or more of the following (or combinations of the foregoing):

- in the course of active input extraction, in which the structure of the recognition modules of language constructs can limit and direct input from the user;

- during natural language processing, in which the recognition modules of language constructs help interpret the input as a language;

- in the course of task and dialog identification, in which language construct recognition modules can help identify tasks, dialogs and/or steps.

[0236] In at least one embodiment, a given instance of component(s) 1060 of a language construct recognition module can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that may be accessed by the language construct recognition module component 1060 may include, but are not limited to, data from any of the models, various models, and data sources that may be part of the helper 1002 embodiments that may include, but are not limited to, one or more of the following (or combinations of the foregoing):

- models of 1056 subject areas;

- dictionary 1058;

- database 1072 data objects of the subject area;

- short-term storage device 1052;

- long-term personal storage device 1054;

- task flow models 1086;

- Dialog flow models 1087;

- service characteristics models 1088.

[0237] In one embodiment, data access from other parts of the helper 1002 embodiments can be coordinated through active ontologies 1050.

[0238] Referring again to FIG. 14, an example of some of the different types of functions, operations, actions, and/or other features that can be provided by language construct recognition module component(s) 1060 is shown. 14 illustrates the language constructs that component(s) 1060 of the language construct recognizer can recognize. For example, the idiom "what's going on" (in the city) can be associated with the task of organizing events and the subject area of local events.

Dialog flow processor component(s) 1080

[0239] In at least one embodiment, the dialog flow processor component(s) 1080 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Given the representation of the user's intent 290 from the language interpretation 200, the identification of the task the user requires and/or the problem the user requires to resolve. For example, the task might be to find a restaurant.

- For a given problem or task, given the representation of the user's intent 290, the identification of the parameters for the task or problem. For example, the user may search for a recommended restaurant that serves Italian food near the user's home. Restrictions such as that the restaurant must be recommended, serving Italian food, and being near the house are parameters for the restaurant finding problem.

- Given the interpretation of the task and the current dialogue with the user, for example, which can be represented in the personal short-term memory 1052, the selection of the appropriate dialogue flow model and the definition of the step in the flow model corresponding to the current state.

[0240] According to specific embodiments, multiple instances or subprocesses of the dialog flow processor component(s) 1080 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. .

[0241] In at least one embodiment, a given instance of dialog flow processor component(s) 1080 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by the dialog flow processor component(s) 1080 may include, but are not limited to, one or more of the following (or combinations of the above):

- task flow models 1086;

- models of 1056 subject areas;

- Dialog flow models 1087.

[0242] Referring now to FIGS. 30 and 31, screen shots are shown illustrating an example of various types of functions, operations, actions, and/or other features that may be provided by the dialog flow processor component(s) according to one embodiment.

[0243] As shown in screen 3001, the user requests a dinner reservation by providing voice input or text entry "Reserve me a table for dinner." The assistant 1002 generates a prompt line 3003 asking the user to provide the time and number of party attendees.

[0244] After these parameters are provided, a screen 3101 is shown. The assistant 1002 displays a dialog box 3102 indicating that the results are being presented and a prompt line 3103 asking the user to click the time. Listings 3104 are also displayed.

[0245] In one embodiment, such a dialogue is implemented as follows. The dialog flow processor component(s) 1080 is provided with a representation of the user's intent from the language interpreter component 1070 and determines that the appropriate response is to prompt the user for the information required to perform the next step in the task flow. In this case, the domain is restaurants, the task is to reserve, and the dialog step is to prompt the user for the information required in order to perform the next step in the task flow. This step of the dialog is illustrated by prompt line 3003 on screen 3001.

[0246] Also referring now to FIG. 32, a flowchart is shown illustrating a method of operation for the dialog flow processor component(s) 1080 according to one embodiment. The flowchart of the method of FIG. 32 is described in connection with the example shown in FIGS. 30 and 31.

[0247] The method begins 200. A representation of the user's intent 290 is received. As described in connection with FIG. 28, in one embodiment, the user intent representation 290 is a set of semantic analysis options. For the example shown in FIGS. 30 and 31, the domain is restaurants, the verb is "order" associated with a restaurant reservation, and the time parameter is the evening of the current day.

[0248] At block 310, the dialog flow processor component(s) 1080 determines whether or not this interpretation of user intent is supported strictly enough to continue and/or is supported or not better than alternative ambiguous semantic parsing options. In the current example, the interpretation is strictly supported without competing ambiguous semantic parsing options. On the other hand, if there are competing ambiguities or sufficient ambiguity, then step 322 is executed to set the dialog flow stage such that the execution phase instructs the dialog to output a prompt string to obtain additional information from the user.

[0249] At 312, the dialog flow processor component(s) 1080 determines the preferred interpretation of the semantic analysis with other information to determine the task to be performed and its parameters. Information can be obtained, for example, from subject matter models 1056, task flow models 1086, and/or dialog flow models 1087, or any combination of the above. In the current example, the task is identified as a reservation, which includes both finding a seat that is available for reservation and free, and making a transaction to reserve a table. The task parameters are a time constraint along with others that are inferred at block 312.

[0250] At 320, the task flow model is consulted to determine the appropriate next step. Information can be obtained, for example, from subject matter models 1056, task flow models 1086, and/or dialog flow models 1087, or any combination of the above. In the example, it is determined that in this task flow, the next step is to extract the missing parameters for finding the availability of restaurants, resulting in prompt line 3003, illustrated in Fig. 30, requesting the number of party attendees and the time to reserve.

[0251] As described above, FIG. 31 illustrates a screen 3101 that is shown as including a dialog element 3102 that is presented after the user responds to a query regarding the number of party attendees and the reservation time. In one embodiment, screen 3101 is presented as the result of another iteration through the automated call and response procedure as described in connection with FIG. 33, which results in another call to the dialogs and flows procedure illustrated in FIG. In this instantiation of the conversations and flows procedure, after receiving user preferences, the dialogue flow processor component(s) 1080 determines another stage of the task flow at block 320: search for availability. When request 390 is composed, it includes task parameters sufficient for the dialog flow processor component 1080 and service orchestration component(s) 1082 to be coordinated for a restaurant ordering service.

Component(s) 1087 Dialog Flow Models

[0252] In at least one embodiment, the component(s) 1087 of the dialog flow models can be configured to provide dialog flow models that represent the steps that are taken in a particular type of conversation between the user and the intelligent automated assistant 1002. For example, the flow of dialogs for the generalized task of executing a transaction includes steps for obtaining the necessary data for the transaction and confirming the parameters of the transaction before it is committed.

Component(s) 1086 task flow models

[0253] In at least one embodiment, taskflow model component(s) 1086 can be configured to provide taskflow models that represent steps that are taken to resolve a problem or satisfy a need. For example, the task flow for a dinner reservation involves finding a desired restaurant, checking availability, and executing a transaction to get a reservation for a specific time at the restaurant.

[0254] According to specific embodiments, multiple instances or sub-processes of component(s) 1086 of task flow models can be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software . For example, in at least some embodiments, various aspects, features, and/or functionality of component(s) 1086 of task flow models may be implemented as programs, state machines, or other means of identifying the appropriate step in a flowchart.

[0255] In at least one embodiment, task flow model component(s) 1086 may use a task modeling framework called generic tasks. Generic tasks are abstractions that model the steps in a task and their required inputs and generated outputs without being dependent on subject areas. For example, a general task for transactions may include steps for collecting the data required for a transaction, executing transactions, and outputting the results of the transaction, all without regard to the specific domain of the transaction or the service to implement it. It can be specific to a domain such as shopping, but it is independent of the shopping domain and can be applied equally well to the domains of reservations, scheduling, and the like.

[0256] At least a portion of the functions, operations, actions, and/or other features associated with the component(s) 1086 of the task flow models and/or the procedure(s) described herein can be implemented at least in part, using the concepts, features, components, processes, and/or other aspects disclosed herein in connection with the Generic Task Modeling Framework.

[0257] Additionally, at least a portion of the functions, operations, actions, and/or other features associated with the component(s) 1086 of the task flow models and/or the procedure(s) described herein may be implemented at least at least in part, using concepts, features, components, processes, and/or other aspects associated with constrained selection tasks as described herein. For example, one embodiment of generalized tasks may be implemented using a constrained selection task model.

[0258] In at least one embodiment, a given instance of taskflow model component(s) 1086 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by taskflow model component(s) 1086 may include, but are not limited to, one or more of the following (or combinations of the above):

- models of 1056 subject areas;

- dictionary 1058;

- database 1072 data objects of the subject area;

- short-term storage device 1052;

- long-term personal storage device 1054;

- Dialog flow models 1087;

- service characteristics models 1088.

[0259] Referring now to FIG. 34, a flowchart is shown illustrating an example task flow for constrained selection task 351 according to one embodiment.

[0260] Constrained selection is a kind of generalized problem in which the goal is to select some element from a set of elements in the world based on a set of constraints. For example, constrained selection task 351 may be specific to the restaurant domain. The constrained selection task 351 begins by requesting criteria and constraints from the user 352. For example, the user may be interested in Asian food and may desire a place to eat near their office.

[0261] At step 353, assistant 1002 presents items that meet the established criteria and restrictions for the user to view them. In this example, this could be a list of restaurants and their properties, which could be used to select from.

[0262] At step 354, the user is given the opportunity to refine the criteria and restrictions. For example, the user may qualify the query by saying "near my office". The system must then present a new result set at step 353.

[0263] Also referring now to FIG. 35, an example of a screen 3501 is shown including a list 3502 of items presented by a constrained selection task 351 according to one embodiment.

[0264] At block 355, the user can select from matching items. Any number of further tasks 359 may then be made available, such as, for example, order 356, reminder 357, or share 358. In various embodiments, further tasks 359 may involve interacting with web-based services and/or functionality local to the device (for example, setting up a calendar appointment, making a phone call, sending an email or text message, setting an alarm, etc.).

[0265] In the example of Fig.35, the user can select an item in the list 3502 to see additional information and perform additional actions. Also referring now to FIG. 36, an example screen 3601 is shown after the user has selected an item from the list 3502. Additional information and options corresponding to further tasks 359 regarding the selected item are displayed.

[0266] In various embodiments, the flow steps may be presented to the user in any of several input modalities, including, but not limited to, any combination of explicit dialog prompts and GUI links.

Component(s) 1084 services

[0267] The service component(s) 1084 represents a set of services that the intelligent automated assistant 1002 can invoke on behalf of the user. Any service that can be called can be offered in the service component 1084 .

[0268] In at least one embodiment, service component(s) 1084 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Exposing API functions that would normally be exposed via a web user interface to a service. For example, an overview web site may provide a service API that automatically returns overviews of a given object when called by a program. The API offers services to the intelligent automated assistant 1002 that a person would otherwise need to receive through the user interface of a web site.

- Exposing API functions that would normally be provided through a user interface to an application. For example, a calendar application might provide a service API that automatically returns calendar entries when called by the program. The API offers services to the intelligent automated assistant 1002 that a person would otherwise receive by interacting with the application's user interface. In one embodiment, assistant 1002 has the ability to initiate and control any number of different functions available on the device. For example, if assistant 1002 is installed on a smartphone, personal digital device, tablet computer, or other device, assistant 1002 can perform functions such as: initiating applications, making calls, sending emails and/or text messages, adding calendar events, setting alarms, and etc. In one embodiment, such functions are activated using component(s) 1084 services.

- Providing services that are not currently implemented in the user interface, but which are available through the API, as a helper in large tasks. For example, in one embodiment, an API for receiving a physical address and returning machine-readable geo-coordinates can be used by helper 1002 as services component 1084, even if it does not have a direct web-based user interface or device.

[0269] According to specific embodiments, multiple instances or subprocesses of service component(s) 1084 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, various aspects, features, and/or functionality of the services component 1084 may be performed, implemented, and/or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes, etc. .P. (or combinations of the above):

- the implementation of the API exposed by the service, locally or remotely, or any combination;

- implication of a database in the automated assistant 1002 or a database service available to the assistant 1002.

[0270] For example, a web site that offers users an interface for watching movies can be used by an embodiment of the intelligent automated assistant 1002 as a copy of the database used by the web site. The service component(s) 1084 must then offer an internal API for the data as if it were provided over a network API, even if the data is stored locally.

[0271] As another example, the service component(s) 1084 for an intelligent automated assistant 1002 that assists with restaurant selection and catering events may include any or all of the following set of services that are available from third parties through networks:

- a set of restaurant listing services that list restaurants that match by name, location, or other restrictions;

- a set of restaurant rating services that return a ranking for restaurants with a name;

- a set of restaurant review services that return written reviews for named restaurants;

- geocoding service to find restaurants on the map;

- a reservation service that provides programmable table reservations in restaurants.

Service orchestration component(s) 1082

[0272] The service orchestration component(s) 1082 of the intelligent automated assistant 1002 performs the service orchestration procedure.

[0273] In at least one embodiment, service orchestration component(s) 1082 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Dynamic and automatic determination of which services can satisfy the user's request and/or specified subject area(s) and task(s);

- Dynamic and automatic calling of multiple services, in any combination of parallel and sequential ordering;

- Dynamic and automatic transformation of task parameters and constraints to meet the requirements for the input of API services;

- Dynamic and automatic tracking and collection of results from multiple services;

- Dynamic and automatic integration of data with the results of the provision of services from various services into a unified results model;

- Orchestration by multiple services to satisfy request constraints;

- Orchestration by multiple services to annotate an existing result set with ancillary information;

- Displaying the result of a call to multiple services in a generic, service-independent view that unifies results from different services (e.g., as the result of calling multiple restaurant services that return lists of restaurants, combine data for at least one restaurant from multiple services, removing redundancy).

[0274] For example, in some cases there may be multiple ways to perform a particular task. For example, a user input such as "remind me to leave for a meeting in town at 2:00 pm" indicates an action that can be performed in at least three ways: set an alarm; create a calendar event; or call the activity list manager. In one embodiment, the service orchestration component 1082 makes a determination as to which method is best suited for the request.

[0275] The service orchestration component(s) 1082 can also make determinations as to which combination of multiple services is best activated to perform a given overall task. For example, in order to find and reserve a table for dinner, service orchestration component(s) 1082 must make determinations as to which services to call to perform functions such as looking up reviews, getting availability, and making reservations. Determining which services to use can depend on any number of different factors. For example, in at least one embodiment, information about reliability, the ability of a service to handle certain types of requests, user feedback, and the like. can be used as factors in determining which service(s) to activate.

[0276] According to specific embodiments, multiple instances or subprocesses of service orchestration component(s) 1082 may be simultaneously implemented and/or initiated through the use of one or more processors and/or other combinations of hardware and/or hardware and software.

[0277] In at least one embodiment, a given instance of service orchestration component(s) 1082 may use explicit service characteristic models 1088 to represent characteristics and other properties of external services, and reason about those characteristics and properties when reaching the component attributes( s) 1082 service orchestrations. This provides advantages over manually programming a set of services, which may include, for example, one or more of the following (or combinations of the above):

- ease of development;

- stability and reliability in execution;

- the ability to dynamically add and remove services without damaging the code;

- the ability to implement general algorithms for optimizing distributed queries that are controlled through properties and characteristics, rather than being hard-coded for specific services or APIs.

[0278] In at least one embodiment, a given instance of service orchestration component(s) 1082 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Examples of various types of data that can be accessed by service orchestration component(s) 1082 may include, but are not limited to, one or more of the following (or combinations of the above):

- concretization of models of subject areas;

- options for syntactic and semantic analysis of input in natural language;

- specification of task models (with values for parameters);

- models of the flow of dialogues and tasks and / or selected stages in them;

- service characteristics models 1088;

- any other information available in the active ontology 1050.

[0279] Referring now to FIG. 37, an example of a procedure for performing a service orchestration procedure according to one embodiment is shown.

[0280] In this particular example, it is assumed that one user is interested in finding a good place to dine in a restaurant and contacts the intelligent automated assistant 1002 in the form of a conversation to help provide these services.

[0281] Consider the problem of finding restaurants that are of high quality, have good reviews, are located near a particular location, are available for reservation at a particular time, and offer a particular type of food. These domain and task parameters are provided as input 390.

[0282] The method begins 400. At block 402, it is determined whether or not a given request may require any services. In some cases, delegation to services may not be required, for example, if helper 1002 has the ability to perform the required task on its own. For example, in one embodiment, assistant 1002 may be able to answer the actual question without activating service delegation. Accordingly, if the request does not require services, then the autonomous flow step is executed at step 403 and its result 490 is returned. For example, if the task request is to request information about the automated assistant 1002 itself, then the dialog response may be processed without activating external services.

[0283] If at block 402 it is determined that delegation to services is required, the service orchestration component(s) 1082 proceeds to block 404. At block 404, the service orchestration component(s) 1082 may match task requirements with declarative descriptions of the characteristics and properties of services in service characteristics models 1088. At least one service provider that can support a particular operation provides declarative, qualitative metadata detailing, for example, one or more of the following (or a combination of the above):

- data fields that are returned with the results;

- what parameter classes the service provider (as statically known) supports;

- policy management functions for parameters that a service provider can support after dynamic checking of parameter values;

- a performance rating that specifies how the service works (eg, relational DB, web service, triplicate, full-text index, or some combination of the above);

- property quality ratings that statically set the expected quality of property values returned with the result object;

- a rating of the overall quality of the results that the service can presumably return.

[0284] For example, when considering the parameter classes that a service may support, the service model may indicate that services 1, 2, 3, and 4 may provide restaurants that are located near a particular location (parameter), services 2 and 3 may filter or rank restaurants by quality (another parameter), services 3, 4, and 5 can return reviews for restaurants (a data field is returned), service 6 can list the types of meals offered by restaurants (a data field is returned), and service 7 can check the availability of restaurants in specific time ranges (parameter). Services 8-99 offer characteristics that are not required for this particular subject area and task.

[0285] Using this declarative, qualitative metadata, task, task parameters, and other information available from the helper runtime, service orchestration component(s) 1082 determines 404 the optimal set of service providers to activate. The optimal set of service providers may support one or more task parameters (returning results that satisfy one or more parameters) and also consider the performance rating of at least one service provider and the overall quality rating of at least one service provider.

[0286] The result of step 404 is a dynamically generated list of services that should be called for this particular user and a request.

[0287] In at least one embodiment, service orchestration component(s) 1082 considers the reliability of services as well as their ability to respond to specific information requests.

[0288] In at least one embodiment, service orchestration component(s) 1082 counters unreliability by calling overlapping or redundant services.

[0289] In at least one embodiment, the service orchestration component(s) 1082 considers personal information about the user (from the personal short-term memory component) to select services. For example, a user may prefer some rating services over others.

[0290] At 450, the service orchestration component(s) 1082 dynamically and automatically activates multiple services on behalf of the user. In at least one embodiment, they are called dynamically in response to a user request. In particular embodiments, multiple service instances or sub-processes may be invoked at the same time. In at least one embodiment, they are called over the network using the API, or over the network using the Web Services API, or over the Internet using the Web Services API, or any combination of the foregoing.

[0291] In at least one embodiment, the frequency with which services are called is programmatically limited and/or controlled.

[0292] Also referring now to FIG. 38, an example of a service activation procedure 450 according to one embodiment is shown. Service activation is used, for example, to obtain additional information or perform tasks through the use of external services. In one embodiment, the request parameters are translated appropriately for the service API. After the results are received from the service, the results are converted into a results view for presentation to the user in helper 1002.

[0293] In at least one embodiment, the services activated by the service activation procedure 450 may be a web service, an application running on a device, an operating system function, or the like.

[0294] A request view 390 is provided, including, for example, task parameters and the like. For at least one service available from service characteristic models 1088, service activation procedure 450 performs transform 452, invoke 454, and output transform 456 steps.

[0295] At transform step 452, the current task parameters from the request view 390 are converted to a form that can be used by at least one service. The options for services that can be offered as APIs or databases can be different from the data representation used in task queries, and at least from each other. Accordingly, the goal of step 452 is to convert at least one task parameter to one or more appropriate formats and values in at least one called service.

[0296] For example, the names of organizations, for example, restaurants, may vary according to the services that consider such organizations. Accordingly, step 452 should include converting any names to forms that are best suited for at least one service.

[0297] As another example, locations are known at different levels of accuracy and using different units and conventions according to the services. Service 1 may, may require postal codes, service 2 may require GPS coordinates, and service 3 may require actual postal addresses.

[0298] The service is called 454 by the API and its data is collected. In at least one embodiment, the results are cached. In at least one embodiment, services that do not return within the specified tier performance (eg, as specified in a service level agreement, or SLA) are discarded.

[0299] At step 456 of the output transformation, the data returned by the service is converted back into a unified representation 490 of the result. This step may include consideration of different formats, units, etc.

[0300] At 412, results from multiple services are validated and combined. In one embodiment, if validated results are collected, the equality policy management function (defined for each subject area) is then called in pairs on one or more results to determine which results represent identical concepts in the real world. When a pair of equal results is found, a set of property policy management functions (also defined for each subject area) is used to combine the property values into a combined result. The feature policy function may use the feature quality ratings from the service performance models, task parameters, domain context, and/or long-term personal storage 1054 to determine the optimal aggregation strategy.

[0301] For example, restaurant listings from different restaurant providers can be combined and duplications removed. In at least one embodiment, criteria for identifying duplications may include fuzzy name matching, fuzzy location matching, fuzzy matching with multiple domain object properties such as name, location, telephone number, and/or web site address, and /or any combination of the above.

[0302] At step 414, the results are sorted and trimmed to return a list of results of the desired length.

[0303] In at least one embodiment, a request attenuation loop is also applied. If, at step 416, the service orchestration component(s) 1082 determines that the current list of results is not sufficient (eg, it has fewer than the required number of matches), then the task parameters may be relaxed 420 to provide more results. For example, if the number of restaurants of the required type found within N miles of the target location is too small, then the relaxation should run the query again with a search in an area greater than N miles and/or with some other search parameter weakened.

[0304] In at least one embodiment, the service orchestration method is applied in the second pass to "annotate" the results with ancillary data that is useful to the task.

[0305] At block 418, service orchestration component(s) 1082 determines whether an annotation is required or not. It may be required if, for example, the task may require a plan with map results, but the primary services do not return the geo-coordinates required for the transformation.

[0306] At block 422, the service characteristics model 1088 is again referred to in order to find services that can return the required additional information. In one embodiment, the annotation process determines whether or not additional or better data can be annotated in the combined result. It does this by delegating to the property policy management function (defined for each domain) at least one property of at least one combined result. The property policy management function may use the combined property value and property quality rating, property quality ratings of one or more other service providers, domain context, and/or user profile to determine whether or not better data can be obtained. If it is determined that one or more service providers can annotate one or more properties for a combined result, the cost function is invoked to determine the optimal set of service providers to annotate.

[0307] At least one service provider in the optimal set of annotated service providers is then activated 450 with a list of combined results to obtain results 424. Changes made to at least one combined result by at least one service provider , are tracked during this process, and the changes are then merged using a feature policy management function process identical to that used in step 412. Their results are merged 426 into an existing result set.

[0308] The resulting data is sorted 428 and unified into a universal representation 490.

[0309] It can be appreciated that one advantage of the methods and systems described above with respect to the service orchestration component(s) 1082 is that they can be advantageously applied and/or used in various technical fields other than those that are , in particular, are associated with intelligent automated assistants. Examples of such other arts in which aspects and/or features of service orchestration procedures include, for example, one or more of the following:

- dynamic "blends" in web sites and web applications and services;

- optimization of distributed queries to the database;

- dynamic configuration of service-oriented architecture.

Component(s) 1088 service characteristics models

[0310] In at least one embodiment, the component(s) 1088 of the service characteristics models can be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Providing machine-readable information about the characteristics of services in order to perform certain classes of calculations;

- Providing machine-readable information about the characteristics of services in order to respond to certain classes of requests;

- Providing machine-readable information about what classes of transactions are provided through various services;

- Providing machine-readable parameter information for APIs exposed through various services;

- Providing machine-readable information about parameters that can be used in database queries for databases provided through various services.

Output processor component(s) 1090

[0311] In at least one embodiment, output processor component(s) 1090 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Formatting the output data, which is presented in a universal internal data structure, into forms and layouts that prepare them appropriately for different modalities. The output may include, for example, natural language communication between the intelligent automated assistant and the user; data about objects in the subject area, for example, properties of restaurants, films, products, etc.; results with domain-specific data from information services, such as weather forecasts, flight status checks, prices, etc.; and/or clickable links and buttons that enable the user to respond through direct interaction with the output view.

- Preparing output for modalities, which may include, for example, any combination of the following: graphical user interfaces; text messages; postal messages; sounds; animations; and/or speech output.

- Dynamic data preparation for various display engines based on query based GUI. For example, using different layouts and output processing formats depending on which web browser and/or device is being used.

- Dynamic preparation of output data in various voices in the form of a speech signal.

- Dynamic preparation for specified modalities based on user preferences.

- Dynamic output preparation using user-specific "skins" that customize appearance and functionality.

- Sending output packets to the stream modality showing intermediate status, feedback or results for all phases of interaction with helper 1002.

[0312] According to specific embodiments, multiple instances or subprocesses of output processor component(s) 1090 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, various aspects, features, and/or functionality of output processor component(s) 1090 may be performed, implemented, and/or initiated by one or more of the following types of systems, components, systems, devices, procedures , processes, etc. (or combinations of the above):

- program modules in the client or server of an embodiment of an intelligent automated assistant;

- remotely called services;

- using a combination of templates and procedural code.

[0313] Referring now to FIG. 39, a flowchart illustrating an example of a multi-phase output procedure according to one embodiment is shown. The multi-phase output procedure includes processing steps 702 in the automated assistant 1002 and multi-phase output steps 704.

[0314] In step 710, the speech input is spoken, and a speech-to-text component (eg, the component described in connection with FIG. 22) interprets the speech to generate a set of speech interpretation options 712. In one embodiment, the speech-to-text component is implemented using, for example, a nuance recognition module available from Nuance Communications, Inc., Burlington, MA. The speech interpretation options 712 may be shown to the user at 730, for example, in paraphrased form. For example, the interface might show "Did you say?" alternatives, listing several possible alternative textual interpretations of a single speech sound sample.

[0315] In at least one embodiment, a user interface is provided to allow the user to interrupt and select from speech interpretation options.

[0316] At block 714, speech interpretation options 712 are sent to language interpreter 1070, which can generate user intent 716 representations for at least one speech interpretation option 712. At step 732, paraphrasing these representations, user intentions 716 are formed and presented to the user. (See related step 132 of procedure 120 in FIG. 22).

[0317] In at least one embodiment, the user interface allows the user to interrupt and select from paraphrases of natural language interpretations 732.

[0318] At 718, task and dialog analysis is performed. At block 734, the task and domain interpretations are presented to the user using the intent rephrasing algorithm.

[0319] Also referring now to FIG. 40, a screenshot is shown illustrating an example of output processing according to one embodiment. Screen 4001 includes a reflection 4002 of the user's speech input generated by step 730. Screen 4001 further includes paraphrasing 4003 of the user's intent generated by step 734. In one embodiment, as illustrated in the example of FIG. 40, for keywords, such as "activities", special formatting/highlighting is used that can be used to make it easier to teach the user to interact with the intelligent automated assistant 1002. For example, by visually observing the formatting of the displayed text, the user can easily identify and interpret back that the intelligent the automated assistant recognizes keywords such as "events", "next wednesday", "San Francisco", etc.

[0320] Referring back to FIG. 39, as requests are coordinated 720 for services and results are dynamically collected, intermediate results may be displayed in the form of real-time execution progress 736 . For example, a list of restaurants could be returned, and then their reviews could be dynamically populated as results come in from the review services. Services may include web-based services and/or services that access information stored locally on the device and/or from any other source.

[0321] A universal response representation 722 is generated and formatted 724 for the proper output modality. After the final output formatting is completed, another kind of rephrasing may be offered at block 738. At this phase, the entire result set may be parsed and compared with the initial query. A summary of the results or an answer to a question may then be offered.

[0322] Also referring to FIG. 41, another example of output processing according to one embodiment is shown. Screen 4101 illustrates paraphrase 4102 of text interpretation generated by step 732, real-time progress 4103 generated by step 736, and paraphrased summary 7104 generated by step 738. Detailed results 4105 are also included.

[0323] In one embodiment, assistant 1002 is capable of generating output in multiple modes. Referring now to FIG. 42, a flowchart illustrating an example of multimodal output processing according to one embodiment is shown.

[0324] The method begins at 600. The output processor 1090 receives the universal representation of the response 490 and formats 612 the response according to the device and modality that is appropriate and applicable. Block 612 may include information from device and modality models 610 and/or domain data models 614.

[0325] Once the response 490 is formatted 612, any number of different output mechanisms can be used in any combination. The examples illustrated in Fig. 42 include:

- Formation 620 text message output, which is sent 630 to the text message channel;

- Formation 622 e-mail output, which is sent 632 as a mail message;

- Formation 624 GUI output, which is sent 634 to the device or web browser for preparation;

- Formation 626 speech output, which is sent 636 to the speech generation module.

[0326] Those skilled in the art will recognize that many other inference mechanisms may be used.

[0327] In one embodiment, the content of the output messages generated by the multi-phase output procedure 700 is adapted to the multi-modal output processing mode 600. For example, if the output modality is speech 626, the language used to paraphrase user input 730, text interpretations 732, task and domain interpretations 734, execution progress 736, and/or results summary 738 may be more or less verbose, or use sentences that are easier to detect in sound form than in written form. In one embodiment, the language is adapted in the steps of the polyphase output procedure 700; in other embodiments, the multi-phase inference procedure 700 produces an intermediate result that is further refined into a particular language by multi-modal inference processing 600 .

Component(s) 1052 short-term personal

storage device

[0328] In at least one embodiment, short-term personal storage component 1052 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Storing a history of the recent dialogue between the assistant embodiment and the user, including a history of user inputs and their interpretations;

- Storing a history of recent selections by the user in the GUI, such as which items are open or explored, which phone numbers have been called, which items have been converted, which movie previews have been played, etc.;

- Storing a history of dialogue and user interaction in a database on the client, server in a user-specific session or client session state such as a web browser cookie or RAM used by the client;

- Save a list of recent user requests;

- Saving the sequence of results of recent user queries;

- Saving a history of click sequences of UI events, including button presses, touches, gestures, voice-activated triggers, and/or any other user input.

- Storage of device sensor data (such as location, time, positional orientation, movement, light level, sound level, etc.) that can be correlated with interactions with the assistant.

[0329] According to specific embodiments, multiple instances or sub-processes of short-term personal storage component(s) 1052 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. security.

[0330] According to various embodiments, one or more different sub-processes or instances of the short-term personal storage component(s) 1052 may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering, at least one instance of the short-term personal storage component(s) 1052. For example, the short-term personal storage component(s) 1052 may be activated when there is a user session with an embodiment of assistant 1002 in at least one form of input or user action or response by the system.

[0331] In at least one embodiment, a given instance of short-term personal storage component(s) 1052 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. For example, the short-term personal storage component(s) 1052 may access data from the long-term personal storage component(s) 1054 (eg, to obtain user identities and personal settings) and/or data from the local device regarding time and location. , which can be included in short-term memory entries.

[0332] Referring now to FIGS. 43A and 43B, screen shots are shown illustrating an example of using short-term personal storage component(s) 1052 to maintain conversational context when changing location, according to one embodiment. In this example, the user asked about the local weather and then simply says "in New York". Screen 4301 shows an initial response including local weather. When the user says "in New York", the assistant 1002 uses the short-term personal memory component(s) 1052 to access the conversational context and thereby determine that the current subject area is weather forecasts. This allows helper 1002 to interpret the new pronunciation "in New York" to mean "what's the weather forecast for New York this coming Tuesday?". Screen 4302 shows the correct response, including weather forecasts for New York.

[0333] In the example of Figs. 43A and 43B, not only the input words "Will it rain the day after tomorrow?" are stored in the short-term memory, but also the semantic interpretation of the input system as the weather domain and the time parameter set as the day after tomorrow.

Component(s) 1054 long-term personal

storage device

[0334] In at least one embodiment, the long-term personal storage device component 1054 can be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- Permanent storage of personal information and data about the user, including, for example, his preferences, identification data, authentication credentials, accounts, addresses, etc.;

- Saving the information that the user has collected through the use of the implementation of the assistant 1002, such as the equivalent of bookmarks, favourites, selections, etc.;

- Permanent storage of saved lists of commercial organizations, including restaurants, hotels, shops, theaters and other venues. In one embodiment, the long-term personal storage component(s) 1054 stores more than just names or URLs, and also stores information sufficient to call up a complete list of objects, including phone numbers, locations on a map , photographs, etc.;

- Permanent preservation of saved movies, videos, music, shows and other entertainment elements;

- Permanent saving of personal calendar(s), list(s) of activities, reminders and alerts, contact databases, lists from the user's social networks, etc.;

- Permanent storage of shopping lists and wish lists for products and services, received coupons and discount codes, etc.;

- Permanent storage of history and receipts for transactions, including reservations, purchases, event tickets, etc.

[0335] According to specific embodiments, multiple instances or subprocesses of long-term personal storage component(s) 1054 may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. security. For example, in at least some embodiments, various aspects, features, and/or functionality of the long-term personal storage component(s) 1054 may be performed, implemented, and/or initiated using one or more databases and/or files on ( or associated) clients 1304 and/or servers 1340 and/or resident in data storage devices.

[0336] According to various embodiments, one or more different sub-processes or instances of the long-term personal storage component(s) 1054 may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of minimum threshold criteria for triggering, at least one instance of the long-term personal storage component(s) 1054. Various examples of conditions or events that may trigger or begin the implementation of one or more different subprocesses or instances of persistent personal storage component(s) 1054 may include, but are not limited to, one or more of the following (or combinations of the foregoing):

- Long-term personal storage entries can be discovered as a side effect of user interaction with the assistant 1002 embodiment. Any kind of interaction with the assistant can generate additions to the long-term personal storage, including browsing, searching, finding, shopping, scheduling, acquisition, reservation, data exchange with other people through the assistant.

- Long-term personal storage may also be populated as a result of registering users to an account or service, allowing assistant 1002 to access accounts in other services, using assistant 1002 services on a client device with access to other databases of personal information, such as calendars, lists classes, contact lists, etc.

[0337] In at least one embodiment, a given instance of long-term personal storage component(s) 1054 can access and/or use information from one or more associated databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices, which may be located, for example, on the client(s) 1304 and/or server(s) 1340. Examples of the various types of data that can be accessed by the long-term personal storage component(s) 1054 may include, but are not limited to, data from other personal information databases such as contact lists or lists. friends, calendars, activity lists, other list managers, personal account and wallet managers provided through external 1360 services, and the like.

[0338] Referring now to FIGS. 44A-44C, screen shots are shown illustrating an example of using a long-term personal storage device component(s) 1054, according to one embodiment. In the example, a feature (titled "My Stuff") is provided that includes access to stored objects such as restaurants, movies, and organizations that are found through interactive sessions with the assistant 1002 embodiment. On screen 4401, 44A, the user has found a restaurant. The user touches Save to My Stuff (Save to My Stuff) 4402, which stores information about the restaurant in the component(s) 1054 long-term personal storage device.

[0339] Screen 4403 of Fig. 44B illustrates user access to My Stuff (My Stuff). In one embodiment, the user may select from categories to navigate to the desired item.

[0340] The screen 4404 of Fig. 44C illustrates the category My Restaurant (My restaurants), which includes items previously saved in My Stuff (My materials).

Automated call and answer procedure

[0341] Referring now to FIG. 33, a flowchart is shown illustrating an automatic call and response procedure according to one embodiment. The procedure of FIG. 33 may be implemented in connection with one or more embodiments of the intelligent automated assistant 1002. It may be appreciated that the intelligent automated assistant 1002, as illustrated in FIG. the basis of an intelligent automated assistant that can be implemented. Other embodiments of systems based on an intelligent automated assistant (not shown) may include additional, smaller, and/or different components/features compared to those illustrated, for example, in the exemplary intelligent automated assistant 1002 shown in FIG.

[0342] In at least one embodiment, the automated call and response procedure of FIG. 33 may be configured to perform and/or implement various types of functions, operations, actions, and/or other features, such as, for example, one or more of the following (or a combination of the above):

- The automated call and answer procedure of FIG. 33 may provide an interface control flow loop for the conversational interface between the user and the intelligent automated assistant 1002. At least one iteration of the automated call and answer procedure may serve as a fork in the conversation. The conversational interface is an interface in which the user and assistant 1002 communicate by alternately speaking phrases in a conversational manner.

- The automated call and response procedure of FIG. 33 may provide an execution flow of control for the intelligent automated assistant 1002. In other words, the procedure controls input collection, input processing, output generation, and output presentation to the user.

- The automated call and response procedure of Fig. 33 can coordinate communication between the components of the intelligent automated assistant 1002. In other words, it can direct where the output of one component is fed to another, and where general input from the environment and action on the environment can occur.

[0343] In at least some embodiments, portions of the automated call and response procedure may also be implemented in other devices and/or computer network systems.

[0344] According to specific embodiments, multiple instances or sub-processes of an automated call and response procedure may be simultaneously implemented and/or initiated through the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. In at least one embodiment, one or more or selected portions of an automated call and response procedure may be implemented in one or more client(s) 1304, one or more server(s) 1340, and/or a combination of the above.

[0345] For example, in at least some embodiments, various aspects, features, and/or functionality of an automated call and response procedure may be performed, implemented, and/or initiated by software components, network services, databases, and the like. or any combination of the above.

[0346] According to various embodiments, one or more different sub-processes or instances of an automated call and response procedure may be initiated in response to the detection of one or more conditions or events that satisfy one or more different types of criteria (such as, for example, minimum threshold criteria) to initiate at least one instance of the automated call and response procedure. Examples of various types of conditions or events that may trigger or begin the implementation of one or more different subprocesses or instances of an automated call and response procedure may include, but are not limited to, one or more of the following (or combinations of the foregoing):

• a user session with an instance of the intelligent automated assistant 1002, such as, for example, but not limited to, one or more of the following:

- launching an application for mobile devices, for example, an application for mobile devices, which is an implementation of an embodiment of an intelligent automated assistant 1002;

- launching a computer application, for example, an application that is an implementation of an embodiment of an intelligent automated assistant 1002;

- pressing a special button on a mobile device, for example, "speech input buttons";

- a button on a peripheral device connected to a computer or mobile device, such as a headset, handset or base station, GPS navigation system, household appliance, remote control or any other device with a button that can be associated with the activation of assistance;

- launching a web session from a web browser to a web site implementing intelligent automated assistant 1002;

- starting an interaction from an existing web browser session for a web site implementing the intelligent automated assistant 1002, in which, for example, the service of the intelligent automated assistant 1002 is requested;

- sending a mail (e-mail) message to the modality server 1426, which mediates communication with an embodiment of the intelligent automated assistant 1002;

- sending a text message to the server 1426 modality, which mediates communication with an embodiment of the intelligent automated assistant 1002;

- making a phone call with a modality server 1434 that mediates communication with an embodiment of the intelligent automated assistant 1002;

- sending an event, such as an alert or notification, to an application that provides an embodiment of an intelligent automated assistant 1002.

• when the device that provides the intelligent automated assistant 1002 turns on and/or starts.

[0347] According to various embodiments, one or more different subprocesses or instances of an automated call and response procedure may be initiated and/or implemented manually, automatically, statically, dynamically, simultaneously, and/or combinations of the above. Additionally, various instances and/or embodiments of the automated call and response procedure may be initiated at one or more different time slots (e.g., during a specific time slot, at regular periodic intervals, at irregular periodic intervals, on demand, etc. ).

[0348] In at least one embodiment, a given instance of an automated call and response procedure may use and/or generate various special types of data and/or other types of information when performing specific tasks and/or operations. They may include, for example, input data/information and/or output data/information. For example, in at least one embodiment, at least one instance of an automated call and response procedure may access, process, and/or otherwise use information from one or more different types of sources, such as, for example, one or more databases. In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and/or remote storage devices. Additionally, at least one instance of the automated call and response procedure may generate one or more different types of output data/information, which, for example, may be stored in local storage and/or remote storage devices.

[0349] In at least one embodiment, the initial configuration of a given automated call and response procedure instance may be performed using one or more different types of initialization parameters. In at least one embodiment, at least a portion of the initialization parameters may be accessed via communication with one or more local and/or remote storage devices. In at least one embodiment, at least a portion of the initialization parameters provided to the automated call and response procedure instance may correspond to and/or may be derived from input data/information.

[0350] In the specific example of FIG. 33, it is assumed that one user is accessing an instance of the intelligent automated assistant 1002 over the network from a client application with speech input characteristics. The user is interested in finding a good place to dine in a restaurant and contacts the intelligent automated assistant 1002 in the form of a conversation to help provide these services.

[0351] At step 100, the user is prompted to enter a query. The client user interface offers several input modes, as described in connection with FIG. They may include, for example:

- an interface for typing input, which can activate the active retrieval of typing input procedure, as illustrated in Fig.11;

- an interface for speech input, which can activate the procedure for active extraction of speech input, as illustrated in Fig.22.

- an interface for selecting inputs from a menu, which can activate active GUI-based input retrieval, as illustrated in FIG.

[0352] Those skilled in the art will recognize that other input modes may be provided.

[0353] In one embodiment, step 100 may include presenting options left over from a previous conversation with assistant 1002, for example, using the techniques described in the dialog recommendation input active retrieval procedure described in connection with FIG.

[0354] For example, through one of the active input retrieval methods at step 100, the user can tell helper 1002: "Where can I find good Italian food in the area?" For example, the user might say this into a speech input component. An embodiment of the active input extraction component 1094 invokes the speech-to-text service, prompts the user for confirmation, and then presents the confirmed user input as a generic annotated input format 2690.

[0355] An embodiment of the language interpreter component 1070 is then invoked at step 200, as described in connection with FIG. The language interpreter component 1070 parses the text input grammatically and generates a list of possible interpretations of the user's intent 290. In one grammatical analysis, the word "Italian" is associated with Italian-style restaurants; "good" is associated with the restaurant recommendation feature; and "in the vicinity" is associated with a position parameter describing a distance from a global sensor reading (eg, the user's location provided via GPS on a mobile device).

[0356] At 300, a representation of the user's intent 290 is passed to the conversation flow processor 1080, which implements an embodiment of the conversation and flow parsing procedure as described in connection with FIG. The conversation flow processor 1080 determines which interpretation of the intent is most likely, converts that interpretation into domain model instances and task model parameters, and determines the next flow step in the conversation flow. In the current example, the restaurant domain model is instantiated with a constrained selection problem to find a restaurant through constraints (cuisine style, recommendation level, and proximity constraints). The conversation flow model indicates that the next step is to get a few examples of restaurants that meet these constraints and present them to the user.

[0357] At 400, an embodiment of the flow and service orchestration procedure 400 is activated via the service orchestration component 1082. It activates service set 1084 on behalf of a user request to find a restaurant. In one embodiment, these services 1084 contribute some data to the overall result. Their data is combined and the resulting list of restaurants is presented in a generic, service-independent form.

[0358] At step 500, the output processor 1092 generates a summary of the results dialog, such as: "I found some recommended Italian restaurants nearby." The output processor 1092 combines this summary with the output result data and then sends the combination to a module that formats the output for the user's particular mobile device in step 600.

[0359] At step 700, this device-specific output packet is sent to the mobile device, and the client software in the device renders it on the screen (or other output device) of the mobile device.

[0360] The user views this view and decides to explore the various options. If the user has completed 790, the method ends. If the user has not completed 490, another loop iteration is initiated by returning to step 100.

[0361] The automatic call and answer procedure may be applied, for example, to the user's request "how about Mexican food?". Such an input can be retrieved at step 100. At step 200, the input is interpreted as Mexican-style restaurants and combined with another state (stored in short-term personal storage 1052) to maintain the same intent interpretation as last time, with one change in the restaurant style parameter. At block 300, this updated intent generates a request refinement, which is provided to the service orchestration component(s) 1082 at block 400.

[0362] At step 400, the updated request is coordinated across multiple services 1084, resulting in a new set of restaurants that is summarized in a dialog at step 500, formatted for the device at 600, and sent over the network to display the new information on the user's mobile device at step 700.

[0363] In this case, the user finds a restaurant they like, shows it on a map, and sends directions to a friend.

[0364] Those skilled in the art will recognize that various embodiments of the automated call and response procedure (not shown) may include additional features and/or operations compared to what is illustrated in the specific embodiment of FIG. 33, and/or may omit at least a portion of the features and/or operations of the automated call and response procedure illustrated in the specific embodiment of FIG.

Choice with restrictions

[0365] In one embodiment, the intelligent automated assistant 1002 uses constrained selection in its user interactions to more effectively identify and present items that are likely to be of interest to the user.

[0366] Constrained selection is a kind of generalized problem. Generalized tasks are abstractions that characterize the types of domain objects, inputs, outputs, and control flow that are common to a class of tasks. The constrained selection task is performed by selecting items from a set of domain object selections (eg, restaurants) based on selection constraints (eg, desired cuisine or location). In one embodiment, assistant 1002 assists the user in exploring the choice space, extracting the user's constraints and preferences, presenting the choices, and recommending actions to take on those choices, such as reserve, purchase, remember, or share them. The task is completed when the user selects one or more items on which to perform an action.

[0367] Constrained choice is useful in many contexts: for example, choosing a movie to watch, a restaurant for dinner, a hotel for the night, a place to buy a book, and the like. In general, constrained selection is useful when a category is known and it is desired to select an instance of a category with some required properties.

[0368] One traditional approach to restricted selection is a directory service. The user selects a category and the system offers a list of choices. In a local directory, you can restrict the directory to a location, such as a city. For example, in a yellow pages service, users select a book for a city and then search for a category, and the book displays one or more items for that category. The main problem with using a directory service is that the number of possibly relevant choices is large (eg, restaurants in a given city).

[0369] Another conventional approach is a database management application that provides a way to generate a set of choices by extracting a request from a user, extracting matching items, and presenting the items in some way that highlights features. The user walks through the rows and columns of the result set, possibly sorting the results or modifying the query, until the user finds multiple matches. A problem with using a database service is that it may require the user to operationalize their human need as a formal query and use abstract sorting, filtering, and viewing machines to examine the resulting data. This is difficult for most people, even with graphical user interfaces.

[0370] A third traditional approach is open search, eg "local search". The search is easy to perform, but there are several problems with search services that make it difficult for people to perform the task of selecting with restrictions. In particular:

- Similar to a directory search, the user cannot simply enter a category and view one or more possible choices, but must narrow down the list.

- If the user can narrow down the selection through restrictions, it is not clear what restrictions can be used (e.g. "can I look for places that are within walking distance or open late?")

- It's not clear how to specify state constraints (e.g. is a cuisine or restaurant type specified and what are the possible values?)

- Conflict of several preferences; there is usually no objectively "best" answer for this case (for example, I want a place that is close and cheap, offers gourmet food with excellent service, and is open until midnight).

- Preferences are relative and they depend on what is available. For example, if a user can seek a table in a restaurant with a very high rating, he can choose it even if it is expensive. However, in general, the user prefers the less expensive options.

[0371] In various embodiments, the assistant 1002 of the present invention helps to simplify the task of constrained selection. In various embodiments, assistant 1002 uses database and search services, as well as other functionality, to reduce the effort on the part of the user to express what they are looking for, given what is available, and select a satisfactory solution.

[0372] In various embodiments, assistant 1002 helps facilitate restricted selection for humans in any of a number of different ways.

[0373] For example, in one embodiment, assistant 1002 may operationalize properties into constraints. The user specifies what he wants in terms of properties of the desired result. Assistant 1002 operationalizes this input into formal constraints. For example, instead of saying "find one or more restaurants less than 2 miles from downtown Palo Alto that serve Italian food", the user could simply say "Italian restaurants in Palo Alto". Helper 1002 may also operationalize user-requested qualities that are not parameters to the database. For example, if a user requests romantic restaurants, the system can operationalize this as a text search or a tag match constraint. Thus, assistant 1002 helps to overcome some of the problems that users may otherwise have during restricted selection. It is easier for the user to imagine and describe a satisfactory solution than to describe the conditions that should distinguish suitable solutions from unsuitable solutions.

[0374] In one embodiment, assistant 1002 may suggest useful selection criteria, and the user need only report which criteria are currently important. For example, helper 1002 may ask: "Which of these matters: price (cheaper is better), location (closer is better), rating (higher rated is better)?" Helper 1002 may also recommend criteria that may require specific values; for example: "You can say which cuisine you prefer or which dish you prefer."

[0375] In one embodiment, assistant 1002 can assist a user in deciding between choices that diverge on a number of competing criteria (such as price, quality, availability, and convenience).

[0376] By providing such guidance, assistant 1002 can assist users in making multivariable decisions in any of several ways:

- One is to reduce the dimension of the space by combining raw data, eg ratings from multiple sources, into a composite "recommendations" score. A composite score may take into account domain knowledge of data sources (for example, Zagat ratings may provide better quality predictions than Yelp).

- Another approach is to focus on a subset of criteria, turning the problem "what are all the possible criteria to consider and how do they combine?" in choosing the most important criteria in this case (for example, "which one is more important, price or proximity?").

- Another way to simplify decision making is to allow for defaults and orders of preference (e.g. all entities are equal, higher ranked, and closer and cheaper is better). The system can also remember previous user responses that indicate their defaults and preferences.

- Fourth, the system may suggest characteristic properties of items in the selection set that are not mentioned in the original query. For example, the user may be requesting local Italian food. The system may offer a set of restaurant choices with and with them, a list of popular tags used by reviewers, or a key phrase from a guidebook (eg, "nice place to date" "great pasta"). This may allow people to select a specific item and complete a task. Research shows that most people make decisions by evaluating specific instances instead of selecting criteria and rationally confirming which one comes first. They also show what people learn about the symptoms from specific cases. For example, when choosing cars, buyers may not pay attention to navigation systems until they see that some cars have them (and subsequently the navigation system can become an important criterion). Helper 1002 can represent characteristic properties of the listed elements that help people choose a winner or that recommend a dimension along which to optimize.

Conceptual data model

[0377] In one embodiment, assistant 1002 offers assistance with a constrained selection task by simplifying the conceptual data model. The conceptual data model is an abstraction presented to users in the interface of helper 1002. To overcome the psychological problems described above, in one embodiment, helper 1002 provides a model that allows users to describe what they want in terms of a few easily recognizable and rememberable properties. appropriate choices, not constraint expressions. Thus, properties can be simple to compose in natural language queries (eg adjectives modifying keyword markers) and can be recognized in prompt strings ("You may also prefer recommended restaurants..."). In one embodiment, a data model is used that allows the helper 1002 to define the domain of interest (eg, restaurants versus hotels) and a general approach for specifying what can be specified with domain-specific properties.

[0378] In one embodiment, the conceptual data model used by helper 1002 includes a selection class. It is a representation of the space of entities from which to choose. For example, in a "find a restaurant" application, the selection class is a restaurant class. The choice class can be abstract and have subclasses such as "occupations at the destination". In one embodiment, the conceptual data model assumes that in a given problem solving situation, the user is interested in choosing from one choice class. This assumption simplifies the interaction and also allows helper 1002 to declare its boundaries of competence ("I know about restaurants, hotels and movies" as opposed to "I know about city life").

[0379] Considering the selection class, in one embodiment, the data model presented to the user for a constrained selection task includes, for example: elements; signs of elements; Criterias of choice; and restrictions.

[0380] Elements are instances of the selection class.

[0381] Element features are properties, attributes, or computed values that can be represented and/or associated with at least one element. For example, the name and phone number of a restaurant are feature elements. Features can be internal (the name or cuisine of a restaurant) or relational (such as distance from the current location of interest). They can be static (like the name of a restaurant) or dynamic (like a rating). They may be composite values calculated from other data (for example, a "value for money" score). Element attributes are abstractions to the user, performed by the domain modeler; they do not have to match the underlying data from the internal front-end services.

[0382] Selection criteria are features of elements that can be used to compare the value or relevance of elements. In other words, they are ways of communicating which elements are preferred. The selection criteria are modeled as features of the elements themselves, whether they are internal properties or computed. For example, proximity (defined as distance from the location of interest) is a selection criterion. Location in space and time is a property, not a selection criterion, and is used in conjunction with the location of interest to calculate the distance from the location of interest.

[0383] The selection criteria may have an inherent order of preference. In other words, the values of any particular criterion can be used to arrange the elements in order according to the best fit criterion. For example, the closeness criterion has such an inherent preference that closer is better. Location, on the other hand, has no inherent preference value. This restriction allows the system to make default assumptions and guide selections if the user only mentions a criterion. For example, the user interface might suggest "sort by rating" and assume that higher rating is better.

[0384] One or more selection criteria are also feature elements; these are signs associated with the choice of possible elements. However, item features are not necessarily associated with preference (for example, restaurant names and phone numbers are usually irrelevant for selection).

[0385] In at least one embodiment, the constraints are constraints on the desired selection criteria values. Formally, constraints can be represented as set membership (e.g. cuisine type includes Italian), construct matches (e.g. restaurant review text includes "romantic"), fuzzy inequalities (e.g. distance less than a few miles), quality thresholds values (for example, highly rated) or more advanced features (for example, good value for money). In order to simplify things sufficiently for ordinary people, this data model reduces at least one or more restrictions to character values that can be matched as words. Time and distance may be excluded from this reduction. In one embodiment, the operators and thresholds used to enforce the restrictions are hidden from the user. For example, a selection constraint called "cuisine" can be represented as a character value, such as "Italian" or "Chinese". The rating limit is "recommended" (binary choice). For time and distance, in one embodiment, assistant 1002 uses custom representations that handle a range of inputs and constraint values. For example, the distance may be "walking distance" and the time may be "tonight"; in one embodiment, assistant 1002 uses special processing to match such input with more accurate data.

[0386] In at least one embodiment, some restrictions may be mandatory restrictions. This means that the task simply cannot be completed without this data. For example, it is difficult to select a restaurant without some notion of the desired location, even if the name is known.

[0387] To summarize, the subject area is modeled as selection classes with feature elements that are important to users. Some functions are used to select and order items that are offered to the user - these characteristics are called selection criteria. Constraints are symbolic constraints on selection criteria that narrow down a set of elements to elements that match.

[0388] Often, several criteria may compete, and restrictions may overlap. The data model reduces the selection problem from an optimization problem (finding the best solution) to a matching problem (finding elements that fit well with a set of given criteria and match a set of symbolic constraints). Algorithms for selecting criteria and constraints and determining ordering are described in the next section.

Technology for choice with limitations

[0389] In one embodiment, assistant 1002 performs constrained selection by considering as input an ordered list of criteria with implicit or explicit constraints on at least one and generating a set of feature element options. Computationally, the selection problem can be characterized as a nested search: first, identifying the selection class, then identifying the important selection criteria, then specifying constraints (boundaries of feasible decisions), and performing searches in instances in order of best approximation to find acceptable elements.

[0390] Referring now to FIG. 45, an example of an abstract model 4500 for a constrained selection task as a nested search is shown. In the example, helper 1002 identifies 4505 a select call for all types of local searches 4501. The identified class is a restaurant. In the set of all restaurants 4502, assistant 1002 selects 4506 criteria. In the example, the criterion is identified as a distance. In the set of restaurants in PA 4503, helper 1002 specifies 4507 search restrictions. In the example, the identified constraint is "Italian cuisine". In the set of Italian restaurants in PA 4504, assistant 4508 selects items to present to the user.

[0391] In one embodiment, such a nested lookup is what helper 1002 performs when it has relevant input, rather than a thread for extracting data and presenting results. In one embodiment, such control flow is controlled through a dialog between the assistant 1002 and the user, which operates through other procedures such as dialog and task flow models. Constrained Choice offers a framework for laying out dialog and task flow models at this level of abstraction (i.e., suitable for constrained choice problems regardless of the domain).

[0392] Referring now to FIG. 46, an example dialog 4600 is shown to help guide the user through the search process so that relevant input can be obtained.

[0393] In the exemplary dialog 4600, the first step is for the user to indicate what they are looking for, i. choice class. For example, the user may do this by saying "dinner in Palo Alto". This enables the assistant 1002 to infer 4601 the task and subject area.

[0394] Once the assistant 1002 has understood the task and domain association (selection class=restaurants), the next step is to understand which selection criteria are important to that user, for example, by querying 4603 criteria and/or constraints . In the example above, "in Palo Alto" indicates the location of interest. In the context of restaurants, the system may interpret location as a proximity constraint (technically, proximity criteria constraints). Assistant 1002 explains what is needed, accepts input. If enough information exists to limit the set of choices to a reasonable size, then helper 1002 paraphrases the input and presents 4605 one or more restaurants that satisfy the proximity constraint, sorted in some useful order. The user can then select 4607 from this list or refine 4606 the criteria and constraints. Helper 1002 reasons about the constraints already indicated and uses domain-specific knowledge to recommend other criteria that may help, also requesting constraints on those criteria. For example, assistant 1002 may reason that when recommending restaurants within walking distance of a hotel, table availability and cuisine should be useful criteria for the query.

[0395] The constrained selection task is completed when the user selects 4607 an instance of the selection class. In one embodiment, additional further tasks 4602 are provided by helper 1002. Thus, helper 1002 can offer services that indicate a choice while providing some other value. Examples 4608 are ordering a restaurant, setting a calendar reminder, and/or sharing a selection with others by sending an invitation. For example, a restaurant order unambiguously indicates that it is selected; other options might be to put the restaurant on your calendar or send it in an invitation with directions to friends.

[0396] Referring now to FIG. 47, a flowchart illustrating a restricted selection method according to one embodiment is shown. In one embodiment, assistant 1002 operates in an adaptive manner with an intermittent initiative, allowing the user to navigate to the inner loop, for example, by specifying a task, domain, criteria, and constraints one or more at a time in the input.

[0397] The method begins 4701. Input is received 4702 from the user according to any of the modes described herein. If, based on the input, the task is unknown, the assistant 1002 requests 4705 to clarify the input from the user.

[0398] At step 4717, assistant 1002 determines whether or not the user provides additional input. If so, assistant 1002 returns to block 4702. Otherwise, the method ends 4799.

[0399] If at step 4703 the task is known, assistant 1002 determines 4704 whether or not the restricted selection is a task. If not, assistant 1002 proceeds 4706 to the specified task thread.

[0400] If at block 4704 the task is restricted selection, assistant 1002 determines 4707 whether or not a selection class can be determined. If not, helper 1002 proposes 4708 a selection of known selection classes and returns to block 4717.

[0401] If at step 4707 the selection class can be determined, assistant 1002 determines 4709 whether or not all mandatory constraints can be determined. If not, assistant 1002 prompts 4710 for the requested information and returns to block 4717.

[0402] If at step 4709 all required constants can be determined, helper 1002 determines 4711 whether or not any elements of the results can be found, subject to constraints. If there are no elements that satisfy the constraints, helper 1002 suggests 4712 ways to relax the constraints. For example, helper 1002 can relax the lowest to highest priority constraints using a filter/sort algorithm. In one embodiment, if there are items that satisfy some constraints, then helper 1002 can rephrase the situation (output, for example: "I can't find any recommended Greek restaurants that deliver on Sundays in San Carlos. However, I found 3 Greek restaurants and 7 recommended restaurants in San Carlos".). In one embodiment, if there are no items that match any of the restrictions, then helper 1002 can rephrase the situation and prompt for various restrictions (output such as "Sorry, I can't find restaurants in Anytown, Texas. You can choose other location".). Assistant 1002 returns to block 4717.

[0403] If at step 4711 the elements of the result can be found, the helper 1002 offers 4713 a list of elements. In one embodiment, assistant 1002 rephrases the currently defined criteria and constraints (output, for example, "Here are some recommended Italian restaurants in San Jose." (recommended=yes, cuisine=Italian, proximity=<in San Jose>)). In one embodiment, helper 1002 presents a sorted, paginated list of items that satisfy known constraints. If an element only shows some restrictions, such a condition may be shown as part of the element display. In one embodiment, assistant 1002 prompts the user for ways to select an item, such as by initiating another task for that item, such as ordering, reminding, scheduling, or sharing. In one embodiment, for any given element, helper 1002 presents element features that are specific to a selection of instances of the selection class. In one embodiment, helper 1002 shows how the element satisfies the constraint; for example, a Zagat rating of 5 satisfies the "recommended=yes" constraint, while "1 mile away" satisfies the "walking distance from the address" constraint. In one embodiment, the assistant 1002 allows the user to drill down for more details on an element, resulting in more element features being displayed.

[0404] Helper 1002 determines 4714 whether or not the user selects an item. If the user selects an item, the task is complete. Any further task is performed 4715, if any, and the method ends 4799.

[0405] If at step 4714 the user does not select an item, assistant 1002 suggests 4716 to the user ways to select other criteria and constraints and returns to step 4717. For example, given the currently specified criteria and constraints, assistant 1002 may suggest criteria that are most likely to limit the set of choices to the required size. If the user selects a constraint value, then the constraint value is added to the previously determined constraints when steps 4703-4713 are repeated.

[0406] Since one or more criteria may have an inherent preference value, the selection of criteria may add information to the query. For example, allowing the user to indicate that positive reviews are appreciated allows helper 1002 to sort by that criterion. This information may be taken into account when steps 4703-4713 are repeated.

[0407] In one embodiment, helper 1002 allows the user to boost a criterion that is already specified so that it should have a higher order of precedence. For example, if the user is requesting the fastest, cheapest, most recommended restaurants within one block of their location, assistant 1002 may prompt the user to select which of these criteria is more important. This information may be taken into account when steps 4703-4713 are repeated.

[0408] In one embodiment, the user may provide additional input at any time while the method of FIG. 47 is being performed. In one embodiment, assistant 1002 checks periodically or continuously for such input and, in response, jumps back to block 4703 to process it.

[0409] In one embodiment, when displaying an element or a list of elements, helper 1002 indicates in the element view the features that are used to select and order them. For example, if the user queries for nearby Italian restaurants, such item attributes for distance and cuisine can be shown in the item view. This may include hit highlighting as well as a list of selection criteria that are involved in the element representation.

Sample subject areas

[0410] Table 1 provides an example of restricted selection subject areas that can be handled by helper 1002 according to various embodiments.

Table 1 Choice Based on these criteria Location Price Availability Type of Duality Name Services Special Search General search Restaurant Proximity Affordability Free tables kitchen rating by guide, review Restaurant name delivery menu items Keywords hotel Proximity Price range Available rooms motel, hotel, B&B, ... rating by guide, review The name of the hotel comfort Keywords movie The proximity of the cinema Session Schedule genre review rating Movie title actors, etc. local organization Proximity Organization category review rating Name of the organization Keywords Local event Proximity to event venue by date event title Keywords concert Proximity to event venue according to tour schedule music style Group name Members of the group Keywords CD, books, DVD for purchase price range online, in store, etc. Loading, physical Popularity Album or song title Artist, title, etc. Keywords

Filtering and sorting results

[0411] In one embodiment, when presenting items that satisfy the currently defined criteria and constraints, a filtering/sorting technique may be used. In one embodiment, the selection constraints may serve as filtering and sorting options for the underlying services. Thus, any selection criteria can be used to determine which items are in the list and calculate the order in which to paginate and display them. The sort order for this task is akin to search relevance rank. For example, proximity is a criterion with symbolic constraint values, such as "driving distance" and the general concept of sorting by distance. The "distance that can be driven by car" constraint can be used to select a group of feature options. Within this group, closer elements may be sorted higher in the list.

[0412] In one embodiment, the selection constraints and associated filtering and sorting are at discrete "levels" that are functions of both underlying data and user input. For example, proximity is grouped into levels such as "walking distance", "taxi driving distance", "driving distance". When sorting, one or more items within walking distance are treated as if they are the same distance. Input from the user can take effect in the way that he specifies the constraint. If the user enters, for example, "in Palo Alto", then one or more elements within the boundaries of the city of Palo Alto are perfect matches and are equivalent. If the user enters "near the train station on University Avenue", then the match should depend on the distance from that address, with the degree of match depending on the selection class (for example, "about" for restaurants is different from "about" for hotels). Even within the limit, which may be specified with a continuous value, discreteness may apply. This may be important for sorting operations, so that several criteria may be involved in determining the best fit ordering.

[0413] In one embodiment, the list of elements (elements that are considered "matching" or "good enough") may be less than or greater than the number of elements shown on one "page" of output. In general, the elements on the first page receive the most attention, but conceptually there is a longer list, and pagination is simply a function of the form factor of the output medium. This means, for example, that if the user is offered a way to sort or view items by some criterion, then it is the full set of items (multiple pages) that is sorted or viewed.

[0414] In one embodiment, the selection criteria are ordered by priority. In other words, some criteria may matter more than others when filtering and sorting. In one embodiment, these user-selected criteria are given higher priority than others and are ordered by default by one or more criteria. This provides a general lexicographic sorting capability. The assumption is that there is a meaningful a priori priority. For example, unless the user specifies otherwise, it may be more important for a restaurant to be closer than it is to be inexpensive. In one embodiment, ordering by a priori priority is domain dependent. The model provides the ability for user-specific preferences to override domain defaults if desired.

[0415] Because constraint values can represent multiple internal data types, there are various ways to negotiate constraints, and they can be specific to a constraint. For example, in one embodiment:

- Binary constraints match as one or more or none. For example, whether a restaurant has the type "fast food" may or may not be true.

- Set membership constraints match as one or more or none based on the value of the property. For example, cuisine=Greek means that the set of cuisines for the restaurant includes Greek.

- Enumeration limits match at threshold. For example, a rating criterion can have constraint values of "rated", "highly rated", or "highest rated". Restriction as "highly rated" is also the same as "highest rated".

- Numeric limits match at a threshold value, which can be a specific criterion. For example, "open late" can be a criterion, and the user can request places that are open after 22:00. This kind of constraint may be a little out of the scope of the constrained selection problem, since it is not a character value of the constraint. However, in one embodiment, helper 1002 recognizes multiple instances of numeric restrictions, such as these, and converts them to character-limited thresholds (e.g., "restaurants in Palo Alto are open now"->"here are 2 restaurants in Palo -Alto, which are open until late").

- Location and time is handled in a special way. The proximity constraint may be a location of interest specified in some granularity that determines a match. If the user specifies a city, then a city-level match is appropriate; a zip code may provide a radius. Assistant 1002 can also understand locations that are "near" other locations of interest, also based on special processing. Time is relevant as a constraint value of criteria that are thresholded based on a service call, such as table availability or flights within a given time range.

[0416] In one embodiment, constraints may be modeled such that there is one threshold value for selection and a small set of discrete values for sorting. For example, an affordability criterion can be modeled as a roughly binary constraint if affordable restaurants are all restaurants below some threshold price range. When the data confirms multiple discrete levels to choose from, constraints can be modeled using a matching gradient. In one embodiment, two levels of matching (eg, strong and loose matching) may be provided; however, those skilled in the art will recognize that, in other embodiments, any number of levels of conformance may be provided. For example, proximity may be consistent with a fuzzy boundary, such that entities that are near the location of interest may not be strictly aligned. The practical consequence of a strict or non-strict match is the filter/sort algorithm, as described below.

[0417] For at least one criterion, a match approach and default thresholds may be set, if relevant. The user may be able to simply speak the name of the constraint, the character value of the constraint, or the exact expression of the constraint if it is handled specifically (eg, time and location).

[0418] An ideal situation for constrained selection occurs when the user specifies constraints that result in a short list of choices, one or more of which satisfies the constraints. The user then selects from the winners based on the features of the items. In many cases, however, the problem is overly or not sufficiently constrained. When it is overly constrained, there are few or no elements that satisfy the constraints. When it is not sufficiently constrained, there are so many options that list analysis is not worthwhile. In one embodiment, the general constrained selection model of the present invention has the ability to handle multiple constraints with a strong match, and typically generate something from which to select. The user may then choose to refine their criteria and constraints, or simply perform the task with a "good enough" solution.

Way

[0419] In one embodiment, the following method is used to filter and sort the results:

1. Given an ordered list of selection criteria selected by the user, define constraints on at least one.

a. If the user specified a limit value, use it. For example, if the user said "Greek food", the constraint is cuisine=Greek. If the user said "San Francisco", the restriction is "In the City of San Francisco". If the user said "south of the market", then the restriction is "In the vicinity of SoMa".

b. Otherwise, use the domain and criteria dependent default. For example, if the user said "a table in some Thai establishment", then he indicates that the availability criterion is relevant, but he did not indicate the value of the restriction. The default constraint values for availability can be a range of times and dates, such as tonight and the default number of party attendees of 2.

2. Selecting the minimum of N results through the specified constraints.

a. Trying to get N results in a strict match.

b. If it fails, attempt to relax constraints in reverse order of precedence. In other words, a strict match for one or more criteria, except for the last one, which may match at a non-strict level. If there is no non-strict match for this constraint, then attempt to use non-strict matches up the line from lowest to highest priority.

c. Then iteration of the loop, resolving the error when the constraints match, from the lowest to the highest priority.

3. After receiving the minimum set of choices, lexicographic sorting by one or more criteria (which may include user-specified criteria as well as other criteria) in order of priority.

a. Considering a set of user-specified criteria as the highest priority, then one or more remaining criteria in their a priori priority. For example, if the prior priority is (accessibility, cuisine, proximity, rating) and the user gives restrictions on proximity and cuisine, then the sort priority is (cuisine, proximity, accessibility, rating).

b. Sorting for criteria using discrete matching levels (strong, loose, none) using an identical approach to constraint relaxation, this time using the full list of criteria.

i. If a set of choices is obtained without relaxing the constraints, then one or more of the set of choices can be "linked" in the sort because they represent one or more matches at strict levels. Next, the next criteria in the priority list may come into effect to sort them. For example, if the user says cuisine = Italian, proximity = In San Francisco, and the sort priority is (cuisine, proximity, availability, rating), then one or more places in the list have equal match values for cuisine and proximity. Thus, the list should be sorted by availability (seats with available tables appear first). Within the available seats, the highest ranked seats should be at the top.

ii. If a set of choices is obtained by loosening constraints, then one or more fully matching elements are placed at the top of the list, followed by partially matching elements. In the matching group, they are sorted by the remaining criteria, as in the partial matching group. For example, if there are only two Italian restaurants in San Francisco, then the available one is shown first and then the unavailable one. After that, the remaining restaurants in San Francisco are shown, sorted by availability and rating.

Order by priority

[0420] The technologies described herein provide helper 1002 with super-reliability despite partially stated limitations and incomplete data. In one embodiment, helper 1002 uses these technologies to generate a custom list of items in best fit order, i. according to relevance.

[0421] In one embodiment, such relevancy sorting is based on ordering by a priori priority. In other words, from the entities that are important in relation to the subject area, a set of criteria is selected and placed in order of importance. If one or more entities match, criteria higher in order of precedence may be more relevant for constrained selection from elements than those lower in order. Assistant 1002 may operate based on any number of criteria. In addition, the criteria can be modified over time without violating existing behaviors.

[0422] In one embodiment, the order of precedence for the criteria may be configured with domain-specific parameters, since the way in which the criteria interact may depend on the selection class. For example, when choosing hotels, availability and price may be the dominant constraints, while for restaurants, food and proximity may be more important.

[0423] In one embodiment, the user can override the default criteria ordering in the dialog. This allows the system to guide the user when the search is overly restrictive by using ordering to determine which restrictions should be relaxed. For example, if the user sets restrictions on cuisine, proximity, recommendation, and dish, and there are no exact matches, the user can say that the dish is more important than the recommendation level, and change the combination so that desired dish matches are sorted first.

[0424] In one embodiment, when the order of precedence is determined, user-specified restrictions take precedence over others. For example, in one embodiment, proximity is a mandatory constraint and thus is always specified and additionally takes precedence over other unselected constraints. Therefore, it is not necessary to be the highest priority constraint for strong dominance. In addition, many criteria may not match as one or more if the constraint is not specified by the user, and thus the precedence of these criteria only matters in user-selected criteria. For example, when a user specifies a kitchen, it is important to them and otherwise irrelevant for sorting items.

[0425] For example, the following is a paradigm for sorting by priority options for the restaurant domain:

1. kitchen* (unsortable if limit value is not set),

2. Availability* (sortable using a default constraint value such as time)

3. recommended

4. proximity* (restriction value is always set),

5. affordability

6. delivery capability

7. dish (not sortable if the constraint value, e.g. keyword, is not set),

8. keywords (unsortable if the constraint value, e.g. keyword is not set),

9. restaurant name

[0426] The following is an example schema explanation for the above sorting paradigm:

- If the user specifies a kitchen, he wants to remember it.

- If one or more entities match, sort by rating level (highest priority for criteria can be used to sort without limit).

- In at least one embodiment, proximity may be more important than most entities. However, since it matches at discrete levels (in a city, within a walking radius, etc.) and it is always indicated, most of the time, most of the matching elements can "pin" by proximity.

- Accessibility (determined by searching a web site such as open-table.com) is a valuable sorting criterion and can be based on a default sorting value if not specified. If the user specifies a time to order, then only the available seats can be listed and the sorting can be based on a recommendation.

- If the user says that he wants the most recommended places, then he can sort the above proximity and accessibility, and these criteria can be relaxed before the recommendation. The assumption is that if someone is looking for a good seat, they may be willing to travel a little further, and this is more important than the availability of a table by default. If a specific time for availability is specified and the user requests recommended places, then places that are both recommended and available may come first, and the recommendation may be relaxed to a loose match before availability no longer matches as one or more.

- The remaining restrictions except for the name are one or more based on incomplete data or a match. So they are the default non-strict sorting heuristic, and when specified, a one-or-more-or-none match.

- The name can be used as a constraint to handle a case that mentions a restaurant by name, such as finding one or more Hobee restaurants near Palo Alto. In this case, one or more elements may match the title and may be sorted by proximity (another specified limitation in this example).

Domain Modeling: Converting Selection Criteria to Basic Data

[0427] It may be desirable to distinguish between the data that is available for calculation by assistant 1002 and the data used to make the selection. In one embodiment, assistant 1002 uses a data model that reduces complexity for the user by collapsing one or more kinds of data used to distinguish between elements into a simple selection criteria model. Internally, this data can take several forms. Instances of a choice class can have internal properties and attributes (for example, a restaurant kitchen), can be compared along dimensions (for example, distance from some location), and can be discovered by some query (for example, whether they match or not with a text construct or are available or not at the moment). They may also be calculated from other data that is not presented to the user as selection criteria (for example, weighted combinations of ratings from multiple sources). This data is relevant as "one or more" to the task, but the differences between these three kinds of data are not relevant to the user. Because the user is thinking in terms of features of the desired selection, rather than properties and dimensions, assistant 1002 operationalizes these various criteria into feature features. Helper 1002 provides a user-friendly domain data model and transforms it into data found in web services.

[0428] One type of transformation is isomorphism from base data to user-friendly criteria. For example, the availability of tables for reservations as observed by the user may be exactly the same as that offered by an online reservation website such as opentable.com, using the same granularity for the time and number of party attendees.

[0429] Another type of transformation is the normalization of data from one or more services into a set of common values, possibly unifying equivalent values. For example, the kitchens of one or more restaurants may be represented as one ontology in helper 1002 and mapped to different vocabularies used by different services. This ontology may be hierarchical and have leaf nodes pointing to specific values from at least one service. For example, one service might have the cuisine meaning "Chinese", another "Sichuan", and a third "Asian". The ontology used by helper 1002 should result in references to "Chinese food" or "Szechuan food" semantically matching one or more of these nodes, with confidence levels reflecting the degree of match.

[0430] Normalization may also be involved in resolving discrepancies in accuracy. For example, the location of a restaurant may be provided at street level in one service, but only as a city in another. In one embodiment, assistant 1002 uses a deep structural representation of locations and times that can be converted to various surface data values.

[0431] In one embodiment, helper 1002 uses a special kind of transformation for open specifiers (eg, romantic, silent) that can be converted to full-text search matches, tags, or other openly textured features. The name of the selection constraint in this case should be something like "described as".

[0432] In at least one embodiment, the constraints can be translated into orderings of practical preference. In other words, given the name of the selection criterion and its constraint values, helper 1002 is able to interpret the criterion as an ordering by possible elements. There are several technical issues that need to be addressed with such a conversion. For example:

- Preference orderings may conflict. The ordering provided by one constraint may be inconsistent or even inversely correlated with the ordering provided by another. For example, price and quality tend to be opposite. In one embodiment, assistant 1002 interprets the constraints selected by the user in a weighted or otherwise combined ordering that reflects the user's desires but is true of the data. For example, a user might query "highly rated cheap French fast food restaurants within walking distance". Many locations may not have such a restaurant at all. However, in one embodiment, helper 1002 may show a list of items that seeks to optimize for at least one constraint and explain why at least one item is listed. For example, one element could be "highly rated French cuisine" and another "cheap fast food restaurant within walking distance".

- The data can be used either as hard or soft limits. For example, a restaurant's price range may be important for restaurant selection, but it may be difficult to specify a threshold for the top price. Even seemingly hard constraints, like the kitchen, can actually be soft constraints due to overlap. Because, in one embodiment, helper 1002 uses a data modeling strategy that seeks to flatten one or more criteria into symbolic values (eg, "cheap" or "near"), these constraints can be translated into a function that correctly interprets the criteria and order without being strict. with respect to meeting specific thresholds. For symbolic criteria with clear objective truth values, helper 1002 may assign higher weights to the objective criteria than to other criteria and make it clear in the explanation that it knows that some elements do not strictly match the requested criteria.

- Items may match some but not one or more of the constraints, and "best fit" items may be shown.

In general, assistant 1002 determines which element features are domain specific and which can serve as selection criteria and, for at least one criterion, possible constraint values. This information may be provided, for example, through work data and API calls.

Text of paraphrases and prompt lines

[0433] As described above, in one embodiment, assistant 1002 provides feedback to indicate that it understands the user's intent and is working towards the user's goal by generating rephrases of their current understanding. In the conversational dialog model of the present invention, a paraphrase is what assistant 1002 outputs after user input as a preface (eg, paraphrase 4003 in FIG. 40) or a summary of results to consider (eg, list 3502 in FIG. 35). The hint line is a suggestion to the user about what else they can do to refine their query or explore the selection space along some dimensions.

[0434] In one embodiment, the targets of paraphrase text and clue lines include, for example:

- show that assistant 1002 understands concepts in user input, not just text;

- indicate the boundaries of understanding 1002 assistant;

- direct the user to enter the text that is required for the intended task;

- help the user to explore the space of possibilities when choosing with restrictions;

- explain the current results obtained from the services in terms of established user criteria and helper 1002 assumptions (eg, explain query results without sufficient or excessive restrictions).

[0435] For example, the following paraphrase and clue line illustrates several of these goals:

User input:

indonesian food in menlo park (Indonesian food in Menlo Park)

System interpretation:

Task=constrainedSelection

SelectionClass=restaurant

Restrictions:

Location=Menlo Park, CA

Cuisine=Indonesian (known in ontology)

Services results:

no strict matches

Paraphrasing:

Sorry, I can't find any Indonesian restaurants near Menlo Park .

Prompt line:

You can try other cuisines or locations (You can try other cuisines or locations).

Line with hint under hypertext links:

Indonesian : You can try other food categories such as Chinese or the favorite food item such as steak.

Menlo Park : Enter the location such as the city, neighborhood, street address or "near" followed by the landmark .

Cuisines : Enter the food category such as Chinese or Pizza.

Locations : Enter the location: the city, zip code or "near" followed by the name of the place.

[0436] In one embodiment, assistant 1002 responds to user input relatively quickly with paraphrasing. The paraphrase is then updated after the results are known. For example, the initial response might be "Looking for Indonesian restaurants near Menlo Park ...". After the results are in, helper 1002 should update the text to read "Sorry, I can't find Indonesian restaurants near Menlo Park . You can try other cuisines or locations ." It should be noted that certain elements are highlighted (indicated here by underlining), indicating that these elements represent restrictions that may be relaxed or changed.

[0437] In one embodiment, special formatting/emphasis is used for keywords in the paraphrase. This may be useful to facilitate user training to interact with the intelligent automated assistant 1002 by indicating to the user which words are most important and more likely to be recognized by the assistant 1002. The user may then be more likely to use such words in the future.

[0438] In one embodiment, the paraphrase and the clue line are generated using any relevant contextual data. For example, any of the following data elements may be used, singly or in combination:

- Grammatical analysis - a tree of ontology nodes linked to matching input markers, with notes and exceptions. For each node in the parse, this may include the node's metadata and/or any tokens in the input that represent the rationale for the node's value.

- Task, if known.

- Choice class.

- Location restriction regardless of selection class.

- Which mandatory parameters are unknown for this selection class (eg location is a mandatory constraint on restaurants).

- The name of an object with a name in grammatical analysis that is an instance of the choice class, if any (for example, the specific name of a restaurant or movie).

- Is this a further clarification or the beginning of a conversation? (Reset starts a new conversation).

- What restrictions in grammatical analysis are associated with the values in the input that changed their meanings? In other words, what restrictions are just changed by the last input?

- Is the selection class inferred or directly specified?

- Sorted by quality, relevance or proximity?

- For each constraint you specify, how well does it match?

- Is the specification entered as text or clicked?

[0439] In one embodiment, the paraphrase algorithm takes into account the request, the domain model 1056, and the results of providing services. The domain model 1056 contains classes and features, including metadata, that are used to determine how to generate text. Examples of such metadata for generating paraphrases include:

- IsConstraint={true | false}

- IsMultiValued={true | false}

- ConstraintType={EntityName, Location, Time, Category Constraint, AvailabilityConstraint, BinaryConstraint, SearchQualifier, GuessedQualifier}

- DisplayName=string

- DisplayTemplateSingular=string

- DisplayTemplatePlural=string

- GrammaticalRole={AdjectiveBeforeNoun,Noun,ThatClauseModifer}

[0440] For example, a grammatical analysis may contain these elements:

Class: Restaurant

IsConstraint=false

DisplayTemplateSingular="restaurant"

DisplayTemplatePlural="restaurants"

GrammaticalRole=Noun

Feature: RestaurantName (Example: "Il Fornaio")

IsConstraint=true

IsMultiValued=false

ConstraintType=EntityName

DisplayTemplateSingular="named $1"

DisplayTemplatePlural="named $1"

GrammaticalRole=Noun

Feature: Restaurant Cuisine (Example: "Chinese")

IsConstraint=true

IsMultiValued=false

ConstraintType=CategoryConstraint

GrammaticalRole=AdjectiveBeforeNoun

Feature: RestaurantSubtype (example: "cafe")

IsConstraint=true

IsMultiValued=false

ConstraintType=CategoryConstraint

DisplayTemplateSingular="$1"

DisplayTemplatePlural="$1s"

GrammaticalRole=Noun

Feature: RestaurantQualifiers (example: "romantic")

IsConstraint=true

IsMultiValued=true

ConstraintType=SearchQualifier

DisplayTemplateSingular="is described as $1"

DisplayTemplatePlural="are described as $1"

DisplayTemplateCompact="matching $1"

GrammaticalRole=Noun

Feature: FoodType (example: "burrito")

IsConstraint=true

IsMultiValued=false

ConstraintType=SearchQualifier

DisplayTemplateSingular="serves $1"

DisplayTemplatePlural="serve $1"

DisplayTemplateCompact="serving $1"

GrammaticalRole=ThatClauseModifer

Feature: IsRecommended (example: true)

IsConstraint=true

IsMultiValued=false

ConstraintType=BinaryConstraint

DisplayTemplateSingular="recommended"

DisplayTemplatePlural="recommended"

GrammaticalRole=AdjectiveBeforeNoun

Feature: RestaurantGuessedQualifiers (example: "impressive")

IsConstraint=true

IsMultiValued=false

ConstraintType=GuessedQualifier

DisplayTemplateSingular="matches $1 in reviews"

DisplayTemplatePlural="match $1 in reviews"

DisplayTemplateCompact="matching $1"

GrammaticalRole=ThatClauseModifer

[0441] In one embodiment, assistant 1002 has the ability to handle mismatched input. To handle such input, domain model 1056 may provide nodes of type GuessedQualifier for each selection class and rules that match otherwise mismatched words if they are in the correct grammatical context. In other words, GuessedQualifiers are parsed as distinct nodes that match when there are words that are not found in the ontology, but that are in the correct context to indicate that they are likely to be select class specifiers. The difference between GuessedQualifiers and SearchQualifiers is that the latter are mapped to a dictionary in the ontology. This difference makes it possible to paraphrase that helper 1002 has uniquely identified an intent in SearchQualifiers and may be less robust when reflecting GuessedQualifiers.

[0442] In one embodiment, assistant 1002 performs the following steps when generating paraphrase text:

1. If the task is unknown, an explanation of what helper 1002 can do and a prompt for additional input.

2. If the task is a constrained selection task and the location is known, then an explanation of the subject areas that the assistant 1002 knows and a prompt query for the selection class.

3. If the selection class is known, but there is no mandatory constraint, then prompt for that constraint (for example, location is required for restricted restaurant selections).

4. If the input contains the EntityName of the selection class, then the output is "search"<name>at<location>.

5. If this is the initial query in a conversation, then the output is "search", followed by a complex noun phrase that describes the restrictions.

6. If this is a stage of further refinement in the dialogue:

a. If the user has just completed the required input, then output "thank you" and then the usual paraphrase. (This happens when there is a mandatory constraint that translates to user input).

b. If the user changes the constraint, acknowledge it and then rephrase normally.

c. If the user has entered a custom name for an instance of the selection class, it is handled in a special way.

d. If the user simply adds an unrecognized phrase, then an indication of how it should collapse as a search. If necessary, the input can be coordinated for the search service.

e. If the user simply adds a normal constraint, then the output is "Good" and the usual paraphrase.

7. To explain the results, use the identical approach for paraphrasing. However, when the results are surprising or unexpected, then explain the results using knowledge of the data and service. Also, when a query has excessive or does not have sufficient constraints, the query is in the form of a hint for additional input.

Grammar for making complex noun phrases

[0443] In one embodiment, when rephrasing the query 734 in a constrained selection task, the basis is a complex noun phrase around a selection class that refers to the current constraints. Each constraint has a grammatical position based on its type. For example, in one embodiment, assistant 1002 may compose a paraphrase, such as:

recommended romantic italian restaurants near menlo park free tables for two who offer ossobuco and are described as " quiet "

The grammar to compose this is as follows:

<paraphraseNounClause>:==<binaryConstraint><searchQualifier><categoryConstraint><itemNoun><locationConstraint><availabiltyCon-straint><adjectivalClauses>

<binaryConstraint>:==single adjective that indicates the presence or absence of a BinaryConstraint (eg recommended (best), affordable (cheap)) You can list multiple in a single query.

<searchQualifier>:==A word or words that match the ontology for the selection class specifier to be passed to the search engine service. (e.g. romantic restaurants, funny movies). Use when ConstraintType=SearchQualifier.

<categoryConstraint:==an adjective that identifies a genre, cuisine, or selection class category (such as a Chinese restaurant or an R-rated file). This is the last prefix adjective because it is the most essential. Use for features of type CategoryConstraint and GrammaticalRole=AdjectiveBeforeNoun.

<itemNoun>:==<namedEntityPhrase>|<selectionClass>| <selectionClassSubType>. Find the most specific way to display a noun.

NamedEntity<SubType<Class

<selectionClass>:==noun which is a generic name for a selection class (e.g. restaurant, movie, place)

<selectionClassSubType>:==noun phrase that is a subtype of the selection class, if known (e.g. diner, museum, shop, bar for selection class "local organization") Use for features where ConstraintType=CategoryConstraint and GrammaticalRole=AdjectiveBeforeNoun.

<namedEntityPhrase>:==<entityName> | "the" (<selectionClass> | <selectionClassSubType>)

<entityName>:==proper name of the selection class instance (e.g. "Il Fornaio" "Animal House" "Harry's Bar")

<locationConstraint>:==<locationPreposition><locationName>

<locationPreposition>:=="at", "near", "on", etc.

<locationName>:==city, actual address, point of interest, or something for GPS, like "Your current location"

<availabilityConstraint>:==an availability constraint, expressed as a prepositional phrase that comes after the noun (eg "with free tables", "with available seats", "available online"). It comes immediately after the noun to give it its characteristic properties.

<adjectivalClauses>:==<modiferVerbPhrase> | "that" <modiferVerbPhrase> "and" <modiferVerbPhrase>

<modiferVerbPhrase>:=a verb phrase that expresses a search keyword style constraint per class of choice (e.g. restaurants that are "described as quiet", "offer meat after 11", "matching 'terribly trendy' in reviews"; films that "contain violence", "starring Billy Bob Thornton"). Include all constraints in the parsing for which GrammaticalRole=ThatClauseModifer, using the "that...and" variant if there is more than one. Use DisplayTemplatePlural to form "that" expressions by placing a GuessedQualifier last. If there is only one such constraint, use the DisplayTemplateCompact variant.

[0444] Table 2 provides several examples of paraphrases that may be provided in response to a first input for a task, according to one embodiment.

table 2 Paraphrasing in response to the first input Input Initial paraphrase tie house Sorry, I don't understand "tie house". I know about restaurants, hotels and other local organizations, flights, weather and buying things online. What are you looking for and where? Palo Alto I can help you find restaurants, organizations, events and activities in Palo Alto. What are you looking for? food Looking for a place to eat? Where? (examples) Italian food I can tell you about Italian restaurants in your area. Where do you want to eat? burrito I can help you find places that offer burritos in your area. Where do you want to eat? best mexican food I can help you find good Mexican restaurants in your area. Where are you? Palo Alto Restaurants Search for restaurants near Palo Alto. Italian Palo Alto Search for Italian restaurants near Palo Alto. Thai food in Menlo Park Search for Thai restaurants near Menlo Park indonesian food in menlo park Search for Indonesian restaurants near Menlo Park best mexican food in palo alto Search for recommended Mexican restaurants near Palo Alto. fast food cafe menlo park Search for fast food restaurants near Menlo Park healthy food in menlo park Search for places near Menlo Park that offer healthy food. cheap bar in menlo park Search for cheap bars near Menlo Park. billiard room in Menlo Park Search for billiard rooms near Menlo Park. hookah in Menlo Park Search for places near Menlo Park described as "hookah". spicy food in sf Search for places near Menlo Park that offer spicy food. burgers in menlo park Search for places near Menlo Park that offer burgers. a place where they dance belly dance in sf Search for places near San Francisco described as "belly dancing". a belly dancer in Menlo Park Search for places near Menlo Park described as "belly dancing". cheap eatery in sf Search for eateries near San Francisco that are described as "inexpensive". tie house menlo park Search for places near Menlo Park described as "tie house". thai house palo alto Finding a Thai House in Palo Alto il fornaio menlo park Search II Fornaio in Palo Alto il fornao in Menlo Park Search for places near Palo Alto described as "il fornao". cliff house in menlo park Finding the Cliff House in Palo Alto romantic restaurant in Palo Alto Finding Romantic Restaurants in Palo Alto romantic restaurant in Palo Alto Finding Romantic Restaurants in Palo Alto quiet romantic restaurant in Palo Alto Search for romantic restaurants in Palo Alto described as "quiet" quiet romantic restaurant in Palo Alto Search for romantic restaurants in Palo Alto described as "quiet" terribly trendy restaurant in Palo Alto Search for restaurants in Palo Alto described as "terribly trendy" terribly trendy restaurant in Palo Alto Search for restaurants in Palo Alto described as "terribly trendy" table for two tonight in Palo Alto Check availability for tonight at restaurants near Palo Alto. table for two italian food tonight in palo alto Checking availability for tonight at Italian restaurants near Palo Alto. il fornaio Palo Alto 18:00 for two Checking availability for tables tonight at Il Fornaio in Palo Alto. table for two in thai house in PA Checking availability for tables tonight at Thai House in Palo Alto. pizza delivery in menlo park i need the best quiet romantic italian restaurant for ossobuco in or near menlo park Okay, search for recommended romantic Italian restaurants near Menlo Park that offer ossobuco and are described as "quiet". Book me a table for two at the best quiet romantic Italian restaurant for ossobuco in or near Menlo Park Okay, search for recommended romantic Italian restaurants near Menlo Park with free tables for two that offer ossobuco and are described as "quiet".

Refine request for places to eat

[0445] Table 3 provides several examples of paraphrases in response to the situation where the user's intention to find a place to eat is known, but he did not choose a specific place to eat. It displays a list of restaurants and provides clarifications.

Table 3 Paraphrasing in response to clarifications Clarifications Menlo Park (was unknown) Thanks, now searching for restaurants in Menlo Park. the cheapest Okay, search for affordable restaurants near Menlo Park. Chinese (was Italian) Okay, I'm changing the cuisine to Chinese. Search for Chinese restaurants near Menlo Park. pizza (add dish) Okay, search for Italian restaurants near Menlo Park that offer pizza. San Francisco (was Menlo Park) Okay, changing location to San Francisco. Search for Italian restaurants near San Francisco that offer pizza. terribly trendy Okay, I'll search the reviews for "terribly trendy." quiet romantic Okay, searching for romantic restaurants in Palo Alto described as "quiet" cliff house Cliff House looks familiar. I'll look for him in Palo Alto.

[0446] Table 4 provides several examples of summaries of results that are provided as soon as the results are received.

Table 4 Summary of results Input Number
results Summary of results tie house There is no data No data (no service call) Palo Alto There is no data No data (no service call) food There is no data No data (no service call) Italian food There is no data No data (no service call) burrito There is no data No data (no service call) best mexican food There is no data No data (no service call) Palo Alto Restaurants many Here are some recommended restaurants near Palo Alto. Do you prefer any particular type of food? Would you like a reservation? Italian Palo Alto some Here are some Italian restaurants near Palo Alto. To reserve a table, click available times. Thai food in Menlo Park some Here are some Thai restaurants near Menlo Park. To reserve a table, click available times. indonesian food in menlo park 0 Sorry I can't find Indonesian restaurants near Menlo Park. You can try other cuisines or locations. best mexican food in palo alto some Here are some recommended Mexican restaurants near Palo Alto. fast food cafe menlo park some Okay, here are some fast food restaurants near Menlo Park. healthy food in menlo park some Here are a few places near Menlo Park that can serve healthy food, based on reviews. cheap bar in menlo park some Here are a few bars near Menlo Park. I don't see anything specific about "cheap bar". Can you say it differently? billiard room in Menlo Park some Here are a few places in Menlo Park that are described as billiard rooms. You can add other words to describe what you want. hookah in Menlo Park 0 I can't find places in Menlo Park that match "shisha". Interested in restaurants, hotels or local organizations? spicy food in sf loose match Here are some places to eat in San Francisco. I don't know which ones offer spicy food. Could it go under a different name? burgers in menlo park some You can find burgers at these places to eat near Menlo Park. a place where they dance belly dance in sf some Here are a few places in Menlo Park that are described as "belly dancing". You can add other words to describe what you want. a belly dancer in Menlo Park 0 I can help you find restaurants, organizations, events and activities in San Francisco. Are there any other words for "a place where they dance belly dancing"? cheap eatery in sf 0 I can help you find restaurants, organizations, events and activities in San Francisco. Are there any other words for "inexpensive diner"? tie house menlo park 0 Sorry, I don't know about the "tie house" in Menlo Park. Any other names for it? thai house palo alto one Found: Thai House restaurant in Palo Alto, California. Would you like to reserve a table? When and for how many people? il fornaio menlo park loose match In Palo Alto there is a restaurant II Fornaio, which is located near Menlo Park. il fornao in Menlo Park correspondence in spelling Did you mean II Fornaio in Palo Alto? If not, try other words. cliff house in menlo park 0 I couldn't find Cliff House in Palo Alto. Can it go under a different name or be located in another city? romantic restaurant in Palo Alto some Here are a few restaurants near Palo Alto that are described as romantic in reviews. romantic restaurant in Palo Alto 0 Here are the restaurants near Palo Alto. Searching the reviews found nothing described as romantic. Any other preference? quiet romantic restaurant in Palo Alto some Here are a few restaurants near Palo Alto that match up with "quiet romantic" in the reviews. quiet romantic restaurant in Palo Alto 0 Here are a few restaurants near Palo Alto. Searching the reviews didn't find anything matching "quiet romantic". Any other preference? terribly trendy restaurant in Palo Alto some Here are a few restaurants near Palo Alto that match up with "terribly trendy" in the reviews. terribly trendy restaurant in Palo Alto 0 Here are a few restaurants near Palo Alto. A search in the reviews did not find anything that matches "terribly fashionable." Any other preference? table for two tonight in Palo Alto some Here are the restaurants with free tables from 6 pm to 8 pm tonight for a dinner party for two in Palo Alto, California. table for two italian food tonight in palo alto some Here are the restaurants with free tables from 6 pm to 8 pm tonight for a dinner party for two in Palo Alto, California. il fornaio Palo Alto 18:00 for two one Here are the table options at II Fornaio in Palo Alto, California at 6:00 pm tonight for a dinner party for two. table for two in thai house in PA one Here is information on Thai House in Palo Alto, California. You can call them on the given phone number. pizza delivery in menlo park Here are the places that offer pizza in Menlo Park. I don't have delivery options at this time. Click on restaurant phone numbers to call for delivery inquiries. i need the best quiet romantic italian restaurant for ossobuco in or near menlo park loose match, sorted by availability Here are the best matches for recommended romantic Italian restaurants near Menlo Park that offer ossobuco and are described as "quiet". Click available times to reserve a table. Book me a table for two at the best quiet romantic Italian restaurant for ossobuco in or near Menlo Park loose match, sorted by availability Here are the best matches for recommended romantic Italian restaurants near Menlo Park with free tables for two that offer ossobuco and are described as "quiet". Click available times to reserve a table. Clarifications Menlo Park (was unknown) some Here are some recommended restaurants near Menlo Park. Do you prefer any particular type of food? the cheapest some I found 4 affordable restaurants near Menlo Park. Chinese (was Italian) I found 4 Chinese restaurants near Menlo Park. pizza (add dish) some I found 4 Italian restaurants near Menlo Park that offer pizza. San Francisco (was Menlo Park) some I found 4 Italian restaurants near San Francisco. terribly trendy some I've found 4 restaurants near Palo Alto that match up with "terribly trendy" in the reviews. quiet romantic some Here are a few restaurants near Palo Alto that match up with "quiet romantic" in the reviews. cliff house 0 I couldn't find Cliff House in Palo Alto. Can it go under a different name or be located in another city?

[0447] Table 5 provides several examples of tooltip strings that are provided when users click active links.

Prompt lines when users click active links

Table 5 Prompt lines when users click active links Anchor text String text with hint Notes Location where Enter a location: city, zip code, or "about" followed by the name of the place. This hint string can be used when the user has not yet provided a location. Palo Alto Enter a location, such as city, neighborhood, actual address or "about", followed by points of interest. This prompt string can be used when the user changes locations. food type Enter a food category, such as Chinese or pizza. Food type and cuisine can be combined Italian You can try other categories of food like Chinese or your favorite food like steak. The user has already spoken Italian. Assistant 1002 assists the user in exploring alternatives. If it is a dish, it dominates the kitchen. reservation Enter the day and time to reserve a table, for example "tomorrow at 8". Reservation Hint healthy food You can also enter menu or cuisine items Known type of food spicy food You can also enter menu or cuisine items Unknown food type restaurants Which restaurant? (e.g. Chinese, pizzeria) Clicking the restaurants link should insert the word "restaurant" at the end of the text entry. organizations You can find local florists, ATMs, doctors, pharmacies and more. What organization are you looking for? Clicking organizations links should add to the machine-readable tag that this is a local search activity You can get information about upcoming concerts, shows, and more. What are you interested in? classes Music, art, theater, sports, etc. What do you want to do in this area? hotels I can help you find an available hotel room. Any preference for comfort or location? weather Enter the city and I'll tell you what the weather is like there. If location is known, just show weather data buying things I can help you find music, movies, books, electronic devices, toys, etc. and buy them on Amazon. What are you looking for?

Suggestion of possible answers in the dialogue

[0448] In one embodiment, assistant 1002 provides contextual recommendations. Recommendations are a way for assistant 1002 to suggest options to the user to move forward from their current situation in the dialog. The set of recommendations offered by helper 1002 is context dependent, and the number of recommendations offered may depend on media and form factor. For example, in one embodiment, most feature recommendations may be offered in accordance with common practice in a dialog, an expanded list of recommendations ("more") may be offered in a scrollable menu, and more additional recommendations are achievable by typing a few characters on the keyboard and selecting from autocomplete options. Those skilled in the art will recognize that other mechanisms may be used to provide recommendations.

[0449] In various embodiments, various types of recommendations may be provided. Examples of recommendation types include:

- options to refine the request, including adding or removing, or changing the values of restrictions;

- options to correct or recover from incorrect situations, for example, "not what I had in mind" or "start again" or "search the Internet";

- options to resolve ambiguity between:

- interpretation of speech;

- text interpretations, including spelling correction and semantic ambiguity;

- context-sensitive commands, such as "show them on the map" or "send directions on my date" or "explain these results";

- recommended cross-selling guidelines, such as next steps in dinner party or event scenarios;

- options to reuse previous commands or parts of them.

[0450] In various embodiments, the context that determines the most relevant recommendations may be derived, for example, from the following:

• dialog state

• user state, including, for example:

- static properties (name, home address, etc.)

- dynamic properties (location, time, network speed)

• history of interaction, including, for example:

- history of requests

- history of results

- the text that was previously entered in autocomplete.

[0451] In various embodiments, recommendations may be generated through any mechanism, such as, for example:

- paraphrasing the subject area, task or constraint based on the ontology model;

- hint in autocomplete based on the current subject area and restrictions;

- paraphrasing ambiguous alternative interpretations;

- alternative interpretations of speech-to-text conversion;

- manual authorship based on special dialogue conditions.

[0452] According to one embodiment, recommendations are generated as operations for commands in some state of filling. Commands are explicit canonical representations of queries, including assumptions and inferences based on the interpretations made for user input. In cases where user input is incomplete or ambiguous, recommendations are an attempt to help the user adjust the input to clarify the command.

[0453] In one embodiment, each command is an imperative sentence having some combination of the following:

- command verb (imperative, for example, "find" or "where");

- subject area (selection class, for example, "restaurants");

- constraint(s), e.g. location = Palo Alto and cuisine = Italian.

[0454] These command parts (verb, domain, constraints) correspond to nodes in the ontology.

[0455] The recommendation, then, can be considered as operations for the command, such as setting it, changing it, or declaring that it is relevant or irrelevant. Examples include:

- assignment of a command verb or subject area ("find restaurants"),

- changing the command verb ("order this", "show this on the map", "save this"),

- changing the subject area ("search for a restaurant, not a local organization")

- an indication that the restriction is relevant ("try to clarify by kitchen"),

- choice of value for restriction ("Italian", "French", etc.)

- choice of limit and value together ("not far from here", "tables for two")

- an indication that the constraint value is incorrect ("not this Boston")

- an indication that the restriction is not relevant ("ignore costs"),

- an indication of the intention to change the value of the restriction ("try another location"),

- changing the value of the constraint ("Italian, not Chinese")

- adding a constraint to the value ("also with a pool")

- bind value to grid ("Los Angeles, not los angelos")

- initiating a new command, reusing context ([after movies] "find nearby restaurants", "send directions to my friend"),

- initiating a command that is "meta" for the context ("explain these results"),

- initiating a new command, resetting or ignoring the context ("start again", "help with speech"),

[0456] The recommendation may also include some combination of the above. For example:

- "the film Milk (Milk), not [restaurants offering] a dish of milk"

- "restaurants offering pizza, not just pizzerias eateries"

- "A place called Costco in Mountain View. I don't care if you think it's a restaurant or a local organization."

- "chinese in mountain view" [recent request]

[0457] In one embodiment, assistant 1002 includes a common mechanism to maintain a list of recommendations ordered by relevance. The format in which the recommendation is offered may differ depending on the current context, mode, and device form factor.

[0458] In one embodiment, assistant 1002 determines which constraints to modify by considering any or all of the following factors:

- Considering whether or not the constraint matters;

- Considering whether the constraint is inferred or explicitly stated;

- Consideration of its characteristic properties (suggestionIndex).

[0459] In one embodiment, helper 1002 determines the output format for the recommendation. Examples of output formats include:

- changing the subject area:

- if the autocomplete option is "find restaurants", then "try something else"

- otherwise [inferred] "this is not a restaurant search"

- changing the restriction in the form of a name:

- if the name is logically deduced, suggestion of an alternative ambiguous interpretation

- substitution in autocomplete of the names of objects from the current results

- other name

- assuming it's not a name search (restriction removed) - maybe suggest a category instead

- "Untitled"

- "not in Berkeley"

- "some other day"

- not in this sense (use ambiguous alternatives)

- inferred date: "any day, no reservation needed"

[0460] In one embodiment, assistant 1002 attempts to resolve ambiguities through recommendations. For example, if the set of current user intent interpretations is too ambiguous 310, then recommendations are one way to prompt for more information 322 as a hint. differences between them to the user. For example, if the user input includes the word "cafe", and this word may be the name of a restaurant or a type of restaurant, then assistant 102 may ask, "Did you mean restaurants called 'cafes' or cafe-restaurants?"

[0461] In one embodiment, assistant 1002 infers restrictions under certain situations. In other words, for constrained selection problems, not all constraints need to be explicitly mentioned in user input; some may be inferred from other information available in the active ontology 1050, short-term storage 1052, and/or other sources of information available to helper 1002. For example:

- Logical inference of subject area or location

- Default assumption such as location

- Loose matching constraint (location with fuzzy and implicit characteristic properties, etc.)

- Ambiguous criteria (comparison with the value of the restriction without a prefix (the name compared to the category, often ambiguous))

[0462] In cases where helper 1002 infers constraint values, it may also suggest those assumptions as recommendations for rejection by the user. For example, it might tell the user "I thought you meant in the area. Would you like to take a look at another location?"

[0463] The present invention has been described with a specific degree of detail regarding possible embodiments. Those skilled in the art will appreciate that the invention may be practiced in other embodiments. First, the naming of components, the capitalization of terms, attributes, data structures, or any other aspect of programming or structural aspect is optional or significant, and the mechanisms that implement the invention or features thereof may have different names, formats, or protocols. . Additionally, the system may be implemented through a combination of hardware and software as described, either entirely in hardware elements or entirely in software elements. In addition, the specific division of functionality between the various system components described in this document is merely exemplary and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0464] In various embodiments, the present invention may be implemented as a system or method for performing the above technologies, alone or in any combination. In another embodiment, the present invention may be implemented as a computer program product comprising a non-volatile computer-readable storage medium and computer program code encoded on the medium for causing a processor in a computing device or other electronic device to perform the techniques described above.

[0465] Reference in this detailed description to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. The occurrences of the phrase "in one embodiment" in various places in the detailed description do not necessarily refer to an identical embodiment.

[0466] Some portions of the foregoing are presented in terms of algorithms and symbolic representations of operations on data bits in the storage device of a computing device. These algorithmic descriptions and representations are a means used by data scientists to most effectively convey the essence of their work to others in the art. The algorithm is presented here and, in general, is conceived as a consistent sequence of steps (instructions) leading to the desired result. Stages - this is what requires the physical processing of physical quantities. Typically, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transmitted, combined, compared, and otherwise processed. It is sometimes convenient, mainly for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and the like. In addition, it is also convenient to refer from time to time to certain arrangements of steps requiring physical processing of physical quantities as modules or encoders without loss of generality.

[0467] However, it should be appreciated that all of these and similar terms are to be associated with the corresponding physical quantities and are merely convenient designations applied to these quantities. Unless expressly stated otherwise, as evident from the following explanation, it is to be understood that throughout the description, explanations using terms such as "processing" or "computation" or "display" or "determination", etc. means the operation and processes of a computer system or similar electronic computing module and/or device that processes and converts data represented as physical (electronic) quantities in computer system storage devices or registers or other such storage, transmission or display devices information.

[0468] Specific aspects of the present invention include the steps and process instructions described herein in the form of an algorithm. It should be noted that the steps and instructions of the process of the present invention may be implemented in software, firmware, or hardware and, if implemented in software, may be downloaded such that they are permanently hosted as well as controlled from various platforms used by multiple operating systems.

[0469] The present invention also relates to a hardware system for performing the operations herein. This hardware system may be specifically designed for the desired purpose, or it may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored on the computing device. Such a computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM). RAM), EPROM, EEPROM, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, each connected to a computer system bus. Additionally, the computing devices referred to herein may include a single processor or may be architectures using multiprocessor schemes to expand computing capabilities.

[0470] The algorithms and screens presented herein are not per se associated with any particular computing device, virtualized system, or other hardware system. Various general purpose systems may also be used for programs in accordance with the teachings herein, or it may be convenient to design a more specialized hardware system to carry out the required method steps. The required structure for a variety of these systems should be apparent from the description provided herein. In addition, the present invention is not described in relation to any particular programming language. It should be appreciated that a variety of programming languages may be used to implement the ideas of the present invention as described herein, and all of the above references to specific languages are provided to disclose the spirit and optimum mode of the present invention.

[0471] Accordingly, in various embodiments, the present invention may be implemented as software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device may include, for example, a processor, an input device (e.g., keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination of the above), an output device (e.g., screen, speaker). and the like), a storage device, a persistent storage device (eg, a magnetic storage device, an optical storage device, and the like), and/or network connections, according to technologies that are known in the art. Such an electronic device may or may not be portable. Examples of electronic devices that can be used to implement the invention include: mobile phone, personal digital assistant, smartphone, kiosk, desktop computer, laptop computer, tablet computer, consumer electronic device, home electronic device; music player; camera; television receiver; subscriber console; electronic game block; etc. An electronic device for implementing the present invention may use any operating system, such as, for example, iOS or MacOS available from Apple Inc., Cupertino, Calif., or any other operating system that is capable of being used on the device.

[0472] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art, taking advantage of the above description, should appreciate that other embodiments may be devised that do not depart from the scope of the present invention as described herein. In addition, it should be noted that the language used in the detailed description is primarily chosen for readability and learning and cannot be chosen to outline or limit the scope of the invention. Accordingly, the disclosure of the present invention is intended to be illustrative and not restrictive of the scope of the invention as defined in the claims.

Claims (84)

1. A method for controlling a user device using voice input, comprising steps in which, in an electronic device containing a processor and a memory that stores instructions for execution by the processor: providing at the user device a graphical user interface including at least partly speech-based conversational interface for user interaction, the graphical user interface displaying at least a portion of the conversational interaction between the user and the user device; obtaining context information related to the interaction between the user and the user device; receiving voice input from the user through the conversational interface; processing the speech input to determine the user's intent associated with the speech input, wherein the user's intent is associated with a task parameter; and by determining that the user's intent associated with speech input is related to the activation of a software application installed on the user device: change the task parameter based on the software application, activating the software application in the user device externally to the graphical user interface including the conversational interface, wherein the software application is activated using the modified task parameter, and issue a response based on the user's intent and contextual information. 2. The method of claim 1, wherein said obtaining context information further comprises the steps of: receiving additional voice input via the conversational interface prior to receiving said voice input; and processing this additional speech input to obtain contextual information. 3. The method of claim 1, wherein said processing the speech input to determine the intent of the user associated with the speech input further comprises disambiguating the speech input using context information. 4. The method of claim 1, further comprising the steps of: receiving additional voice input from the user through the conversational interface; processing this additional speech input to determine the additional intent of the user associated with this additional speech input; and based on the optional user intent that was defined: receive another input related to this additional user intent from the user through the conversational interface, and perform a task to implement this additional user intent within the conversational interface. 5. The method of claim 1, further comprising the steps of: receiving additional voice input from the user through the conversational interface; processing this additional speech input to determine the additional intent of the user associated with this additional speech input; and upon determining that this additional user intent associated with the additional speech input is associated with the activation of an additional software application installed in the user device, activating this additional software application in the user device outside the conversational interface. 6. The method of claim 1, wherein said obtaining context information further comprises the steps of: receiving non-voice input from the user prior to said voice input, and process this non-speech input to obtain contextual information. 7. The method of claim 1, wherein the voice input comprises a command to activate a software application installed on the user device. 8. The method of claim 1, wherein said displaying at least a portion of the conversational interaction includes displaying a paraphrase of the user input. 9. An electronic device configured to be controlled using speech input, the electronic device comprising a processor and a memory storing instructions which, when executed by the processor, cause the electronic device to: providing in the electronic device a graphical user interface including at least partly speech-based conversational interface for user interaction, the graphical user interface displaying at least a portion of the conversational interaction between the user and the electronic device; receive contextual information related to the interaction between the user and the electronic device; receive voice input from the user through the conversational interface; process the speech input to determine the user's intent associated with the speech input, wherein the user's intent is associated with a task parameter; and by determining that the intention of the user associated with speech input is related to the activation of a software application installed in an electronic device: change the task parameter based on the software application, activate the software application in the electronic device externally to the graphical user interface including the conversational interface, wherein the software application is activated using the modified task parameter, and issue a response based on the user's intent and contextual information. 10. The electronic device according to claim 9, wherein said obtaining context information further comprises: receiving additional voice input via the conversational interface prior to receiving said voice input, and processing this additional speech input to obtain contextual information. 11. The electronic device of claim 9, wherein said speech input processing for determining a user's intent associated with the speech input further comprises disambiguating the speech input using contextual information. 12. The electronic device of claim 9, wherein the instructions further direct the electronic device to: receive additional speech input from the user through the conversational interface; process this additional speech input to determine the additional user intent associated with this additional speech input; and based on the optional user intent that was defined: receive other input related to this additional user intent from the user through the conversational interface, and perform a task to implement this additional user intent within the conversational interface. 13. The electronic device of claim 9, wherein the instructions further direct the electronic device to: receive additional speech input from the user through the conversational interface; process this additional speech input to determine the additional user intent associated with this additional speech input; and upon determining that this additional user intent associated with the additional speech input is associated with the activation of an additional software application installed in the electronic device, to activate this additional software application in the electronic device outside the conversational interface. 14. The electronic device according to claim 9, wherein said obtaining context information further comprises: receiving non-voice input from the user prior to said voice input, and processing this non-speech input to obtain contextual information. 15. An electronic device according to claim 9, wherein the voice input comprises a command to activate a software application installed in the electronic device. 16. The electronic device of claim 9, wherein said displaying at least a portion of the conversational interaction includes displaying a paraphrase of a user input. 17. A non-volatile computer-readable medium that stores one or more programs containing instructions that, when executed by a user device, cause that device to: provide in the user device a graphical user interface including at least partly speech-based conversational interface for user interaction, the graphical user interface displaying at least a portion of the conversational interaction between the user and the user device; obtain contextual information related to the interaction between the user and the user device; receive voice input from the user through the conversational interface; process the speech input to determine the user's intent associated with the speech input, wherein the user's intent is associated with a task parameter; and by determining that the user's intent associated with speech input is related to the activation of a software application installed on the user device: change the task parameter based on the software application, activate the software application in the user device externally to the graphical user interface including the conversational interface, wherein the software application is activated using the modified task parameter, and issue a response based on the user's intent and contextual information. 18. The computer-readable medium of claim 17, wherein said instructions further cause the device to: receive additional voice input via the conversational interface prior to receiving said voice input, and process this additional speech input to obtain contextual information. 19. The computer-readable medium of claim 17, wherein said speech input processing for determining the user's intent associated with the speech input further comprises disambiguating the speech input using context information. 20. The computer-readable medium of claim 17, wherein said instructions further cause the device to: receive additional speech input from the user through the conversational interface; process this additional speech input to determine the additional user intent associated with this additional speech input; and based on the optional user intent that was defined: receive other input related to this additional user intent from the user through the conversational interface, and perform a task to implement this additional user intent within the conversational interface. 21. The computer-readable medium of claim 17, wherein said instructions further cause the device to: receive additional speech input from the user through the conversational interface; process this additional speech input to determine the additional user intent associated with this additional speech input; and by determining that the additional user intent associated with the additional speech input is associated with the activation of an additional software application installed in the user device, to activate this additional software application in the user device outside the conversational interface. 22. The computer-readable medium of claim 17, wherein said obtaining context information further comprises: receiving non-voice input from the user prior to said voice input, and processing this non-speech input to obtain contextual information. 23. The computer-readable medium of claim 17, wherein the voice input comprises a command to activate a software application installed on the user device. 24. The computer-readable medium of claim 17, wherein said displaying at least a portion of the conversational interaction includes displaying a paraphrase of user input.

Images