KR101683083B1 - Using context information to facilitate processing of commands in a virtual assistant - Google Patents

Abstract

The virtual assistant uses context information to replace gesture input from natural language or from the user. The context helps to clarify the intent of the user and reduce the number of interpretation candidates of the user input, and reduces the need for the user to provide excessive clarification input. The context may include any available information that the secretary can not substitute for clear user input to limit information processing problems and / or to customize the results. The context may be used to limit the solution during processing of various aspects including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention < RTI ID = 0.0 > [0001] < / RTI > A method of using context information to facilitate command processing in a virtual assistant,

Cross reference to related applications

This application claims the benefit of US Provisional Application No. 12 / 479,477 (Attorney Docket No. P7393US1) filed on June 5, 2009, entitled " Contextual Voice Commands ", the entire disclosure of which is incorporated herein by reference. Claim priority as an application.

This application claims the benefit of US Provisional Application No. 12 / 987,982 (Attorney Docket No. P10575US1) filed on January 10, 2011, entitled " Intelligent Automated Assistant ", the entire disclosure of which is incorporated herein by reference. Further claim priority as an application.

U.S. Provisional Patent Application No. 61 / 295,774, filed on January 18, 2010, entitled " Intelligent Automated Assistant ", which is incorporated herein by reference in its entirety, Document No. SIRIP003P).

This application is a continuation-in-part of US Provisional Application No. 61 / 493,201, filed on June 3, 2011, entitled " Generating and Processing Data Items That Represent Tasks to Perform, " No. P11337P1).

This application is a continuation-in-part of United States Provisional Patent Application, filed with the present application entitled "Generating and Processing Task Items that Represent Tasks to Perform ", the entire disclosure of which is incorporated herein by reference, / (Attorney Docket No. P11337US1).

This application is a continuation-in-part of U. S. Provisional Patent Application, entitled " Automatically Adapting User Interfaces for Hands-Free Interaction, " filed on even date herewith and incorporated herein by reference in its entirety, / (Attorney Docket No. P11357US1).

The present invention relates to a virtual assistant, and more particularly, to mechanisms for improving the interpretation and processing of commands provided to such a secretary.

Today's electronic devices are capable of accessing large, growing, and diverse amounts of functions, services and information over the Internet and from other sources. The functionality of such devices is rapidly increasing as many consumer devices, smartphones, tablet computers, etc. can run software applications to perform various tasks and provide different types of information. Sometimes, each application, function, website, or feature has its own user interface and its own operating paradigms, many of which are either too burdensome to learn or overwhelm users. In addition, many users may find it difficult to find out what functions and / or information are available to their electronic devices or various web sites, and therefore such users may feel frustrated or overwhelmed, or simply use them The available resources may not be available in an efficient manner.

In particular, novice users, or individuals who are not functioning normally, have / have a disability, are old, busy, unable to concentrate / fail, or are in the running of a vehicle can efficiently interfere with their electronic devices and / It can be difficult to participate efficiently in online services. In particular, such users will have difficulties with a large number of diverse and inconsistent functions, applications and websites that may be available for their use.

Thus, existing systems sometimes have difficulty in use and navigation, and sometimes provide inconsistent and overwhelming interfaces to users, hindering users from using technology efficiently.

An intelligent automated assistant, also referred to herein as a virtual secretary, can provide an improved interface between a person and a computer. (Attorney Docket No. P10575US1) filed on January 10, 2011, entitled " Intelligent Automated Assistant ", the entire disclosure of which is incorporated herein by reference. Such a secretary, which may be, allows users to interact with a device or system using a natural language in the form of words and / or text. Such a secretary interprets user inputs, enables the user to manipulate the intent of the user with the tasks and parameters for those tasks, executes services to support those tasks, and generates output that the user can understand .

The virtual assistant can derive any of a number of information sources, for example, to process user input including knowledge bases, models and / or data. In many cases, the user's input alone is not sufficient to clearly define the user's intent and the task to be performed. This may be due to the input stream, individual differences between users, and / or natural language ambiguity. For example, a user of a text messaging application on a telephone may invoke a virtual assistant to say a command "call her ". Such a command is perfectly reasonable in English, but it is not an accurate, executable expression because there can be many interpretations and possible solutions to such a request. Thus, without having additional information, the virtual assistant may not be able to correctly interpret and process such input. This type of ambiguity can result in errors, incomplete operations, and / or excessive burden on the user with requests to clarify input.

According to various embodiments of the present invention, the virtual assistant uses contextual information (also referred to herein as a " context ") to supplement natural language or gestural input from the user. This helps clarify the user's intention, helps reduce the number of candidate interpretations of user input, and reduces the need for the user to provide excessive clarification input. The context may include any available information that can be used by a secretary to supplement explicit user input to limit information-processing problems and / or personalize results. For example, if an input from a user includes a pronoun (such as " her "in a " call her" command), the virtual assistant uses the context to indicate that pronoun For example, the identity of the individual to receive the call and / or the telephone number to use. Other uses of the context are described herein.

According to various embodiments of the present invention, any number of mechanisms may be implemented to obtain and apply context information for performing computations in a virtual assistant implemented on an electronic device. In various embodiments, the virtual assistant was filed on January 10, 2011 under the heading "Intelligent Automated Assistant ", and the entire disclosure of which is hereby incorporated by reference in its entirety into U.S. Provisional Application No. 12 / 987,982 (Attorney Docket No. P10575US1 ) Is an intelligent automatic secretary as described in U.S. Pat. Such a secretary forms a relationship with the user in an integrated way of communicating using natural language conversation, and invokes external services when it is appropriate to obtain information or perform various actions. According to the techniques described herein, context information is used in such secretaries to reduce ambiguity in performing information processing functions such as speech recognition, natural language processing, task flow processing, and dialog generation.

According to various embodiments of the present invention, the virtual assistant uses contexts to perform a variety of different types of operations, functions and / or features, and / or uses a plurality of features of the electronic device in which it is installed May be configured, designed, and / or operable to combine, manipulate, and operate applications, systems, operations, and applications. In some embodiments, the virtual assistant of the present invention may be used to actively derive input from a user, interpret user intent, clarify differences in contention analysis, request and receive clear information as needed And / or performing (or initiating) actions based on the identified intent.

For example, operations may be performed by activating and / or interfacing services and / or services available on an electronic network, such as the Internet, as well as applications or services available on an electronic device. In various embodiments, activation of such external services may be performed by application programming interfaces (APIs) or some other suitable mechanism. As such, the virtual assistant implemented in accordance with various embodiments of the present invention can be used to integrate, simplify, and improve the user experience for many different applications and functions of the electronic device, and for services available over the Internet can do. This allows the user to know what functions are available on the device and in web access services, how to interface with those services to get what they want, how to interpret the output received from those services, less burden to learn . Rather, the inventive virtual assistant can act as an intermediary between a user and such a variety of services.

In addition, in various embodiments, the virtual assistant of the present invention provides an interactive interface that a user may find more convenient and less burdensome than a conventional graphical user interface. A user may use any of a number of available input and output mechanisms, such as voice, graphical user interfaces (buttons and links), text entry, etc., to create a relationship in virtual conversation Can be concluded. The system may be implemented using any of a number of different platforms, such as device APIs, web, email, and the like. Requests for further input may be presented to the user in the context of such a conversation. Long-term memory can be used to interpret user input in an appropriate context, taking into account historical events and communications within a given session as well as history and profile information about the user.

Further, in various embodiments, context information from a feature, an action, or an interaction of an application with a user on a device may be used to simplify the operation of other features, operations, or applications on the device or other device . For example, a virtual assistant can simplify the initiation of a text message using the context of a telephone call (such as the person making the call) (e.g., the user must send a text message to the same person without explicitly specifying the recipient of the text message Can be determined). Thus, the virtual assistant of the present invention can interpret commands such as "Send him a text message ", where" that " includes context information from the current telephone call and / or from any feature, . In various embodiments, the virtual assistant considers the various types of available context data to determine which address book contacts to use, which contact data to use, which phone number to use with the contact, Thereby eliminating the need to manually reassign such information.

The context information source may be the current state of the device used as an interface to the virtual secretary, such as the current time, location, application or data object; Personal data such as a user's address book, calendar, and application usage history; And the state of the conversation between the user and the virtual assistant, such as the recently mentioned person and / or place, and this is illustrative and not limiting.

The context can be applied to various calculations and inferences in the operation of the virtual assistant. For example, contexts can be used to reduce ambiguity or limit the number of solutions when processing user input. Thus, contexts can be used to constrain solutions during various processing steps, and this is illustrative and not limiting.

Voice recognition - Receives voice input and generates candidate interpretations as text, for example, "call her", "collar" and "call Herb". Contexts can be used to constrain which words and phrases are to be considered by the speech recognition module, how to order them, and what to accept as thresholds in consideration. For example, a user's address book may add personal names to other language-generic models of speech, allowing these names to be recognized and prioritized.

Natural Language Processing ( NLP ) - parses text and associates words with syntactic and semantic rules, such as determining that the user input is about making a telephone call to the person the pronoun "she" is about, It is looking for a particular data representation. For example, the context of a text messaging application may help constrain the interpretation of "her " to mean" the person who is speaking with text. &Quot;

Task flow processing - Identifies which telephone number to use for the user task, the task steps, and the task parameters used to assist the task, such as the person she represents. In addition, the context of the text messaging application may constrain the interpretation of the telephone number to indicate that the system should use a current or recently used number in a text messaging conversation.

Generate Dialog - Create secretary responses as part of a conversation with the user about their tasks, for example by changing the user's intent in response to "I'll call Rebecca to her mobile ..." (paraphrase ). The level of verbosity and informal tone is an option that can be guided by contextual information.

In various embodiments, the virtual assistant of the present invention may control various features and operations of the electronic device. For example, the virtual assistant can invoke services and functions that interface with the functions and applications on the device by APIs or other means to perform functions and operations that had to be initiated using a conventional user interface on the device. Such functions and actions may include, for example, setting an alarm, dialing, sending a text or email message, adding calendar events, and the like. Such functions and actions may be performed as add-ons in the context of the conversation used in the daily conversation between the user and the virtual assistant. Such functions and actions may be specified by the user in the context of such conversations, or may be performed automatically based on the context of the conversation. Those skilled in the art will appreciate that the virtual assistant can be used as a control mechanism to initiate and control various operations on an electronic device and thus can be used as an alternative to conventional mechanisms such as buttons or a graphical user interface. As described herein, context information can be used to inform and improve the use of such virtual assistant as a control mechanism.

The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention in accordance with the embodiments. Those skilled in the art will recognize that the specific embodiments shown in the figures are illustrative only and are not intended to limit the scope of the invention.
1 is a block diagram illustrating some examples of sources of context that may affect a virtual assistant and its operation in accordance with one embodiment;
2 is a flow diagram illustrating a method for using a context in various processing stages in a virtual assistant, in accordance with one embodiment;
3 is a flow diagram illustrating a method for using context in speech derivation and interpretation, in accordance with one embodiment;
4 is a flow diagram illustrating a method for using context in natural language processing, according to one embodiment.
5 is a flow diagram illustrating a method for using a context in task flow processing, in accordance with one embodiment;
6 is a block diagram illustrating examples of sources of context that are distributed between a client and a server, according to one embodiment.
7A-7D are event diagrams illustrating examples of mechanisms for obtaining and adjusting context information in accordance with various embodiments.
Figures 8A-8D illustrate examples of various representations of contextual information that may be used in connection with various embodiments of the present invention.
9 illustrates an example of a settings table for caching policies and specifying communications for various context information sources, according to one embodiment;
10 is an event diagram illustrating an example of accessing the context information sources depicted in FIG. 9 during processing of an interaction sequence, in accordance with one embodiment;
11-13 are a series of screenshots illustrating an example of using an application context in a text messaging domain to derive an indicative target for a pronoun, in accordance with one embodiment.
Figure 14 is a screen shot illustrating a virtual secretary prompting for name disambiguation, in accordance with one embodiment.
15 is a screen shot illustrating a virtual secretary using a conversation context to infer a location for a command, in accordance with one embodiment.
16 is a screen shot illustrating an example of using a telephone preference list as a source of context, according to one embodiment;
17 through 20 are a series of screen shots illustrating an example of utilizing the current application context to interpret and operate the command, according to one embodiment.
21 is a screen shot illustrating an example of utilizing the current application context to interpret a command that invokes a different application.
Figures 22-24 are a series of screen shots illustrating an example of using an event context in the form of an incoming text message, according to one embodiment.
25A and 25B are a series of screen shots illustrating an example using previous conversation text, according to one embodiment.
26A and 26B are screen shots illustrating an example of a user interface for selecting one of the candidate interpretations, according to one embodiment.
27 is a block diagram showing an example of an embodiment of a virtual secretarial system;
28 is a block diagram illustrating a computing device suitable for implementing at least a portion of a virtual secretary in accordance with at least one embodiment;
29 is a block diagram illustrating an architecture for implementing at least a portion of a virtual assistant on a standalone computing system, in accordance with at least one embodiment.
30 is a block diagram illustrating an architecture for implementing at least a portion of a virtual assistant on a distributed computing network, according to at least one embodiment.
31 is a block diagram illustrating a system architecture illustrating various different types of clients and modes of operation.
32 is a block diagram illustrating a client and server communicating with one another to implement the invention in accordance with one embodiment.

In accordance with various embodiments of the present invention, various context information is obtained and applied to perform information processing functions in support of operations of the virtual assistant. For purposes of illustration, the term "virtual secretary" is equivalent to the term "intelligent automation secretary ", and both refer to any information handling system that performs one or more of the following functions.

음성(spoken) 또는 텍스트 형태로, 인간 언어 입력을 해석

Interpret human language input in spoken or text form

단계들 및/또는 파라미터들로 태스크를 표현하는 것과 같은, 실행될 수 있는 형태로 사용자 의도를 표현하는 것의 동작화.

Activation of expressing user intent in an executable form, such as expressing a task with steps and / or parameters.

프로그램들, 메소드들, 서비스들, API들 등을 호출함으로써 태스크 표현들을 실행; 및

Executing task expressions by calling programs, methods, services, APIs, etc.; And

언어 및/또는 그래픽 형태로 사용자에 대한 출력 응답들을 생성.

Generate output responses for users in language and / or graphical form.

An example of such a virtual assistant is disclosed in related U.S. Utility Model 12 / 987,982 entitled " Intelligent Automation Assistant "(reference P10575US1) filed on January 10, 2011, the entire disclosure of which is incorporated herein by reference .

Various techniques will now be described in detail with reference to the embodiments as shown in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects and / or features described herein or the content of the references. However, it will be apparent to one skilled in the art that one or more aspects and / or features or references described herein may be practiced without some or all of these specific details. In other instances, well-known process steps and / or structures have not been described in detail so as not to obscure aspects and / or features or parts of the references described herein.

In the present application, one or more different inventions may be described. In addition, various embodiments for one or more of the inventions (s) described herein may be described in this patent application and are presented for illustrative purposes only. The described embodiments are not intended to be limiting in any sense. One or more of the present invention (s) can be widely applied to various embodiments, as will be apparent from the disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the invention (s), and other embodiments may be utilized, and structural, logical, software, electrical and other changes (S) can be made without departing from the scope of the present invention (s). Accordingly, those skilled in the art will recognize that one or more of the inventions may be practiced with various modifications and alternatives. It will be appreciated that one or more of the specific features of the present invention may form one or more specific embodiments or portions of the disclosure, and that one or more specific embodiments of the invention (s) may be described with reference to the drawings, . It is to be understood, however, that such features are not limited to use with one or more specific embodiments or drawings described in the context of the reference. This disclosure is not intended to be a literal description of all or any one or more of the embodiments of the present invention or to list one or more features of the invention (s) that must be present in all embodiments.

The headings of the sections provided in this patent application and the title of the present patent application are for convenience only and are not to be construed as limiting the present disclosure in any way.

Devices that communicate with each other do not need to be in continuous communication with each other unless explicitly specified otherwise. Also, devices that communicate with each other can communicate directly or indirectly through one or more intermediaries.

The description of embodiments utilizing several components that communicate with each other does not imply that all such components are required. Conversely, various optional components are described to illustrate one or more of the broadest possible embodiments of the present invention (s).

Also, process steps, method steps, algorithms, and the like may be described in a sequential order, but such processes, methods, and algorithms may be configured to operate in any appropriate order. In other words, any sequence or sequence of steps that may be described in this patent application does not by itself indicate a requirement that the steps be performed in that order. Also, although some aspects are described or suggested as occurring asynchronously (e.g., because one step is described after another), some steps may be performed concurrently. Also, an example of a process by its description in the figures does not mean that the illustrated process excludes other variations and modifications thereto, and it is understood that any of the steps of the illustrated process, or steps thereof, Quot;), and does not imply that the illustrated process is preferred.

Where a single device or article is described, more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Likewise, it will be apparent that if more than one device or article is described (whether they cooperate or not), a single device / article may be used in place of more than one device or article.

The functionality and / or features of a device may alternatively be implemented by one or more other devices that are not explicitly described as having such functionality / features. Accordingly, one or more other embodiments of the present invention do not require the device itself.

The techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. It should be noted, however, that certain embodiments, unless otherwise stated, include multiple instantiations of a mechanism or repetition of multiple techniques.

Although described in the context of techniques for implementing an intelligent automatic secretary, also known as a virtual assistant, various aspects and techniques described herein may also be employed and / or computerized with human and / or software Lt; RTI ID = 0.0 > interactions. ≪ / RTI >

Virtual secretarial techniques (e.g., utilized, provided, and / or implemented by one or more virtual secretarial system embodiments described herein) are disclosed in one or more of the following, Are incorporated herein by reference.

미국 실용 출원 번호 제12/987,982호, "Intelligent Automated Assistant", 대리인 정리 번호 P10575US1, 2011.1.10 제출.

U.S. Utility Application No. 12 / 987,982, "Intelligent Automated Assistant", Attorney Docket No. P10575US1, submitted on January 10, 2011.

미국 가특허 출원 번호 제61/295,774호, "Intelligent Automated Assistant", 대리인 정리 번호 SIRIP003P, 2010.1.18 제출.

U.S. Provisional Patent Application No. 61 / 295,774, "Intelligent Automated Assistant", Attorney Docket No. SIRIP003P, submitted on January 18, 2010.

미국 특허 출원 번호 제11/518,292호, "Method And Apparatus for Building an Intelligent Automated Assistant", 2006년 9월 8일 제출.

U.S. Patent Application Serial No. 11 / 518,292 entitled " Method And Apparatus for Building Intelligent Automated Assistant ", filed on September 8, 2006.

미국 가특허 출원 번호 제61/186,414호, "System and Method for Semantic Auto-Completion", 2009.6.12 제출.

U.S. Provisional Patent Application No. 61 / 186,414 entitled "System and Method for Semantic Auto-Completion"

Hardware architecture

In general, the virtual secretarial techniques disclosed herein may be implemented on hardware or on a combination of software and hardware. For example, they may be implemented as a library bound to network applications, on a specially configured machine and / or as an operating system kernel on a network interface card, as a separate user process. In certain embodiments, the techniques disclosed herein may be implemented in software such as an operating system or an application running on an operating system.

The software / hardware hybrid implementation (s) of at least a portion of the virtual assistant embodiment (s) disclosed herein may be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in memory. Such network devices may have multiple network interfaces that may be configured or designed to use different types of network communication protocols. The general architecture for some of these machines may be apparent from the description disclosed herein. According to certain embodiments, at least some of the features and / or functionality of the various virtual assistant embodiments disclosed herein may be implemented in one or more general purpose network host machines, such as an end user computer system, a computer, a network server or a server system, Any other suitable electronic device, such as a mobile computing device (e.g., a personal digital assistant (PDA), a mobile phone, a smartphone, a laptop, a tablet computer, etc.), a consumer device, a music player or router, Can be implemented. In at least some embodiments, at least some of the features and / or functions of the various virtual assistant embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, etc.) .

Referring now to FIG. 28, a block diagram illustrating a computing device 60 suitable for implementing at least some of the virtual assistant features and / or functions described herein is shown. The computing device 60 may be, for example, an end-user computer system, a network server or server system, a mobile computing device (e.g., a personal digital assistant (PDA), a mobile phone, a smartphone, a laptop, , A consumer device, a music player, or any other suitable electronic device, or any combination or portion thereof. The computing device 60 may be configured to communicate wirelessly or wiredly with other computing devices, such as clients and / or servers, using a known protocol for such communication over a communication network such as the Internet.

In one embodiment, the computing device 60 includes a central processing unit (CPU) 62, interfaces 68, and a bus 67 (such as a peripheral component interconnect (PCI)). When operating under the control of appropriate software or firmware, the CPU 62 may be responsible for implementing specific functions associated with the functions of the specifically configured computing device or machine. For example, in at least one embodiment, a user's personal digital assistant (PDA) or smart phone may function as a virtual assistant system utilizing CPU 62, memory 61, 65, and interface Or may be designed. In at least one embodiment, the CPU 62 is configured to execute one or more different types of software modules under the control of software modules / components-which may include, for example, an operating system and any suitable application software, drivers, To perform virtual assistant functions and / or tasks.

CPU 62 may include one or more processor (s) 63, such as, for example, a Motorola processor or Intel microprocessor family or MIPS microprocessor family. In some embodiments, processor (s) 63 may comprise hardware (e.g., application-specific integrated circuits (ASICs), electrically erasable programmable read- only memories, field-programmable gate arrays (FPGAs), etc.). In certain embodiments, memory 61 (such as non-volatile random access memory (RAM) and / or read-only memory (ROM)) also forms part of CPU 62. However, there are many different ways in which memory can be combined with a system. The memory block 61 may be used for various purposes such as, for example, caching and / or storage of data, programming instructions, and the like.

As used herein, the term "processor" is not limited to such integrated circuits referred to in the art as just a processor in the art, but may be a microcontroller, a microcomputer, a programmable logic controller, an ASIC (application-specific integrated circuit), and any other programmable circuit.

In one embodiment, the interfaces 68 are provided as interface cards (sometimes referred to as "line cards"). In general, they control the transmission and reception of data packets on the computing network and sometimes support other peripherals used with the computing device 60. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, it is also possible to use, for example, a universal serial bus (USB), serial, Ethernet, wirewire, PCI, parallel, radio frequency (RF), Bluetooth ^TM , near field communication (e.g. using near field magnetic) ), Frame Relay, TCP / IP, ISDN, Fast Ethernet interfaces, Gigabit Ethernet interfaces, Asynchronous Transfer Mode (ATM) interfaces, High Speed Serial Interface (HSSI) interfaces, POS (Point of Sale) fiber data distributed interfaces), and the like. In general, these interfaces 68 may include ports suitable for communication with the appropriate media. In some cases, they may also include independent processors and may include volatile and / or nonvolatile memory (e.g., RAM) in some instances.

Although the system shown in FIG. 28 illustrates one particular architecture for computing device 60 for implementing the inventive techniques described herein, it is by no means that at least some of the features and techniques described herein It is not the only device architecture that can be. For example, architectures having one or any number of processors 63 may be used, and such processors 63 may be within one device or distributed among any number of devices. In one embodiment, one processor 63 handles communication with the routing computation. In various embodiments, different types of virtual assistant features and / or functions may be implemented on a client device (such as a smartphone running personal digital assistant or client software) and a server system (such as a server system, (S) in the virtual secretarial system.

Regardless of the network device configuration, the system of the present invention may include one or more memories (e. G., ≪ RTI ID = 0.0 > For example, a memory block 65). The program instructions may, for example, control the operation of the operating system and / or one or more application programs. The memory or memories may also be configured to store data structures, keyword classification information, advertising information, user click and impression information, and / or other specific non-program information described herein.

Because such information and program instructions may be employed to implement the systems / methods described herein, at least some of the network device embodiments may include a nontransitory machine-readable storage medium, Temporary machine-readable storage media may be configured or designed to store, for example, program instructions, status information, and the like, for performing the various tasks described herein. Examples of such non-volatile machine-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; Optical media such as CD-ROM disks; Read only memory devices (ROM), flash memory, memristor memory, random access memory (RAM), and magneto-optical media, such as magneto-optical media, such as floppy disks, , And the like, but are not limited thereto. Examples of program instructions include both machine code as generated by the compiler and files containing higher level code that can be executed by a computer using the interpreter.

In one embodiment, the system of the present invention is implemented on a standalone computing system. Referring now to FIG. 29, a block diagram illustrating an architecture for implementing at least a portion of a virtual secretary on a standalone computing system in accordance with at least one embodiment is shown. Computing device 60 includes process (s) 63 for executing software implementing virtual secretary 1002. The input device 1206 may be, for example, a keyboard, a touch screen, a microphone (e.g., for voice input), a mouse, a touchpad, a trackball, a five-way switch, a joystick, and / May be any type suitable for receiving user input, including combining. Output device 1207 may be a screen, a speaker, a printer, and / or a combination thereof. Memory 1210 may be a random access memory having the same architecture and architecture as known in the art, used by processor (s) 63 during execution of software. The storage device 1208 may be any magnetic, optical, and / or electrical storage device for storing data in digital form; Examples include flash memory, magnetic hard disk, CD-ROM, and the like.

In another embodiment, the system of the present invention is implemented on a distributed computing network, such as having any number of clients and / or servers. Referring now to FIG. 30, there is shown a block diagram illustrating an architecture for implementing at least a portion of a virtual secretary on a distributed computing network, according to at least one embodiment.

In the arrangement shown in FIG. 30, any number of clients 1304 are provided; Each client 1304 may execute software for implementing the client-side portions of the present invention. In addition, any number of servers 1340 may be provided to handle requests received from clients 1304. Clients 1304 and servers 1340 may communicate with each other via an electronic network 1361, such as the Internet. The network 1361 may be implemented using any known network protocols including, for example, wired and / or wireless protocols.

In addition, in one embodiment, the servers 1340 may invoke external services 1360 when it is necessary to acquire additional information or to store data relating to previous interactions with particular users. Communication with external services 1360 may occur via network 1361, for example. In various embodiments, the external services 1360 include functionality related to or installed on the web-enabled services and / or the hardware device itself. For example, in an embodiment where a secretary 1002 is implemented on a smartphone or other electronic device, the secretary 1002 may obtain information stored in a calendar application ("app"), contacts and / can do.

In various embodiments, the secretary 1002 can control a number of features and operations of the electronic device in which it is installed. For example, the secretary 1002 may interact with the functions and applications of the device via APIs or by other means to perform functions and operations that may otherwise be initiated using the device's conventional user interface May call external services 1360. These functions and operations may include, for example, setting up an alarm, calling, sending a text or email message, adding calendar events, and the like. These functions and actions may be performed as additional functions in the context of the conversation used in the daily conversation between the user and the secretary 1002. [ These functions and actions may be specified by the user in the context of such a conversation, or may be performed automatically based on the context of the conversation. Those skilled in the art will appreciate that the secretary 1002 can thereby be used as a control mechanism for initiating and controlling various operations of the electronic device and the electronic device can be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces ≪ / RTI >

For example, the user may provide the secretary 1002 with an entry saying "I need to get up at 8am tomorrow ". If the secretary 1002 determines the intent of the user using the techniques described herein, the secretary 1002 invokes external services 1340 to interface with the device's alarm clock function or application. Secretary 1002 sets an alarm on behalf of the user. In this manner, the user can use the secretary 1002 as an alternative to conventional mechanisms for setting alarms or performing other functions of the device. If the user's requests are ambiguous or require additional clarification, the secretary 1002 may include an active elicitation, paraphrasing, suggestions, etc., Various techniques may be used, so that the correct services 1340 are invoked and the intended operation is taken. In one embodiment, the secretary 1002 may prompt the user to request and / or confirm additional contextual information from any suitable source before invoking the service 1340 to perform the function. In one embodiment, the user may selectively disable the ability of the secretary 1002 to call specific services 1340, or may disable all such service calls, if desired.

The system of the present invention may be implemented using any of a number of different types of operating modes and clients 1304. Referring now to FIG. 31, a block diagram illustrating a system architecture illustrating operating modes and several different types of clients 1304 is shown. Those skilled in the art will appreciate that the various types of operating modes and clients 1304 shown in Figure 31 are merely exemplary and that the system of the present invention may be implemented using operating modes and / As will be appreciated by those skilled in the art. In addition, the system may include any or all of the operating modes and / or such clients 1304, either alone or in any combination. Illustrative examples include:

입력/출력 디바이스들 및/또는 센서들(1402)을 갖는 컴퓨터 디바이스들. 클라이언트 컴포넌트는 임의의 이러한 컴퓨터 디바이스(1402)에 배치될 수 있다. 적어도 하나의 실시예는 네트워크(1361)를 통한 서버들(1340)과의 통신을 가능하게 하기 위해 웹 브라우저(1304A) 또는 다른 소프트웨어 애플리케이션을 사용하여 구현될 수 있다. 입력 및 출력 채널들은 예를 들어 시각 및/또는 청각 채널들을 포함하는 임의의 유형일 수 있다. 예를 들어, 일 실시예에서, 본 발명의 시스템은, 웹 브라우저의 동등물이 음성에 의해 구동되고 출력을 위해 음성을 사용하는 맹인용 비서의 실시예를 가능하게 하는, 음성 기반 통신 방법들을 사용하여 구현될 수 있다.

Computer devices having input / output devices and / or sensors 1402. The client components may be located in any such computer device (1402). At least one embodiment may be implemented using a web browser 1304A or other software application to enable communication with the servers 1340 over the network 1361. [ The input and output channels may be of any type, including, for example, visual and / or auditory channels. For example, in one embodiment, the system of the present invention may be implemented using voice-based communication methods that enable embodiments of the blind secretary using the voice for output and the equivalent of a web browser driven by voice Can be implemented.

클라이언트가 모바일 디바이스(1304B)의 애플리케이션으로서 구현될 수 있는 I/O 및 센서들(1406)을 갖는 모바일 디바이스들. 이것은 휴대폰, 스마트폰, PDA, 태블릿 디바이스, 네트워크 게임 콘솔 등을 포함하지만 이에 한정되는 것은 아니다.

Mobile devices having I / O and sensors 1406 that a client can be implemented as an application of mobile device 1304B. This includes, but is not limited to, cell phones, smart phones, PDAs, tablet devices, network game consoles, and the like.

클라이언트가 기기(1304C)의 임베딩된 애플리케이션으로서 구현될 수 있는 I/O 및 센서들(1410)을 갖는 소비자 가전들.

Consumer appliances having I / O and sensors 1410, wherein the client may be implemented as an embedded application of device 1304C.

클라이언트가 임베딩된 시스템 애플리케이션(1304D)으로서 구현될 수 있는 대시보드 인터페이스들 및 센서들(1414)을 갖는 승용차들 및 다른 차량들. 이것은 네비게이션 시스템, 음성 제어 시스템, 차량내 엔터테인먼트 시스템 등을 포함하지만 이에 한정되는 것은 아니다.

Passenger cars and other vehicles having dashboard interfaces and sensors 1414 that can be implemented as a client embedded system application 1304D. This includes, but is not limited to, navigation systems, voice control systems, in-vehicle entertainment systems, and the like.

클라이언트가 디바이스 상주 애플리케이션(1304E)으로서 구현될 수 있는, 라우터들(1418) 또는 네트워크에 상주하거나 네트워크와 인터페이스하는 임의의 다른 디바이스 등의 네트워킹된 컴퓨팅 디바이스들.

Networked computing devices such as routers 1418, or any other device that resides on or interfaces with a network, where the client may be implemented as a device resident application 1304E.

비서의 실시예가 이메일 모달리티 서버(1426)를 통해 접속되는 이메일 클라이언트들(1424). 이메일 모달리티 서버(1426)는, 예를 들어 사용자로부터의 입력을 비서로 전송된 이메일 메시지들로서 수취하고 응답들로서 비서로부터의 출력을 사용자에게 전송하는, 통신 브리지로서 동작한다.

Email clients 1424 in which embodiments of the secretary are connected via email modality server 1426. The email modality server 1426 acts as a communication bridge, for example receiving input from a user as email messages sent to a secretary and sending the output from a secretary as a response to the user.

비서의 실시예가 메시지 모달리티 서버(1430)를 통해 접속되는 인스턴트 메시지 클라이언트들(1428). 메시지 모달리티 서버(1430)는, 사용자로부터의 입력을 비서로 전송된 메시지들로서 수취하고 비서로부터의 출력을 응답 시의 메시지들로서 사용자에게 전송하는, 통신 브리지로서 동작한다.

Instant messaging clients 1428 in which embodiments of a secretary are connected through a message modality server 1430. The message modality server 1430 acts as a communication bridge that receives input from the user as messages sent to the secretary and sends the output from the secretary as messages in response to the user.

비서의 실시예가 VoIP(Voice over Internet Protocol) 모달리티 서버(1430)를 통해 접속되는 음성 전화들(1432). VoIP 모달리티 서버(1430)는, 사용자로부터의 입력을 비서에게 말한 음성으로서 수취하고 비서로부터의 출력을 예를 들어 응답 시의 합성된 음성으로서 사용자에게 전송하는, 통신 브리지로서 동작한다.

Voice telephones 1432 to which embodiments of the secretary are connected via a Voice over Internet Protocol (VoIP) modality server 1430. The VoIP modality server 1430 acts as a communication bridge that receives the input from the user as a voice spoken to the secretary and transmits the output from the secretor as a synthesized voice, for example, in response.

For messaging platforms, including but not limited to e-mail, instant messaging, discussion forums, group chat sessions, real-time help or customer support sessions, secretary 1002 can act as a participant in conversations have. Secretary 1002 may monitor the conversation and respond to individuals or groups using one or more of the techniques and methods described herein for one-to-one interactions.

In various embodiments, functionality for implementing the techniques of the present invention may be distributed among any number of client and / or server components. For example, various software modules may be implemented to perform various functions related to the present invention, and such modules may be variously implemented to operate on a server and / or client components. Further details on such arrangements may be found in the relevant U.S. Patent Application No. 12 / 987,982, "Intelligent Automated Assistant ", Attorney Docket No. P10575US1, filed January 10, 2011, which is incorporated herein by reference in its entirety / RTI >

32, the input and output processing functions are distributed to the client 1304 and the server 1340, and the client portion 2794a of the input induction and the client portion 2792a of the output processing are distributed to the client 1304 And the server portion 2794b of the input induction and the server portion 2792b of the output processing are located at the server 1340. [ The following components are located at server 1340:

완성된 어휘(2758b);

Completed vocabulary (2758b);

언어 패턴 인식기들의 완성된 라이브러리(2760b);

A completed library 2760b of language pattern recognizers;

단기 개인 메모리의 마스터 버전(2752b);

A master version 2752b of short term private memory;

장기 개인 메모리의 마스터 버전(2754b).

Master version of long-term personal memory (2754b).

In one embodiment, client 1304 locally maintains subsets and / or portions of these components to improve responsiveness and reduce dependence on network communications. Such subsets and / or portions may be maintained and updated in accordance with known cache management techniques. Such subsets and / or portions include, for example, the following:

어휘의 서브세트(2758a);

A subset of lexicals 2758a;

언어 패턴 인식기들의 라이브러리의 서브세트(2760a);

A subset 2760a of libraries of language pattern recognizers;

단기 개인 메모리의 캐시(2752a);

Cache 2752a of short term private memory;

장기 개인 메모리의 캐시(2754a).

A cache of long term private memory (2754a).

Additional components may be implemented as part of server 1340, including, for example,

언어 해석기(2770);

A language interpreter 2770;

대화 플로우 프로세서(2780);

A conversation flow processor 2780;

출력 프로세서(2790);

An output processor 2790;

도메인 엔티티 데이터베이스들(2772);

Domain entity databases 2772;

태스크 플로우 모델들(2786);

Task flow models 2786;

서비스 편성(2782);

Service organization 2782;

서비스 능력 모델들(2788).

Service Capability Models (2788).

Each of these components will be described in more detail below. Server 1340 obtains additional information by interfacing with external services 1360 as needed.

Conceptual Architecture

Referring now to FIG. 27, a simplified block diagram of an embodiment of a specific example of virtual secretary 1002 is shown. Different embodiments of the virtual secretary 1002, as described in more detail in the above-mentioned related U.S. patent applications, may provide various different types of operations, functions, and / or features And / or may be operable. It will also be appreciated that many of the various operations, functions, and / or features of the virtual secretary 1002 disclosed herein may be different for different entities that interact with the virtual secretary 1002, Type advantages and / or advantages of the present invention. The embodiment shown in FIG. 27 may be implemented using any of the hardware architectures described above, or using different types of hardware architectures.

For example, in accordance with different embodiments, the virtual secretary 1002 may include a variety of different types of operations, functionality, and / or features, such as, for example, one or more of the following Be designed, designed, and / or operable to provide:

제품들 및 서비스들을 찾거나, 발견하거나, 그 중에서 선택하거나, 구매하거나, 예약하거나, 또는 주문하기 위해 인터넷을 통해 이용가능한 데이터 및 서비스들의 애플리케이션을 자동화함. 이들 데이터 및 서비스들을 이용하는 것의 프로세스를 자동화하는 것에 더하여, 가상 비서(1002)는 또한 한번에 데이터 및 서비스들의 몇몇 소스들의 결합된 사용을 가능하게 할 수 있다. 예를 들면, 그것은 몇몇 리뷰 사이트들로부터 제품들에 관한 정보를 결합하고, 다수의 유통업자로부터 가격 및 입수가능성을 확인하고, 그들의 위치들 및 시간 제약들을 확인하고, 사용자가 그들의 문제에 대한 개인화된 해법을 찾는 것을 도와줄 수 있다.

Automate the application of data and services available over the Internet to find, discover, select, purchase, reserve, or order products and services. In addition to automating the process of using these data and services, virtual assistant 1002 can also enable the combined use of several sources of data and services at one time. For example, it combines information about products from several review sites, identifies pricing and availability from multiple distributors, identifies their location and time constraints, and allows users to customize their problems You can help find the solution.

(영화들, 이벤트들, 공연들, 전시들, 쇼들 및 어트랙션들을 포함하나 이에 제한되지 않는) 해야할 일들; (여행 목적지들, 호텔들 및 머무르기 위한 그외의 장소들, 랜드마크들 및 관심 있는 그외의 사이트들, 등을 포함하나 이에 제한되지 않는) 가야할 장소들; (레스토랑 및 바와 같은) 먹거나 또는 마시기 위한 장소들, 다른 사람들을 만나기 위한 시간들 및 장소들, 및 인터넷 상에서 찾을 수 있는 엔터테인먼트 또는 소셜 상호작용의 임의의 그외의 소스들에 대해 찾거나, 조사하거나, 그 중에서 선택하거나, 예약하거나, 그렇지 않은 경우 배우기 위해 인터넷을 통해 이용가능한 데이터 및 서비스들의 이용을 자동화함.

(Including, but not limited to, movies, events, performances, exhibitions, shows and attractions); (Including but not limited to travel destinations, hotels and other places to stay, landmarks and other sites of interest, etc.); Find or search for places to eat or drink (such as restaurants and bars), times and places for meeting other people, and any other sources of entertainment or social interaction that can be found on the Internet, Automate the use of data and services available over the Internet to select, reserve, or otherwise learn from them.

(위치 기반 탐색을 포함하는) 탐색; 네비게이션(지도들 및 방향들); (이름 또는 그외의 속성들에 의해 비지니스들 또는 사람들을 찾는 것과 같은) 데이터베이스 검색; 기상 상태 및 예보를 얻는 것, 마켓 아이템들의 가격 또는 금융 거래의 상태를 확인하는 것; 교통 상황 또는 항공편의 상태를 모니터링하는 것; 캘린더들 및 스케쥴들을 액세스하고 업데이트하는 것; 리마인더들, 경보들, 태스크들, 및 프로젝트들을 관리하는 것; 이메일 또는 그외의 메시징 플랫폼들을 통해 통신하는 것; 및 국부적으로 또는 원격으로 디바이스들을 동작시키는 것(예를 들면, 전화를 거는 것, 전등 및 온도를 제어하는 것, 가정 보안 디바이스들을 제어하는 것, 음악 또는 비디오를 플레이하는 것, 등)을 포함하는 그래픽 사용자 인터페이스들을 갖는 전용 애플리케이션들에 의해 다르게 제공되는 자연어 대화를 통해 애플리케이션들 및 서비스들의 동작을 가능하게 함. 일 실시예에서, 가상 비서(1002)는 디바이스 상에서 이용가능한 많은 기능들 및 앱들을 시작하고, 동작시키고, 제어하는 데에 이용될 수 있다.

(Including location based search); Navigation (maps and directions); Database search (such as finding businesses or people by name or other attributes); Obtaining weather conditions and forecasts, confirming the prices of market items or the status of financial transactions; Monitoring the status of traffic or flight; Accessing and updating calendars and schedules; Managing reminders, alerts, tasks, and projects; Communicating via email or other messaging platforms; And operating devices locally or remotely (e.g., dialing, controlling light and temperature, controlling home security devices, playing music or video, etc.) Enabling the operation of applications and services through natural language conversations that are otherwise provided by dedicated applications with graphical user interfaces. In one embodiment, the virtual secretary 1002 can be used to start, operate, and control many of the functions and applications available on the device.

활동들, 제품들, 서비스들, 엔터테인먼트의 소스, 시간 관리, 또는 자연어의 상호작용 대화 및 데이터 및 서비스들에 대한 자동화된 액세스로부터 이익을 얻는 임의의 그외의 유형의 추천 서비스에 대한 개인적 추천들을 제공함.

Provides personal recommendations for activities, products, services, sources of entertainment, time management, or any other type of referral service that benefits from interactive communication of natural language and automated access to data and services .

In accordance with different embodiments, at least some of the various types of functions, operations, actions, and / or other features provided by virtual secretary 1002 may be implemented in one or more client systems (s) System (s), and / or combinations thereof.

In accordance with different embodiments, at least some of the various types of functions, operations, actions, and / or other features provided by the virtual assistant 1002 can be used to interpret user input, Context-related information can be used.

For example, in at least one embodiment, the virtual secretary 1002 may be operable to utilize and / or to generate various different types of data and / or other types of information when performing a specific task and / have. This may include, for example, input data / information and / or output data / information. For example, in at least one embodiment, virtual secretary 1002 may access information from one or more different types of sources, devices, and / or systems, such as, for example, one or more local and / Process, and / or utilize the functions described herein. Further, in at least one embodiment, the virtual secretary 1002 may generate one or more different types of output data / information that may be stored in, for example, memory and / or systems of one or more local and / or remote devices .

Examples of different types of input data / information that may be accessed and / or used by virtual secretary 1002 may include, but are not limited to, one or more of the following (or combinations thereof).

이동 전화 및 태블릿과 같은 이동 디바이스, 마이크를 구비한 컴퓨터, 블루투스 헤드셋, 자동차 음성 제어 시스템, 전화를 통한 시스템, 응답 서비스 상의 기록, 통합 메시징 서비스 상의 오디오 음성 메일, 클록 라디오와 같은 음성 입력을 갖는 소비자 애플리케이션, 전화국, 홈 엔터테인먼트 제어 시스템, 및 게임 컨솔로부터의 : 음성 입력

Consumers with voice inputs such as mobile devices such as mobile phones and tablets, computers with microphones, Bluetooth headsets, automotive voice control systems, telephony systems, records on answering services, audio voicemail on unified messaging services, From applications, phone stations, home entertainment control systems, and gaming consoles: voice input

컴퓨터 또는 이동 디바이스 상의 키보드, 원격 제어 또는 다른 소비자 전자 디바이스 상의 키패드, 비서에 보내지는 이메일 메시지, 비서에 보내지는 인스턴트 메시지 또는 유사한 단문 메시지, 멀티유저 게임 환경에서의 플레이어들로부터 수신한 텍스트, 및 메시지 피드에서 스트리밍되는 텍스트로부터의 텍스트 입력

A keyboard on a computer or mobile device, a keypad on a remote control or other consumer electronic device, an email message sent to a secretary, an instant or similar short message sent to a secretary, text received from players in a multi- Text input from text streamed from the feed

센서 또는 위치 기반 시스템으로부터 들어오는 위치 정보. 예시는 글로벌 위치확인 시스템(GPS) 및 이동 전화 상의 어시스트티드 GPS(A-GPS)를 포함한다. 일 실시예에서, 위치 정보는 명확한 사용자 입력과 결합된다. 일 실시예에서, 본 발명의 시스템은 알려진 어드레스 정보 및 현재 위치 결정에 기초하여, 사용자가 집에 있을 때를 검출할 수 있다. 이러한 방식으로, 사용자가 집 밖에 있을 때와는 달리 집에 있을 때에 흥미가 있을 수 있는 정보의 유형뿐만 아니라 사용자가 집에 있는지의 여부에 따라 사용자를 대신해 호출되어야 하는 서비스 및 액션의 유형들에 대한 특정 추론들이 행해질 수 있다.

Location information from sensors or location-based systems. Examples include Global Positioning System (GPS) and Assisted GPS (A-GPS) on mobile phones. In one embodiment, the location information is combined with explicit user input. In one embodiment, the system of the present invention can detect when the user is at home, based on known address information and current positioning. In this way, the types of services and actions that must be invoked on behalf of the user, depending on whether the user is home or not, as well as the type of information that may be of interest when the user is at home, Certain inferences can be made.

클라이언트 디바이스 상의 클록으로부터의 시간 정보. 이는 예를 들면, 로컬 시간 및 시간 존을 가리키는 전화기 또는 다른 클라이언트 디바이스로부터의 시간을 포함할 수 있다. 또한, 시간은 사용자 요청의 내용, 예를 들면 "한 시간 안에" 및 "오늘밤"과 같은 어구를 해석하는 데에 사용될 수 있다.

Time information from the clock on the client device. This may include, for example, time from a telephone or other client device pointing to local time and time zones. In addition, the time can be used to interpret the content of the user request, such as "within an hour" and "tonight ".

나침반, 가속도계, 자이로스코프, 및/또는 이동 속도 데이터, 뿐만 아니라 이동 또는 핸드헬드 디바이스로부터의 또는 자동차 제어 시스템과 같은 임베디드 시스템으로부터의 다른 센서 데이터. 이는 원격 제어로부터 응용들 및 게임 콘솔까지의 디바이스 위치확인 데이터를 포함할 수 있다.

Compass, accelerometer, gyroscope, and / or movement speed data, as well as other sensor data from a mobile or handheld device or from an embedded system such as a car control system. This may include device location data from remote control to applications and game consoles.

클릭킹 및 메뉴 선택 및 그래픽 유저 인터페이스(GUI)를 갖는 임의의 디바이스 상의 GUI로부터의 다른 이벤트. 추가 예시들은 터치 스크린에의 터치를 포함한다.

Other events from the GUI on any device with click-kings and menu selections and a graphical user interface (GUI). Additional examples include a touch to the touch screen.

센서들 및 알람 클록, 캘린더 알림, 가격 변동 트리거, 위치 트리거, 서버로부터의 디바이스로의 푸시 통지 등과 같은 다른 데이터-구동 트리거로부터의 이벤트.

Events from other data-driven triggers such as sensors and alarm clocks, calendar notifications, pricing triggers, location triggers, push notifications to devices from the server, and so on.

Inputs to the embodiments described herein may also include the contents of a user interaction history, including dialogue and request history.

Many different types of output data / information may be generated by the virtual secretary 1002, as described in the above-referenced related US utility model application. These may include, but are not limited to, one or more of the following (or combinations thereof).

출력 디바이스에 및/또는 디바이스의 사용자 인터페이스에 직접 보내지는 텍스트 출력;

Text output sent directly to the output device and / or to the user interface of the device;

이메일을 통해 사용자에게 보내지는 텍스트 및 그래픽;

Text and graphics sent to users via email;

메시징 서비스를 통해 사용자에게 보내지는 텍스트 및 그래픽;

Text and graphics sent to users through messaging services;

하기의 하나 이상(또는 그들의 조합)을 포함할 수 있는 음성 출력:

Voice output that may include one or more of the following (or combinations thereof):

Synthesized speech;

O sampled voice;

O recorded voice;

사진, 리치 텍스트, 비디오, 사운드, 및 하이퍼링크(예를 들면, 웹 브라우저에 렌더링되는 컨텐츠)를 갖는 정보의 그래픽 레이아웃;

Graphic layout of information with pictures, rich text, video, sound, and hyperlinks (e.g., content rendered in a web browser);

디바이스가 턴온 또는 턴오프되게 하고, 사운드를 발생시키게 하고, 컬러를 변경하고, 진동하고, 빛을 제어하게 하는 등의 디바이스상의 물리적 작용을 제어하는 액츄에이터 출력;

An actuator output to control the physical action on the device such as to cause the device to turn on or off, to generate sound, to change color, to vibrate, to control the light;

매핑 애플리케이션을 호출, 전화기를 음성 다이얼링, 이메일 또는 인스턴트 메시지를 전송, 미디어를 재생, 캘린더, 태스크 매니저, 및 노트 애플리케이션의 엔트리를 작성, 및 다른 애플리케이션들과 같은 디바이스 상의 다른 애플리케이션을 호출;

Calling a mapping application, voice dialing a phone, sending an email or an instant message, playing media, creating entries for calendars, task managers, and note applications, and calling other applications on the device, such as other applications;

원격 카메라, 휠체어의 제어, 원격 스피커 상의 음악 재생, 원격 디스플레이 상의 비디오 재생 등과 같은 디바이스에 부착 또는 제어되는 디바이스들에의 물리적 액션을 제어하는 액츄에이터 출력.

An actuator output that controls physical actions on devices attached to or controlled by a device such as a remote camera, control of a wheelchair, music playback on a remote speaker, video playback on a remote display,

It is to be understood that the virtual secretary 1002 of FIG. 27 is one example from a wide range of virtual assistant system embodiments that may be implemented. Other embodiments of virtual assistant systems (not shown) may include additional, fewer and / or different components / features than those described in the virtual assistant system embodiment of FIG. 27, for example.

The virtual assistant 1002 may include a plurality of different types of components, devices, modules, processes, systems, etc., which may be implemented and / or illustrated, for example, through the use of hardware and / . 27, secretary 1002 may include one or more of the following types of systems, components, devices, processes, etc. (or a combination thereof).

하나 이상의 액티브 온톨로지(ontology)들(1050);

One or more active ontologies 1050;

액티브 입력 유도 컴포넌트(들)(2794)(클라이언트 부분(2794a) 및 서버 부분(2794b)을 포함할 수 있음);

Active input induction component (s) 2794 (which may include a client portion 2794a and a server portion 2794b);

단기 개인 메모리 컴포넌트(들)(2752)(마스터 버전(2752b) 및 캐시(2752a)를 포함할 수 있음);

Short term private memory component (s) 2752 (may include master version 2752b and cache 2752a);

장기 개인 메모리 컴포넌트(들)(2754)(마스터 버전(2754b) 및 캐시(2754a)를 포함할 수 있으며, 예를 들어, 개인 데이터베이스들(1058), 애플리케이션 선호도 및 사용 히스토리(1072) 등을 포함할 수 있음);

May include long term private memory component (s) 2754 (master version 2754b and cache 2754a), including, for example, personal databases 1058, application preferences and usage history 1072, Lt; / RTI >

도메인 모델 컴포넌트(들)(2756);

Domain model component (s) 2756;

어휘 컴포넌트(들)(2758)(완전한 어휘(2758b) 및 서브세트(2758a)를 포함할 수 있음);

Lexical component (s) 2758 (which may include a complete vocabulary 2758b and a subset 2758a);

언어 패턴 인식기(들) 컴포넌트(들)(2760)(전체 라이브러리(2760b) 및 서브세트(2760a)를 포함할 수 있음);

Language pattern recognizer (s) component (s) 2760 (which may include the entire library 2760b and subset 2760a);

언어 해석기 컴포넌트(들)(2770);

Language interpreter component (s) 2770;

도메인 엔티티 데이터베이스(들)(2772);

Domain entity database (s) 2772;

대화 플로우 프로세서 컴포넌트(들)(2780);

Conversation flow processor component (s) 2780;

서비스 편성 컴포넌트(들)(2782);

Service organization component (s) 2782;

서비스 컴포넌트(들)(2784);

Service component (s) 2784;

태스크 플로우 모델 컴포넌트(들)(2786);

Task flow model component (s) 2786;

대화 플로우 모델 컴포넌트(들)(2787);

Conversation flow model component (s) 2787;

서비스 모델 컴포넌트(들)(2788);

Service model component (s) 2788;

출력 프로세서 컴포넌트(들)(2790);

Output processor component (s) 2790;

In certain client / server based embodiments, some or all of these components may be distributed between client 1304 and server 1340.

In one embodiment, virtual assistant 1002 receives user input 2704 via any suitable input modality, including, for example, touch screen input, keyboard input, voice input, and / do. As described in more detail herein, secretary 1002 also receives context information 1000, which may include event context 2706 and / or any of several other types of contexts .

Upon processing user input 2704 and contextual information 1000 in accordance with the techniques described herein, virtual assistant 1002 generates output 2708 to be presented to the user. Output 2708 may be generated according to any suitable output modality, which may be notified by context 1000 as well as other factors as appropriate. Examples of output modalities include visual output displayed on the screen, auditory output (which may include audio output and / or beeps and other sounds), haptic output (such as vibration), and / . ≪ / RTI >

Additional details regarding the operation of the various components shown in FIG. 27 may be found in related U.S. Patent Application Nos. 12 / 987,982, filed January 10, 2011, Attorney Docket No. P10575US1, It is provided in the "Intelligent Automated Assistant".

Context

As described above, in one embodiment, virtual secretary 1002 acquires and applies various contextual information to perform information processing functions. The following description is presented:

가상 비서(1002)에 의해 사용될 컨텍스트 정보의 소스들의 범위

A range of sources of context information to be used by virtual secretary 1002

컨텍스트 정보를 나타내고, 조직하고, 탐색하기 위한 기법들

Techniques for representing, organizing, and navigating context information

컨텍스트 정보가 가상 비서의 몇몇 기능들의 동작을 지원할 수 있는 방법들

Methods in which the context information can support the operation of some functions of the virtual assistant

분산된 시스템 내에서 컨텍스트 정보를 효과적으로 획득하고, 액세스하고, 적용하기 위한 방법들

Methods for effectively acquiring, accessing, and applying context information within a distributed system

Those skilled in the art will appreciate that the following description of sources, techniques, and methods for using context information is exemplary only and that other sources, techniques, and methods may be used without departing from the essential characteristics of the present invention As will be appreciated by those skilled in the art.

Contextual  Sources

Through the steps of information processing performed by virtual secretary 1002, several different kinds of contexts can be used to reduce possible interpretations of user input. Examples include an application context, a personal data context, and a previous conversation history. Those skilled in the art will recognize that other sources of context may also be available.

Referring now to FIG. 1, there is shown a block diagram illustrating some examples of sources of context that may affect the operation of virtual assistant 1002, and in accordance with one embodiment. The virtual assistant 1002 takes user input 2704, such as a voice or typed language, creates an output 2708 to the user, and / or performs actions on behalf of the user (2710). The virtual secretary 1002 shown in Figure 1 is only one example of a wide range of virtual secretarial system embodiments that can be implemented. Other embodiments (not shown) of the virtual assistant system may include additional, fewer and / or different components / features than those illustrated in the example of virtual assistant 1002 shown in FIG.

As described in more detail herein, the virtual assistant 1002 can access any of a number of different sources of knowledge and data, such as dictionaries, domain models and / or task models. In the context of the present invention, such sources, referred to as background sources, are inside the secretary 1002. In addition to user input 2704 and background sources, virtual assistant 1002 may also include device sensor data 1056, application preferences and usage history 1072, conversation history and auxiliary memory 1052, personal database 1058 ), Personal audio context data 1080, current application context 1060, and event context 2706. In some embodiments of the present invention, This will be described in detail herein.

Application Context (1060)

Application context 1060 indicates the status of an application or similar software in which the user is doing something. For example, a user can use a text messaging application to chat with a particular person. The virtual assistant 1002 need not be specific to or part of the user interface of the text messaging application. Rather, the virtual secretary 1002 can receive the context from any number of applications, and each application can pass on its context and notify the virtual secretary 1002.

If the user is currently using the application when the virtual assistant 1002 is invoked, the state of the application may provide useful context information. For example, when virtual assistant 1002 is invoked from within an email application, the context information may include sender information, recipient information, send date and / or time, title, data extracted from email content, mailbox or folder name, .

Referring to Figures 11-13, a set of screen shots is shown illustrating an example of deriving an object to which a pronoun is directed using an application context in a text messaging domain, according to one embodiment. 11 shows a screen 1150 that can be displayed while the user is using a test messaging application. Figure 12 shows a screen 1250 after the virtual secretary 1002 is activated in the context of a text messaging application. In this example, the virtual secretary 1002 presents a prompt 1251 to the user. In one embodiment, a user may provide a spoken input by tapping the microphone icon 1252. In another embodiment, the secretary 1002 can accept voice input at any time and does not require the user to tap the microphone icon 1252 before providing input; Thus, the icon 1252 may be a reminder that the secretary 1002 is waiting for a voice input.

In FIG. 13, as shown on screen 1253, the user is in conversation with the virtual secretary 1002. The input "call him" said by the user has been echoed back and the virtual secretary 1002 is responding that he will call a particular person at a particular telephone number. To interpret the ambiguous input of the user, the virtual assistant 1002 combines the contexts of the various sources, as described in more detail herein, to derive the subject to which the pronoun is pointing.

Referring to Figures 17-20, there is shown another example of enabling a command to be interpreted and manipulated using the current application context, according to one embodiment.

17, a user's email inbox 1750 is presented to the user, and the user selects to view a particular email message 1751. [ 18 shows an email message 1751 after an email message to view is selected; In this example, the email message 1751 includes an image.

In FIG. 19, the user activated the virtual secretary 1002 while viewing an email message 1751 from within the email application. In one embodiment, an email message 1751 is moved and displayed above the screen to create a space for the prompt 150 from the virtual secretary 1002. This display enhances the notion that the virtual assistant 1002 is providing assistance in the context of the email message 1751 currently being viewed. Thus, the user's input to the virtual assistant 1002 will be interpreted in the current context of the email message 1751 shown.

In FIG. 20, the user has provided a command 2050: "Reply let's get this to marketing right away." Context information is used to interpret the command 2050, including the email message 1751 and information about the email application for which the email message is displayed. This context can be used to determine the meaning of the words "reply" and "this " of the command 2050 and to determine how to configure an email configuration transaction for a particular recipient on a particular message thread. In this case, the virtual assistant 1002 can access the context information to determine that "marketing" refers to a recipient named "John Applecore" and can determine the email address to use for the recipient. Thus, the virtual secretary 1002 configures the email 2052 to be approved and sent by the user. In this manner, the virtual assistant 1002 may be able to manipulate tasks (constituting an email message) based on user input with context information describing the current application state.

The application context can also help to identify the meaning of the user's intent across applications. 21, a user invokes a virtual assistant 1002 in the context of a viewing email (e.g., email message 1751), and the user's command 2150 calls the "Send him a text ..." Fig. Command 2150 is interpreted by virtual correspondent 1002 as indicating that a text message should be sent rather than e-mail. However, the use of the word "him " indicates that the same recipient (John Appleseed) is intended. Thus, the virtual assistant 1002 recognizes that this communication should go to this recipient on a different channel (a text message to the telephone number of the person obtained from the contact information stored in the device). Thus, the virtual secretary 1002 constructs a text message 2152 to be accepted and transmitted by the user.

Examples of context information that can be obtained from the application (s) include, but are not limited to:

애플리케이션의 아이덴티티;

The identity of the application;

현재 이메일 메시지, 현재 음악 또는 재생목록 또는 재생되는 채널, 현재 책 또는 영화 또는 사진, 현재 캘린더 일/주/달, 현재 리마인더 리스트, 현재 전화 통화, 현재 텍스트 메시징 대화, 현재 지도 위치, 현재 웹 페이지 또는 검색 쿼리, 위치 감지 애플리케이션들을 위한 현재 도시 또는 다른 위치, 현재 소셜 네트워크 프로필, 또는 현재 객체들의 임의의 다른 애플리케이션 특정 개념;

Current email message, current music or playlist or channel being played, current book or movie or photo, current calendar day / week / month, current reminder list, current phone call, current text messaging conversation, current map location, A search query, a current city or other location for location sensing applications, a current social network profile, or any other application specific concept of current objects;

이름, 장소, 날짜, 및 현재 객체들로부터 추출될 수 있는 식별가능한 다른 엔티티 또는 값.

Name, place, date, and other identifiable entities or values that can be extracted from the current objects.

Personal databases (1058)

Another source of context data is the user's personal database (s) 1058 on the device, such as a telephone, for example an address book containing names and telephone numbers. Referring to Fig. 14, there is shown an example of a screenshot 1451 in which the virtual secretary 1002 prompts for an ambiguous name, according to one embodiment. Here, the user said "Call Herb"; The virtual assistant 1002 prompts the user to select from matching contacts in the address book of the user. Thus, the address book is used as the source of the personal data context.

In one embodiment, the user's personal information is obtained from the personal database 1058 for use as a context to enable the user's intention or other functions of the virtual assistant 1002 to be interpreted and / or manipulated. For example, the data in the user's contact database can be used to reduce ambiguity in interpreting the user's commands when the user mentions someone by name alone. Examples of context information that may be obtained from personal databases 1058 include, but are not limited to, the following:

사용자의 연락처 데이터베이스(어드레스 북) -- 이름, 전화 번호, 물리 어드레스, 네트워크 어드레스, 계정 식별자, 중요한 날짜에 관한 정보가 포함됨 -- 사람, 회사, 조직, 위치, 웹 사이트, 및 사용자가 참조할 수 있는 다른 엔티티에 대한 것;

User's contact database (address book) - Includes information about name, phone number, physical address, network address, account identifier, and important date. - User, company, organization, location, It's for another entity;

사용자 자신의 이름, 선호되는 발음, 어드레스, 전화 번호, 등;

User's own name, preferred pronunciation, address, telephone number, etc.;

어머니, 아버지, 자매, 상사 등과 같은 사용자가 명명한 관계;

A relationship named by a user such as a mother, father, sister, or boss;

캘린더 이벤트, 특별한 날의 명칭, 사용자가 참조할 수 있는 명명된 다른 엔트리들을 포함하는, 사용자의 캘린더 데이터;

A calendar event of a user, a name of a special day, user's calendar data including other named entries that the user can refer to;

해야 하거나 기억해야 하거나 얻어야 할 것의 목록을 포함하여 사용자가 참조할 수 있는 사용자의 리마인더 또는 작업 목록;

A list of reminders or tasks of the user that the user can reference, including a list of what to do or need to remember or get;

노래, 장르, 재생 목록, 및 사용자가 참조할 수 있는 사용자의 뮤직 라이브러리와 연관된 다른 데이터;

Songs, genres, playlists, and other data associated with the user's music library that the user can refer to;

사람, 장소, 카테고리, 태그, 레이블, 또는 사용자의 미디어 라이브러리 내의 포토 또는 비디오 또는 다른 미디어 상의 다른 상징적 이름;

A person, place, category, tag, label, or other symbolic name on a photo or video or other media in your media library;

제목, 저자, 장르, 또는 사용자의 개인 라이브러리에 있는 다른 문헌 또는 어드레스 북의 다른 상징적 이름 .

Title, author, genre, or other symbolic name of another document or address book in your personal library.

Conversation history (1052)

Another source of context data is the virtual secretary 1002 and the user's conversation history 1052. Such a history may include, for example, references to domains, people, places, and the like. Referring to FIG. 15, there is shown an example in which the virtual secretary 1002 uses a conversation context to deduce a location for a command, according to one embodiment. On screen 1551, the user first asks "What's the time in New York "; The virtual secretary 1002 responds by providing the current time in New York City (1552). Later, the user asks "What's the weather?" The virtual secretary 1002 uses the previous conversation history to infer that the intended location of the weather query is the last location mentioned in the conversation history. Accordingly, the response 1553 provides weather information for New York City.

As another example, if a user says "find camera shops near here," after examining the results, say "how about in San Francisco?" And the secretary uses the conversation context to say "how about" the same task (find camera stores) "and the word" in San Francisco "means" changing the locus of the search from here to San Francisco ". The virtual secretary 1002 may also use context details of the previous conversation, such as the previous output provided to the user. For example, if the virtual assistant 1002 used a humorous response intended to be humorous, such as "Sure thing, you're the boss ", the virtual assistant remembers this response already, You can avoid repeating those phrases.

Examples of context information from the conversation history and virtual auxiliary memory include,

대화에서 언급된 사람들;

People mentioned in the conversation;

대화에서 언급된 장소들 및 위치들;

Places and locations mentioned in the conversation;

포커스되는 현재의 시간 프레임;

The current time frame being focused;

이메일이나 캘린더와 같이, 포커스되는 현재의 애플리케이션 도메인;

The current application domain being focused, such as an email or calendar;

이메일을 읽는 것 또는 캘린더 엔트리를 생성하는 것과 같이, 포커스되는 현재의 태스크;

The current task being focused, such as reading an email or creating a calendar entry;

막 읽은 이메일 메시지 또는 막 생성한 캘린더 엔트리와 같이, 포커스되는 현재의 도메인 객체들;

The current domain objects being focused, such as the email message just read or the calendar entry just created;

질문을 하고 있는지 및 어떤 가능한 대답들이 예상되는지와 같이, 대화 또는 트랜잭션 플로우의 현재의 상태;

The current state of the conversation or transaction flow, such as whether it is asking questions and what possible responses are expected;

"good Italian restaurants"와 같이, 사용자 요청들의 히스토리;

the history of user requests, such as "good Italian restaurants ";

리턴된 레스토랑 세트와 같이, 사용자 요청들의 결과들의 히스토리;

A history of the results of user requests, such as returned restaurant sets;

대화에서 비서에 의해 이용된 어구들의 히스토리;

The history of the phrases used by the secretary in the conversation;

"my mother is Rebecca Richards" 및 "I liked that restaurant"와 같이, 사용자가 비서에게 말한 사실들

Facts the user told the secretary, such as "my mother is Rebecca Richards" and "I liked that restaurant"

But are not limited thereto.

Referring now to Figures 25A and 25B, there is shown a series of screen shots illustrating an example of the use of a prior conversation context in accordance with one embodiment. 25A, the user has entered a request 2550 for any new email from John. The virtual secretary 1002 responds by displaying an email message 2551 from John. In Fig. 25B, the user inputs a command 2552 "Reply let's get this to marketing right away ". The virtual secretary 1002 interprets the command 2552 using the previous conversation context; In particular, this command is interpreted to refer to the email message 2551 displayed in Fig.

device  Sensor data (1056)

In one embodiment, the physical device executing the virtual secretary 1002 may have more than one sensor. These sensors may provide sources of context information. As an example of such information,

사용자의 현재 위치;

Your current location;

사용자의 현재 위치에서의 현지 시간;

Local time at the user's current location;

디바이스의 위치, 방위 및 모션;

Device position, orientation and motion;

현재의 광 레벨; 온도 및 다른 환경 척도;

Current light level; Temperature and other environmental measures;

사용 중인 마이크로폰들 및 카메라들의 속성들;

Properties of the microphones and cameras in use;

이용되고 있는 현재의 네트워크들 및 접속된 네트워크들의 서명들 - 이 네트워크는 이더넷, Wi-Fi 및 블루투스를 포함함 -

Current networks being used and signatures of connected networks - This network includes Ethernet, Wi-Fi and Bluetooth -

But are not limited thereto. Signatures include MAC addresses of network access points, assigned IP addresses, device identifiers such as Bluetooth names, frequency channels, and other attributes of wireless networks.

The sensors may be any type of sensor including, for example, an accelerometer, a compass, a GPS unit, an altitude detector, an optical sensor, a thermometer, a barometer, a clock, a network interface,

Application affinity and usage history (1072)

In one embodiment, the user's preferences and settings for various applications, as well as information describing the user's usage history 1072 may be used to interpret and / or manipulate the user's intent or other function of the virtual secretary 1002 And is used as a context in order to be able to do so. As an example of such preferences and history 1072,

단축키, 즐겨찾기, 북마크, 친구 리스트, 또는 사람들, 회사들, 어드레스들, 전화 번호들, 장소들, 웹 사이트들, 이메일 메시지들 또는 임의의 다른 참조들에 대한 사용자 데이터의 임의의 다른 집합;

Any other set of user data for shortcuts, favorites, bookmarks, friend lists, or people, companies, addresses, phone numbers, places, websites, email messages or any other references;

디바이스 상에서 이루어진 최근의 통화들;

Recent calls made on the device;

대화 상대방들을 포함하는 최근의 텍스트 메시지 대화들;

Recent text message conversations involving conversation partners;

지도 또는 방향에 대한 최근의 요청들;

Recent requests for maps or directions;

최근의 웹 검색들 및 URL들;

Recent web searches and URLs;

주식 애플리케이션에 열거된 주식들;

Stocks listed in stock applications;

재생되는 최근의 노래나 비디오 또는 다른 미디어;

Recent songs, videos or other media being played;

경보 애플리케이션들 상에 설정된 알람의 명칭들;

The names of the alarms set on the alarm applications;

디바이스 상의 애플리케이션들 또는 다른 디지털 객체들의 명칭들;

The names of applications or other digital objects on the device;

사용자의 선호 언어 또는 사용자의 위치에서 사용 중인 언어

The language you are using in your preferred language or user's location

But is not limited thereto.

Referring now to Figure 16, an example of the use of a telephone book favorites list as a source of context is illustrated in accordance with one embodiment. On screen 1650, a list of favorite contacts 1651 is shown. If the user provides an input for "call John ", a list of these favorite contacts 1651 may be used to determine that" John " refers to the mobile number of John Appleseed, This is because it appears in this list.

event Context (2706)

In one embodiment, the virtual secretary 1002 may utilize the context associated with the asynchronous events that occur independently of the interaction of the user with the virtual secretary 1002. [ 22 to 24, there is shown an example of the activation of the virtual secretary 1002 after an event that can provide an event context or an alert context according to an embodiment. In this case, the event is an incoming text message 2250, as shown in FIG. 23, the virtual secretary 1002 is invoked and a text message 2250 appears with a prompt 1251. 24, the user has entered the command "call him" (2450). The virtual secretary 1002 uses the event context to clarify the command by interpreting "him " to mean the person who sent the incoming text message 2250. [ Virtual assistant 1002 also uses the event context to determine which phone number to use for outgoing calls. A confirmation message 2451 is displayed to indicate that a call is being made.

Examples of alert context information include,

착신 텍스트 메시지들 또는 페이지들;

Incoming text messages or pages;

착신 이메일 메시지들;

Incoming email messages;

착신 전화 통화들;

Incoming phone calls;

리마인더 통지들 또는 태스크 경보들;

Reminder notifications or task alerts;

캘린더 경보들;

Calendar alerts;

알람 클록, 타이머 또는 다른 시간-기반 경보들;

Alarm clock, timer or other time-based alarms;

게임으로부터의 스코어나 다른 이벤트의 통지들;

Notifications of scores or other events from the game;

주가 경보와 같은 금융 이벤트의 통지들;

Notifications of financial events such as stock price warnings;

뉴스 플래시들 또는 다른 방송 통지들;

News flashes or other broadcast notifications;

임의의 애플리케이션으로부터의 푸시 통지들

Push notifications from any application

But are not limited thereto.

Personal acoustics Context  Data (1080)

In interpreting the speech input, the virtual assistant 1002 may also consider the acoustic environment into which the speech is input. For example, the noise profile of a quiet office is different from the noise profile of a car or public place. If the speech recognition system can identify and store the sound profile data, such data can also be provided as context information. The acoustic context may support recognition and interpretation of the input when combined with other context information, such as the properties of the microphone in use, the current location, and the current conversation state.

Contextual  Expression and Access

As described above, the virtual secretary 1002 may use context information from any of a number of different sources. Any of a number of different mechanisms may be used to represent the context to be available to the virtual secretary 1002. [ Referring now to Figures 8A-8D, several examples of representations of contextual information as may be utilized in connection with various embodiments of the present invention are shown.

People, places, Time , Representations of domains, tasks and objects

8A shows examples of context variables 801-809 that represent simple attributes such as the geographical coordinates of the user's current location. In one embodiment, the current value may be maintained for the core set of context variables. For example, there may be a current user, the current location being focused, the current time frame being focused, the current application domain being focused, the current task being focused, and the current domain object being focused. A data structure as shown in Fig. 8A may be used for such representation.

8B shows an example of a more complex representation 850 that may be used to store context information for a contact. Also shown is an example of an expression 851 that includes data for a contact. In one embodiment, the contact (or person) may be represented as an object having a name, a gender, an address, an attribute for a telephone number, and other attributes that can be maintained in the contact database. Similar expressions can be used for location, time, application domain, task, domain object, and so on.

In one embodiment, a set of current values of a given type is represented. This set can refer to the current people, current locations, current times, and so on.

In one embodiment, the context values are arranged into a history such that in the iteration N, there is a frame of current context values, and there is also a frame of context values that was current in the iteration N-1, . 8C shows an example of an array 811 that includes a history of context values. In particular, each column in Figure 8C represents a context variable, where the rows correspond to different times.

In one embodiment, the set of typed context variables are arranged into a history as shown in Figure 8D. In this example, a set of context variables 861 referring to people is shown with a different set of context variables 871 referring to places. Thus, relevant context data for a particular time in the history can be retrieved and applied.

Those skilled in the art will appreciate that the specific representations shown in Figures 8A-8D are exemplary only and that many other mechanisms and / or data formats for representing the context may be used. Examples include:

일 실시예에서는, 가상 비서(1002)가 사용자에게 어떻게 어드레스하고 사용자의 집, 업무, 이동 전화 등을 참조할지를 알 수 있도록 시스템의 현재 사용자가 일부 특수한 방식으로 표현될 수 있다.

In one embodiment, the current user of the system may be represented in some special way so that the virtual secretary 1002 knows how to address and refer to the user's home, work, mobile phone, and the like.

일 실시예에서는, 가상 비서(1002)가 "나의 엄마" 또는 "나의 상사의 집"과 같은 참조(reference)들을 이해할 수 있도록 사람들 간의 관계가 표현될 수 있다.

In one embodiment, relationships between people can be expressed such that the virtual secretary 1002 can understand references such as "my mom" or "my boss's home. &Quot;

장소는, 이름, 거리 어드레스, 지리적 좌표 등과 같은 속성들을 갖는 객체로서 표현될 수 있다.

A place may be represented as an object having attributes such as name, street address, geographical coordinates, and so on.

시간은 (년, 월, 일, 시간, 분 또는 초 등의) 유니버셜 타임, 타임존 오프셋, 레졸루션을 포함하는 속성들을 갖는 객체로서 표현될 수 있다. 시간 객체는 또한 "오늘", "이번 주", "이번 (다가오는) 주말", "다음 주", "애니의 생일" 등과 같은 심볼 시간을 나타낼 수도 있다. 또한, 시간 객체는 시기들 또는 시점들을 나타낼 수 있다.

The time can be represented as an object having properties including universal time (such as year, month, day, hour, minute or second), time zone offset, resolution. The time object may also represent symbol times such as "today", "this week", "this coming weekend", "next week", "Annie's birthday" Also, a time object may represent periods or points in time.

이메일, 텍스트 메시징, 전화, 캘린더, 연락처, 사진, 비디오, 지도, 날씨, 리마인더, 시계, 웹 브라우저, 페이스북, 판도라 등과 같은 담화(discourse)의 도메인 또는 애플리케이션 또는 서비스를 나타내는 애플리케이션 도메인의 면에서 또한 컨텍스트가 제공될 수 있다. 현재의 도메인은 어떤 도메인에 초점이 맞추어져 있는지를 표시한다.

In terms of an application domain representing a domain or application or service such as email, text messaging, phone, calendar, contact, photo, video, map, weather, reminder, clock, web browser, facebook, Pandora, Context can be provided. The current domain indicates which domain is focused.

컨텍스트는 또한 도메인 내에서 수행하기 위한 하나 이상의 태스크, 또는 동작들을 정의할 수 있다. 예를 들어, 이메일 도메인 내에서는, 이메일 메시지 판독, 이메일 검색, 새로운 이메일 작성 등과 같은 태스크들이 있다.

The context may also define one or more tasks, or actions, to perform within the domain. For example, within an email domain, there are tasks such as reading email messages, searching for emails, creating new emails, and so on.

도메인 객체들은 다양한 도메인과 연관된 데이터 객체들이다. 예를 들면, 이메일 도메인은 이메일 메시지 상에서 동작하며, 캘린더 도메인은 캘린더 이벤트 상에서 동작한다.

Domain objects are data objects associated with various domains. For example, an email domain operates on an email message, and a calendar domain operates on a calendar event.

For purposes of explanation provided herein, the expression of context information is referred to as a context variable of a given type. For example, the expression of the current user is a context variable of the type person.

Context  Expression of derivation

In one embodiment, derivation of the context variable is explicitly expressed so that it can be used for information processing. Derivation of the context information is characterized by a source and / or set of interferences that are to conclude and retrieve the information. For example, the person context value 851 shown in FIG. 8B may be derived from the text message domain object obtained from the event context 2706. The source of this context value 851 may be represented.

User requests and / or intentional History  expression

In one embodiment, a history of user requests may be stored. In one embodiment, a history of profound structural representations of user intent (derived from natural language processing) can also be stored. This allows the virtual secretary 1002 to make the new input meaningful in the context of the previously interpreted input. For example, if the user asks "What is New York weather? &Quot;, the language interpreter 2770 can interpret the question by referring to the location of New York. Then, if the user says, "What about this week?", The virtual assistant 1002 can refer to the previous interpretation and determine that "what" should be interpreted as meaning "what is the weather?"

result History  expression

In one embodiment, a history of the results of a user request may be stored in the form of a domain object. For example, for a user request of "Find a good Italian restaurant ", a set of domain objects representing a restaurant may be returned. Thereafter, when the user inputs a command such as "Please call Amilio", the virtual assistant 1002 will search for a restaurant named Amilio, the smallest of all possible places that can be recalled within the search results You can search for results for.

Context  Delayed binding of variables

In one embodiment, the context variable may represent information that is retrieved or derived on demand. For example, a context variable representing the current location may, upon access, call the API to retrieve the current location data from the device and then do other processing to calculate the street address, for example. The value of the context variable may be maintained for a specific time period, depending on the caching policy.

Context  Search

The virtual assistant 1002 may use any of a number of different approaches for retrieving relevant context information to solve information processing problems. Examples of different types of searches include, but are not limited to, the following:

컨텍스트 변수 이름에 의한 검색. "현재 사용자의 이름(first name)"과 같이, 요구된 컨텍스트 변수의 이름이 알려져 있는 경우, 가상 비서(1002)는 그 이름의 인스턴스를 검색할 수 있다. 히스토리가 유지되고 있는 경우, 가상 비서(1002)는 먼저 현재 값들을 검색한 다음, 매칭이 발견될 때까지 이전 데이터를 컨설팅한다.

Search by context variable name . If the name of the requested context variable is known, such as "first user name ", virtual assistant 1002 can retrieve an instance of that name. If the history is maintained, the virtual assistant 1002 first retrieves the current values and then consults the previous data until a match is found.

컨텍스트 변수 유형에 의한 검색. 개인과 같은, 요청된 컨텍스트 변수의 유형이 알려져 있는 경우, 가상 비서(1002)는 이러한 유형의 컨텍스트 변수의 인스턴스들을 검색할 수 있다. 히스토리가 유지되는 경우, 가상 비서(1002)는 먼저 현재 값들을 검색한 다음, 매칭이 발견될 때까지 이전 데이터를 컨설팅할 수 있다.

Retrieval by context variable type . If the type of the requested context variable, such as an individual, is known, the virtual assistant 1002 can retrieve instances of this type of context variable. If the history is maintained, the virtual secretary 1002 may first retrieve the current values and then consult the previous data until a match is found.

In one embodiment, if the current information processing problem requires only one match, the search is terminated once a match is found. If multiple matches are allowed, matching results can be retrieved until some limit is reached.

In one embodiment, if appropriate, the virtual secretary 1002 may limit the search for data with a particular derivation. For example, if a person is looking for a person object in a task flow to send an e-mail, the virtual secretary 1002 only has to consider the context variable whose derivation is the application associated with that domain.

In one embodiment, the virtual secretary 1002 uses rules to rank the matches according to heuristics using any available attributes of the context variable. For example, when processing a user input that includes a command "Tell her to be late ", the virtual assistant 1002 interprets" her " Meanwhile, the virtual secretary 1002 may apply the sequencing so that its derivation represents a preference for a person object that is an application usage history for a communication application, such as text messaging and e-mail. As another example, when interpreting the command "Call her ", the virtual assistant 1002 can apply the sequencing by preferring the person objects with the telephone number rather than the person objects whose telephone number is not known have. In one embodiment, the sequencing rules may be associated with domains. For example, different ranking rules may be used for ranking personal variables for e-mail and telephone domains. Those skilled in the art will recognize that any such sorting rule (s) may be created and / or applied in accordance with a particular representation and access to the required context information.

To improve virtual secretary processing Contextual  Use

As described above, the context can be applied to various calculations and inferences with respect to the operation of the virtual secretary 1002. [ Referring now to FIG. 2, there is shown a flow diagram illustrating a method 10 of using a context in various processing steps in a virtual assistant 1002, in accordance with one embodiment.

The method 10 may be implemented in association with one or more embodiments of the virtual assistant 1002.

In at least one embodiment, the method 10 may be implemented to perform various types of functions, operations, actions, and / or other features such as, for example, one or more of the following steps (or combinations thereof) Can:

사용자와 가상 비서(1002) 간의 대화 인터페이스의 인터페이스 제어 흐름 루프를 실행한다. 방법(10)의 적어도 한 번의 반복은 대화에서 플라이(ply)로서 역할을 할 수 있다. 대화 인터페이스는, 대화 방식으로, 발성을 앞뒤로 함으로써 사용자와 비서(1002)가 통신하는 인터페이스이다.

And executes the interface control flow loop of the conversation interface between the user and the virtual secretary 1002. [ At least one iteration of method (10) can act as a ply in the conversation. The conversation interface is an interface through which the user and the secretary 1002 communicate with each other in a conversation manner by moving the talk back and forth.

가상 비서(1002)에 대한 실행적인 제어 흐름을 제공한다. 즉, 절차는 입력의 수집, 입력의 처리, 출력의 생성, 및 출력의 사용자에의 제시를 제어한다.

And provides an executive control flow to the virtual secretary 1002. That is, the procedure controls the collection of inputs, processing of inputs, generation of outputs, and presentation of the outputs to the user.

가상 비서(1002)의 컴포넌트들 간의 통신을 조정한다. 즉, 하나의 컴포넌트의 출력이 다른 하나의 컴포넌트에 피딩되는 장소 그리고 환경으로부터의 전체 입력 및 환경에 대한 액션이 발생할 수 있는 장소를 지시할 수 있다.

And coordinates the communication between the components of the virtual secretary 1002. That is, it can indicate where the output of one component is fed to another component and where all actions from the environment and environment can occur.

In at least some embodiments, portions of the method 10 may also be implemented in other devices and / or systems of a computer network.

According to a particular embodiment, multiple instances or threads of the method 10 may be implemented simultaneously and / or concurrently with the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software have. In at least one embodiment, one or more selected portions of method 10 may be implemented in one or more client (s) 1304, one or more server (s) 1340, and / or combinations thereof.

For example, in at least some embodiments, various aspects, features, and / or functionality of the method 10 may be implemented, implemented, and / or provided by software components, network services, databases, and / or the like, / ≪ / RTI >

According to different embodiments, one or more threads or instances of the method 10 may be used to trigger the initiation of at least one instance of the method 10 (e. G., A minimum threshold criterion) May be initiated in response to the detection of one or more conditions or events that satisfy the criteria. Examples of various types of conditions or events that can trigger the initiation and / or implementation of examples of one or more different threads or instances of a method may include (but are not limited to) one or more of the following :

예를 들어, 다음 중 하나 이상(이에 제한되지 않음)과 같은 가상 비서(1002)의 인스턴스를 갖는 사용자 세션:

For example, a user session with an instance of a virtual secretary 1002, such as but not limited to one or more of the following:

For example, a mobile device application starting up a mobile device application implementing an embodiment of virtual secretary 1002;

A computer application that is running an application that is implementing an embodiment of virtual assistant 1002, for example;

Dedicated buttons on the pressed mobile device, such as "voice input buttons";

Buttons on a peripheral device attached to a computer or mobile device, such as a headset, a telephone handset or base station, a GPS navigation system, a consumer device, any other device having a button that can be associated with calling a remote control or assistant;

A web session initiated from a web browser to a web site that implements the virtual secretary 1002;

An interaction initiated from within an existing web browser session, for example, a web site that implements a virtual secretary 1002 where a virtual secretary 1002 service is requested;

An email message sent to the modality server 1426 that coordinates communications with the implementation of the virtual secretary 1002;

A text message is sent to the modality server 1426 that coordinates communication with the implementation of the virtual secretary 1002;

A phone call is made to the modality server 1434 that coordinates communications with the implementation of the virtual secretary 1002;

An event, such as an alert or a notification, is sent to the application providing the implementation of the virtual secretary (1002).

가상 비서(1002)를 제공하는 디바이스가 턴온 및/또는 시작될 때.

When the device providing the virtual secretary 1002 is turned on and / or started.

According to different embodiments, one or more different threads or instances of method 10 may be initiated and / or implemented manually, automatically, statistically, dynamically, concurrently, and / or in combination. Further, different instances and / or embodiments of the method 10 may be initiated in one or more different time periods (e.g., during a specific time period, at regular periodic intervals, at irregular periodic intervals, Etc).

In at least one embodiment, a given instance of the method 10 may use and / or manipulate various different types of data and / or other types of information when performing particular tasks and / or actions, including contextual data as described above. Or < / RTI > The data may also include any other type of input data / information and / or output data / information. For example, in at least one embodiment, at least one instance of method 10 may access, process, and / or otherwise use information from one or more different types of sources, such as, for example, one or more databases . In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and / or remote memory devices. Also, at least one instance of the method 10 may generate one or more different types of output data / information that may be stored, for example, in local memory and / or remote memory devices.

In at least one embodiment, the initial configuration of a given instance of the method 10 may be performed using one or more different types of initialization parameters. In at least one embodiment, at least some of the initialization parameters may be accessed through communication with one or more local and / or remote memory devices. In at least one embodiment, at least some of the initialization parameters provided to the instance of method 10 may correspond to input data / information, and / or may be derived from input data / information.

In the particular example of FIG. 2, assume that a single user is accessing an instance of virtual secretary 1002 over the network from a client application having voice input capabilities.

The speech input is interpreted by elicitation (100). Induction may include presenting prompts in any suitable mode. In various embodiments, the user interface of the client provides various modes of input. These may include, for example:

액티브 타이핑된 입력(active typed-input) 유도 절차를 호출할 수 있는, 타이핑된 입력을 위한 인터페이스;

An interface for a typed input that can invoke an active typed-input derivation procedure;

액티브 음성 입력 유도 절차를 호출할 수 있는, 음성 입력을 위한 인터페이스;

An interface for voice input, capable of calling an active voice input inducing procedure;

액티브 GUI 기반 입력 유도를 호출할 수 있는, 메뉴로부터 입력들을 선택하기 위한 인터페이스.

An interface for selecting inputs from a menu that can invoke an active GUI-based input derivation.

Techniques for performing each of these are disclosed in the above referenced related patent applications. Those skilled in the art will recognize that other input modes may be provided. The output of step 100 is a set 190 of candidate interpretations of the input speech.

The set of candidate interpretations 190 is processed 200 by a language interpreter 2770 (also referred to as a natural language processor or NLP) and the language interpreter 2770 parses the text input and generates a set of possible interpretations 290 ).

At step 300, the indication (s) 290 of the user's intent is conveyed to a dialog flow processor 2780 which implements an embodiment of the flow analysis procedure and dialogue described with respect to Fig. Conversation flow processor 2780 determines which interpretation is most likely, maps this interpretation to the instances of the domain models and the parameters of the task model, and determines the next flow step in the task flow.

In step 400, the identified flow step is executed. In one embodiment, a call to the flow step is performed by a service organizing component 2782 that invokes a set of services instead of a user request. In one embodiment, these services contribute some data as a common result.

At step 500, a conversation response is generated. At step 700, a response is sent to the client device for output. The client software on the device renders it on the screen (or other output device) of the client device.

After viewing the response, if the user completes (790), the method ends. If the user does not terminate, another iteration of the loop is initiated by returning to step 100.

Context information 1000 may be used by various components of the system at various points in the method 10. For example, as shown in FIG. 2, a context 1000 may be used in steps 100, 200, 300, and 500. Additional explanations of the use of contexts 1000 in these steps are provided below. However, those skilled in the art will recognize that the use of context information is not limited to these specific steps, and that the system can use context information at other points without departing from the essential features of the present invention.

Further, those skilled in the art will appreciate that the different embodiments of method 10 may include additional features and / or operations other than those shown in the specific embodiment shown in FIG. 2, It will be appreciated that at least some of the features and / or operations of the same method 10 may be omitted.

In speech induction and interpretation Context  use

Referring now to FIG. 3, there is shown a flow chart illustrating a method for using context in speech induction and interpretation 100 to improve speech recognition in accordance with an embodiment. Context 1000 may be used for disambiguation in speech recognition, for example, to guide the generation, sequencing, and filtering of candidate hypotheses that match phonemes to words. Different speech recognition systems use various mixtures of productions, sequences and filters, but the context 1000 can generally be applied to reduce the hypothetical space at any stage.

The method begins (100). The virtual secretary 1002 receives a voice or voice input in the form of an auditory signal (121). The speech-to-text conversion service 122 or processor generates a set of candidate text interpretations 124 of the auditory signals. In one embodiment, the voice-to-text conversion service 122 is implemented using a Nuance Recognizer available, for example, from Nuance Communications, Inc. of Burlington, Massachusetts.

In one embodiment, virtual secretary 1002 uses statistical language models 1029 to generate candidate text interpretation 124 of speech input 121. In one embodiment, the context 1000 is applied to bias the generation, filtering, and / or sequencing of the candidate interpretations 124 generated by the voice-to-text conversion service 122. E.g,

음성-텍스트 변환 서비스(122)는 통계 언어 모델들(1029)을 바이어스하기 위하여 사용자 퍼스널 데이터베이스(들)(1058)로부터의 어휘를 사용할 수 있다.

The voice-to-text conversion service 122 may use the vocabulary from the user personal database (s) 1058 to bias the statistical language models 1029.

음성-텍스트 변환 서비스(122)는 커스텀 통계 언어 모델(1029)을 선택하기 위하여 대화 상태 컨텍스트를 사용할 수 있다. 예를 들어, 예/아니오 질문을 할 때, 이들 단어들을 듣는 쪽으로 바이어스되는 통계 언어 모델(1029)이 선택될 수 있다.

The speech-to-text conversion service 122 may use the dialog state context to select a custom statistical language model 1029. [ For example, when asking yes / no questions, a statistical language model 1029 that is biased towards listening to these words may be selected.

음성-텍스트 변환 서비스(122)는 적절한 단어들 쪽으로 바이어스하기 위하여 현재 애플리케이션 컨텍스트를 사용할 수 있다. 예를 들어, "그녀에게 전화하는 것(call her)"는 텍스트 메시지 애플리케이션 컨텍스트에서 "칼라(collar)"보다 선호(prefer)될 수 있는데, 왜냐하면 이러한 컨텍스트는 전화걸 수 있는 사람 객체(Person Object)들을 제공하기 때문이다.

The voice-to-text conversion service 122 may use the current application context to bias towards the appropriate words. For example, "call her" may be preferred over a "collar" in the context of a text message application because such a context may contain a Person object, As shown in FIG.

For example, a given speech input may lead to a speech-to-text conversion service 122 to generate interpretations "call her" and "color". A speech-to-text service 122, guided by statistical language models (SLMs) 1029, is tune to hearing names after hearing a "call " . The voice-to-text conversion service 122 may also be tuned based on the context 1000. For example, if "Herb" is the name of the user address book, then this context can be used to lower the threshold to regard "Herb " as an interpretation of the second syllable. That is, the presence of names in the user's personal data context may affect the adjustment and selection of the statistical language model 1029 used to construct the hypotheses. The name "Herb" may be part of the generic SLM 1029, or may be added directly by the context 1000. In one embodiment, this may be added as an additional SLM 1029, which is adjusted based on the context 1000. In one embodiment, this may be an adjustment of the existing SLM 1029, which is adjusted based on the context 1000.

In one embodiment, statistical language models 1029 may also be used to store words or phrases from application preferences and usage history (s) 1072 and / or personal database (s) 1058 that may be stored in long term personal memory 2754. [ Names, and phrases. For example, the statistical language model 1029 may include to-do items, list items, personal notes, calendar entries, person names of contact / address book, email addresses, contact / The street or city name mentioned, and so on.

The ranking component analyzes the candidate interpretations 124 and orders these candidate interpretations according to how they fit into the syntactic and / or semantic model of the virtual assistant 1002 126). Any source of constraints on user input can be used. For example, in one embodiment, the secretary 1002 can parse how well the interpretations are in a syntactic and / or semantic perspective, a domain model, a task flow model, and / or an interactive model, The output of the voice-to-text interpreter can be sequenced according to whether it is a related US As described in the utility model specification, how many combinations of words in the candidate interpretation 124 fit into the concepts, relationships, entities, properties of the active ontology and their associated models .

The sequencing 126 of candidate interpretations may also be influenced by the context 1000. For example, if the user is currently delivering a conversation to the text messaging application when the virtual assistant 1002 is invoked, the phrase " call her "is more likely to be a better interpretation than the word " collar" This is because there is a potential "her" to call in this context. This bias may be used to pre-empt the dominant phrases such as "call her" or "call <contact name>" when the current application context indicates an application capable of providing "callable entities & (126). &Lt; / RTI &gt;

In various embodiments, the algorithm or procedure used by the secretary 1002 for the interpretation of textual inputs, including any embodiment of the natural language processing procedure shown in FIG. 3, is performed by the voice-to-text translation service 122 May be used to sequence and score the generated candidate text interpretations (124).

Context 1000 may also be used to filter candidate interpretations 124 instead of or in addition to affecting or limiting the sequencing of candidate interpretations 124. For example, the filtering rule should be considered the highest candidate 130, even if the context of the address book entry for "Herb ", and the phrase containing this context, is otherwise below the filtering threshold It can be stipulated to be sufficient. Depending on the particular speech recognition technique being used, constraints based on the bias of the context may be applied to the generation, sequencing, and / or filtering steps.

In one embodiment, if the ranking component 126 determines 128 that the highest ranking speech interpretation from the interpretations 124 is sequenced above a specified threshold, then this highest ranking interpretation may be automatically selected 130 ). If no interpretation is sequenced beyond a specified threshold, possible candidate interpretations 134 of speech are provided to the user (132). The user can then select among the displayed choices (136).

Referring now to Figures 26A and 26B, there is shown a screen shot illustrating an example of a user interface for selection among candidate analyzes, in accordance with one embodiment. 26A shows an indication of a user's voice with a dotted line under ambiguous interpretation 2651. FIG. If the user taps on the text, alternative interpretations 2652A and 2652B are shown, as shown in Figure 26B. In one embodiment, the context 1000 includes a selection of candidate interpretations 2652A and 2652B in a finite set of proposed alternatives as shown in Figure 26 (b) (which is shown in the initial default in Figure 26a) .

In various embodiments, the user selection 136 among the displayed options may be accomplished by any input mode, including, for example, multi-mode inputs. Such input modes include, but are not limited to, actively elicited typed input, aggressive inductive voice input, aggressive GUI for input, and / or the like. In one embodiment, the user may make a choice among the candidate interpretations 134, by example, tapping or speaking. In the case of speech, the possible interpretation of the new speech input is highly constrained by the small set of choices 134 provided.

The resulting one or more text interpretations (s) 190 are returned between whether the input is automatically selected (130) or selected (136) by the user. In at least one embodiment, this returned input is annotated so that information about what choice was made in step 136 is retained along with the input of the context. This improves the accuracy of subsequent language interpretation, for example, by allowing a semantic concept or entity underlying the string to be associated with the string at this string return.

Any of the sources described in connection with FIG. 1 may provide the context 1000 to the speech derivation and interpretation method shown in FIG. For example:

개인 음향 컨텍스트 데이터(Personal Acoustic Context Data; 1080)가, 가능한 SLM(1029)들로부터 선택을 하거나 다른 경우 이들 SLM을 인식된 음향 컨텍스트들에 최적화시키도록 튜닝하는 데에 이용될 수 있다.

Personal Acoustic Context Data 1080 can be used to select from possible SLMs 1029 or otherwise tune to optimize these SLMs to recognized acoustic contexts.

사용 중인 마이크로폰 및/또는 카메라의 속성들을 기술하는 디바이스 센서 데이터(1056)가, 가능한 SLM(1029)들로부터 선택을 하거나 다른 경우 이들 SLM을 인식된 음향 컨텍스트들에 최적화시키도록 튜닝하는 데에 이용될 수 있다.

Device sensor data 1056 describing the properties of the microphone and / or camera in use may be used to tune to select from possible SLMs 1029 or otherwise optimize these SLMs to recognized acoustic contexts .

개인 데이터베이스(1058) 및 애플리케이션 선호도 및 사용 히스토리(1072)로부터의 어휘(Vocabulary)가 컨텍스트(1000)로서 이용될 수 있다. 예를 들어, 미디어 명칭 및 아티스트의 이름이 언어 모델(1029)을 튜닝하는 데에 이용될 수 있다.

The personal database 1058 and the vocabulary from the application preferences and usage history 1072 may be used as the context 1000. For example, the name of the media and the name of the artist may be used to tune the language model 1029.

대화 히스토리 및 보조 메모리(1052)의 일부인 현재의 대화 상태(Current dialog state)가 음성-텍스트 변환 서비스(122)에 의해 후보 해석들(124)의 생성/필터링/서열화를 바이어스하는 데에 이용될 수 있다. 예를 들어, 대화 상태의 한 종류는 예/아니오 질문을 하는 것이다. 이러한 상태일 때, 절차(100)는 이들 단어를 듣는 방향으로 바이어싱하는 SLM(1029)을 선택할 수 있거나, 122에서 컨텍스트-특정 조정으로 이들 단어의 서열화 및 필터링을 바이어싱할 수 있다.

The current dialog state that is part of the conversation history and auxiliary memory 1052 can be used to bias generation / filtering / sequencing of candidate interpretations 124 by the voice-to-text conversion service 122 have. For example, one type of conversation state is asking yes / no questions. In this state, the procedure 100 may select the SLM 1029 which biases these words in the listening direction, or may bias the ranking and filtering of these words in a context-specific adjustment at 122.

In natural language processing Context  use

Context 1000 can be used to facilitate natural language processing (NLP) - parsing text input into a semantic structure representing possible parse. Referring now to FIG. 4, there is shown a flowchart illustrating a method for using a context in natural language processing, which may be performed by a language interpreter 2770, in accordance with an embodiment.

The method starts at 200. The input text 202 is received. In one embodiment, input text 202 is matched 210 to words and phrases using pattern recognizer 2760, lexicon database 2758, ontologies and other models 1050, And associations between concepts. Step 210 produces a set of candidate syntactic parses 212 that match for semantic relevance 220 that yields candidate semantic puzzles 222. The candidate parses are then processed at 230 to remove ambiguous alternatives, filtered, sorted by relevance (232), and returned.

Throughout the natural language processing, context information 1000 can be applied to reduce hypothesis space and constraintable parses. For example, if the language interpreter 2770 has received two candidates "call her" and "call Herb", the language interpreter 2770 receives a binding 212 for the words "call", "her", and "Herb" ). The application context 1060 can be used to constrain the possible word meaning for "call " to mean" phone call. &Quot; The context can also be used to find referents for "her" and "Herb ". For "her, " context sources 1000 may be searched to find sources of entities that can dial. In this example, the subject of the text messaging conversation is an entity that can dial, and this information is part of the context coming from the text messaging application. In the case of "Herb ", the address book of the user is used for application preferences (such as the most preferred numbers from the domain entity database 2772) and application usage (such as recent calls from the domain entity database 2772) It is the source of a disambiguating context, as in other personal data such as history. In this example, if the current text messaging is Rebec-caRichards and there is a HerbGowen in the address book of this user, the two parses generated by the language interpreter 2770 are "Phone-Call (Rebecca Richards)" and "PhoneCall (HerbGowen) ".

Data from the application preferences and usage history 1072, conversation history and auxiliary memory 1052, and / or personal database 1058 are used by language interpreter 2770 to generate candidate syntactic parses 212 It can also be used. Such data may be obtained, for example, from short term memory and / or long term memory 2752, 2754. In this manner, information known to the user and / or to the input that was previously provided in the same session may be used to enhance performance, reduce ambiguity, and enhance the interactive nature of the interaction. In addition, data from active ontology 1050, domain models 2756, and task flow models 2786 to implement evidential reasoning in determining valid candidate syntactic parses 212 Can be used.

In semantic matching 220, language interpreter 2770 considers the combination of possible parsing results according to how well they match semantic models such as domain models and databases. Semantic matching 220 may use data from, for example, active ontology 1050, short-term private memory 2752, and long-term private memory 2754. For example, the semantic matching 220 can be used to identify the locations of the conversations or previous references to the local events (conversation history and auxiliary memory 1052) or the individual's most preferred locations (application preferences and usage history 1072) Can be used. The semantic matching step 220 also uses the context 1000 to interpret the phrases into domain intent structures. A set of candidates, or potential semantic parsing results is generated 222.

In the disambiguation step 230, the language interpreter 2770 focuses on the proof strength of the candidate semantic pars results 222. Clarification 230 relates to reducing the number of candidate semantic parses 222 by eliminating unwanted or redundant alternatives. Clarification 230 may utilize data from, for example, the structure of an active ontology 1050. In at least one embodiment, connections between nodes in the active ontology provide proof support for clarification among the candidate semantic pars results 222. [ In one embodiment, the context 1000 is used to assist with such things as clarification. Examples of such clarification include: determining one of several persons having the same name; Determining a referent to a command such as "reply" (e-mail or text message); And pronoun dereferencing of pronouns.

For example, an input such as "call Herb" potentially represents any entity that matches the "hub ". Such entities may be any number in the database of business names from personal databases 1058 and / or domain entity database 2772, as well as the address book (personal databases 1058) of the user. Some sources in the context may limit and / or rank and match the sets that match the "hubs " at step 232. E.g:

인기 전화 번호 리스트 상에 있거나 최근에 호출된 허브와 같은 다른 애플리케이션 선호들 및 사용 히스토리(1072), 또는 텍스트 메세지 대화 또는 이메일 스레드(email thread)에 대한 최근 단체(party);

Recent application preferences and usage history 1072, such as a hub that is on the popular phone number list or recently called, or a recent party to a text message conversation or email thread;

아버지 또는 형제와 같이, 관계로서 이름지어진 허브같은, 개인 데이터베이스(1058)에서 언급된 허브, 또는 최근 캘린더 이벤트의 리스트된 참석자. 태스크가 전화 호출 대신에 미디어 재생인 경우, 미디어 제목, 제작자 등으로부터의 이름들은 제한의 소스들이 될 수 있다;

A hub referred to in a personal database 1058, such as a hub named as a relationship, such as a father or a sibling, or a listed attendee of a recent calendar event. If the task is media playback instead of a phone call, names from the media title, author, etc. may be sources of restrictions;

요청이나 결과들 내의 대화(1052)의 최근 플라이(ply). 예를 들어, 도 25a 내지 25b에 연계하여 위에서 설명된 바와 같이, 존(John)으로부터의 이메일을 검색한 후에, 대화 컨텍스트 내에 여전히 있는 검색 결과를 이용하여, 사용자는 답변을 작성할 수 있다. 비서(1002)는 특정한 애플리케니션 도메인 객체 컨텍스트를 식별하기 위해 대화 컨텍스트를 사용할 수 있다.

Recent ply of conversation 1052 in request or results. For example, after retrieving an email from John, as described above in connection with Figures 25a-25b, the user can create a reply using the search results that are still in the conversation context. Secretary 1002 may use the conversation context to identify a particular application domain object context.

Also, the context 1000 may help to reduce ambiguity in words other than proper names. For example, if the user of the email application (as shown in FIG. 20) speaks to the secretary 1002 as "reply, " the context of the application is the email response Helping to determine that they should be associated.

At step 232, the language interpreter 2770 filters and classifies the highest semantic pars as a representation of the user intent 290 (232). Context 1000 may be used to announce such filtering and classification 232. The result is a representation of the user intent 290.

task Flow  In treatment Context  use

Referring now to FIG. 5, and in accordance with one embodiment, as may be done by the conversation flow processor 2780, A flow diagram illustrating a method of using a context in task flow processing is shown. In task flow processing, the candidate fragments generated from the method of FIG. 4 are sequenced and illustrated to produce task descriptions of the actions that can be performed.

The method begins (300). A number of candidate representations of user intent 290 are received. In one embodiment, as described in connection with FIG. 4, the expressions of user intent 290 comprise a set of semantic pars.

In step 312, the conversation flow processor 2780 determines a preferred interpretation of the semantic pars (s), along with other information for determining the task and its parameters to perform, based on the determination of the user's intention . The information may be obtained, for example, from domain models 2756, task flow models 2786 and / or conversation flow models 2787, or a combination thereof. For example, the task may be a telephone call, and the task parameter is a telephone number for calling.

In one embodiment, the context 1000 is used in an execution step 312 to guide combinations of parameters 312 by deducing initial values to resolve ambiguities. For example, the context 1000 may guide an example of a task description and determining whether there is a best interpretation of the user's intent.

For example, suppose the Intent inputs 290 are "RebeccaRichards" and "HerbGowen." The dialing task requires the PhoneNumber parameter. Some of the sources of the context 100 are Rebecca And an address book entry for Rebecca in the contact database has two telephone numbers, entries for Herb do not have telephone numbers, and one for Herb. Context information 1000 from a personal database 1058, such as a contact database, allows virtual assistant 1002 to favor Rebecca over Herb, which is Rebecca's phone number, In order to determine which phone number to use for Rebecca, the application context 1060, You can consult with Rebecca to choose the number used to carry out the text messaging conversation, so "call her" in the context of a text messaging conversation with Rebecca Rechards calls the mobile phone Rebecca uses for text messaging The virtual secretary 1002 can determine that it is a girl. This particular information is returned at step 390.

Context 1000 may be used for anything other than reducing the ambiguity of the telephone number. It can be used whenever there are a number of possible values for a task parameter, as long as any source of the context 1000 having a value for the task parameter is available. Other examples in which the context 1000 can reduce ambiguity (and avoid having the user choose between candidates); Email addresses; Physical addresses; Time and dates; Place; List name; Media titles; Artist names; Business names; Or any other value space, without limitation.

Other types of reasoning needed for task flow processing 300 may also benefit from context 1000. For example, the initial value inference can use the current position time and other current values. The initial value inference can be useful in determining the values of the task parameters implied in the user's request. For example, if someone says, "How is the weather?", That implies what the current weather is in the vicinity.

In step 310, the conversation flow processor 2780 determines whether this interpretation of user intent is strongly supported and / or supported by alternative ambiguous parcels to proceed. If there is a compelling ambiguity or sufficient uncertainty, then step 322 is performed to set up a dialog flow step so that the action step causes the dialog to output a prompt for more information from the user. A screen shot is shown in FIG. 14 that allows the user to resolve ambiguities. Context 1000 may be used in step 322 to classify and annotate the displayed menu of candidate items that the user selects.

At step 320, the task flow model is consulted to determine an appropriate next step. The information may be obtained, for example, from domain models 2756, task flow models 2786 and / or conversation flow model 2787, or any combination thereof.

The result of step 320 or step 322 is a representation of the user's request 390 which may include sufficient task parameters for the conversation flow processor 2780 and service combination 2782 to send to the appropriate service .

To improve conversation creation Contextual  use

During the conversation response creation 500, the secretary 1002 may again paraphrase an understanding of its intent to the user and how it behaves in the task. An example of such an output is "Okay, I'll call Rebecca on her mobile phone ...". This allows the user to authenticate the secretary 1002 to perform associated task automation such as placing a call. In the dialog creation step 500, the secretary 1002 determines how much detail the user can convey to the user in expressing the understanding of the user's intention.

In one embodiment, context 1000 can also be used to filter based on previous output (to avoid repeating information) as well as to guide selection of an appropriate level of detail in conversation . For example, the secretary 1002 can take advantage of the fact that the person and phone number can be deduced from the context 1000 to determine whether to mention the name and the level of detail. Examples of rules that may be applied include, without limitation, the following:

컨텍스트에 의해 대명사가 해결되었을 때, 전화 걸 사람의 이름을 언급하라.

When the pronoun is resolved by context, mention the name of the person to call.

사람이 텍스트 메세지와 같은 유사 컨텍스트로부터 추론되는 경우, 이름(first name)만을 사용하라.

If a person is inferred from a similar context, such as a text message, use only the first name.

전화 번호가 애플리케이션 또는 개인 데이터 컨텍스트로부터 추론되는 경우, 다이얼 할 실제 번호보다 "이동 전화"와 같은 전화 번호의 상징적 이름을 사용하라.

If the phone number is deduced from the application or personal data context, use the symbolic name of the phone number, such as "mobile phone", rather than the actual number to dial.

In addition to guiding the appropriate level of detail, the context 1000 may also include, for example, filtering out previous utterances to avoid repetition, and displaying previously mentioned entities in a conversation May be used in the dialogue generation step 500.

Those skilled in the art will recognize that the context 1000 may be used in other ways. For example, in conjunction with the techniques described herein, the context 1000 may be incorporated into the "Contextual Voice Commands ", Attorney Docket No. P7393US1, filed June 5, 2009, May be utilized in accordance with the mechanism described in related US utility application No. 12 / 479,477.

Context  Collection and communication Mechanism

In various embodiments, different mechanisms are used to collect and communicate contextual information within virtual secretary 1002. [ For example, in one embodiment, the virtual secretary 1002 is implemented in a client / server environment and its services are distributed between the client and the server, and the sources of the context 1000 may be distributed.

6, an example of the distribution of the sources of context 1000 between client 1304 and server 1340 is shown, according to one embodiment. Client device 1304, which may be a mobile computing device or other device, may be the source of context information 1000, such as device sensor data 1056, current application context 1060, event context 2706, and the like. Other sources of context 1000 may be distributed over some combination of client 1304 or server 1340 or both. Examples include application preferences and usage histories 1072c and 1072s; Conversation history and auxiliary memory 1052c and 1052s; Personal databases 1058c and 1058s; And personal acoustic context data 1080c, 1080s. In each of these examples, the sources of context 1000 may reside on server 1340, client 1304, or both. Moreover, as described above, the various steps shown in FIG. 2 may be performed by some combination of client 1304 or server 1340, or both.

In one embodiment, context 1000 may be communicated between distributed components, such as client 1304 and server 1340. Such communication may be by a local API or a distributed network, or some other means.

Referring now to Figures 7A-7D, there are illustrated event diagrams illustrating examples of mechanisms for obtaining and coordinating contextual information 1000, in accordance with various embodiments. Various techniques are available for loading or communicating contexts and are available to the virtual secretary 1002 when needed or useful. Each of these mechanisms includes an operation of the virtual secretary 1002; Device or application initialization 601; Initial user input 602; Initial input processing 603; And the context-dependent processing 604. [0060]

7A illustrates an approach in which context information 1000 is loaded using a "pull" mechanism when user input 602 is initiated. If the user calls virtual assistant 1002 and provides at least some input 602, virtual assistant 1002 loads context 1000 (610). Loading 610 may be performed by requesting or retrieving context information 1000 from an appropriate source. The input process 603 begins when the context 1000 is loaded (610).

7B illustrates an exemplary case where the context information 1000 is loaded (620) when a device or application is initialized (601), when the user input is initiated (602), additional context information (1000) Lt; / RTI &gt; In one embodiment, the context information 1000 loaded in initialization 620 includes a static context (i.e., a context that does not change often); When user input is initiated (602), the context information (1000) that is loaded (621) includes a dynamic context (i.e., a context that may be changed due to the static context being loaded (620)). Such an approach can improve performance by eliminating the cost of loading static context information 1000 from the runtime capabilities of the system.

Figure 7c shows a variation of the approach of Figure 7b. In this example, the dynamic context information 1000 allows the continuation of the loading 621 after the input process has begun (603). Thus, the loading 621 may occur in parallel with the input processing. The virtual secretary 1002 procedure is only blocked at step 604 when the process is dependent on the received context information 1000.

Figure 7d shows a fully configurable version that deals with the context in any of the following five different ways:

정적 컨텍스트(static contextual) 정보(1000)는 컨텍스트 소스에서 가상 비서(1002)를 실행하는 환경 또는 디바이스로의 일 방향으로 동기화(640)된다. 데이터가 컨텍스트 소스에서 변경될 때, 그 변경은 가상 비서(1002)에게 푸시(push)된다. 예컨대, 어드레스 북은 초기에 생성 또는 인에이블될 때 가상 비서(1002)에 동기화된다. 어드레스 북이 수정될 때마다, 변경은 가상 비서(1002)에게 즉시 또는 일괄 접근 방식으로 푸시된다. 도 7d에 도시된 바와 같이, 이런 동기화(640)는 사용자 입력이 시작(602)되기 전을 포함하는 어느 때나 일어날 수 있다.

Static contextual information 1000 is synchronized 640 in one direction to the environment or device executing the virtual secretary 1002 at the context source. When the data is changed at the context source, the change is pushed to the virtual secretary 1002. For example, the address book is synchronized to the virtual secretary 1002 when it is initially created or enabled. Each time the address book is modified, the changes are pushed to the virtual secretary 1002 either immediately or in a batch approach. As shown in FIG. 7D, this synchronization 640 may occur at any time, including before the user input begins (602).

일 실시예에서, 사용자 입력이 시작(602)될 때, 정적 컨텍스트 소스는 동기화 상태를 체크할 수 있다. 필요한 경우, 나머지 정적 컨텍스트 정보(1000)를 동기화하는 프로세스가 시작된다(641).

In one embodiment, when user input is initiated (602), the static context source may check the synchronization state. If necessary, the process of synchronizing the remaining static context information 1000 begins (641).

사용자 입력이 시작(602)될 때, 일부 동적 컨텍스트(1000)는 사실상 610 및 621에서 로딩(642)된다. 컨텍스트(1000)를 소비하는 절차들은 이들이 필요로 하는 아직까지 로딩되지 않은 컨텍스트 정보(1000)를 대기하도록 단지 차단된다.

When the user input is started 602, some dynamic context 1000 is actually loaded (642) at 610 and 621. Procedures consuming the context 1000 are simply blocked to wait for context information 1000 that has not yet been loaded that they need.

다른 컨텍스트 정보(1000)는 이들이 필요로 할 때 프로세스에 의해 온디맨드(On demand)(643)식으로 로딩된다.

Other context information 1000 is loaded on demand 643 by the process when they are needed.

이벤트 컨텍스트(2706)는 이벤트가 일어날 때 소스에서 가상 비서(1002)를 실행하는 디바이스로 전송(644)된다. 이벤트 컨텍스트(2706)를 소비하는 프로세스는 준비될 이벤트의 캐시를 단지 대기하며, 그 이후 어떠한 시간도 차단함이 없이 처리될 수 있다. 이런 식으로 로딩된 이벤트 컨텍스트(2706)는 다음 중 어느 하나를 포함할 수 있다:

The event context 2706 is transmitted (644) from the source to the device executing the virtual secretary 1002 when the event occurs. The process consuming the event context 2706 may just be waiting for a cache of events to be prepared and then processed without blocking any time thereafter. The event context 2706 loaded in this way may include any of the following:

사용자 입력이 시작(602)되기 전에 로딩된 이벤트 컨텍스트(2706), 예컨대, 비판독 메시지 통지. 이런 정보는 예컨대 동기화된 캐시를 이용하여 유지될 수 있다.

A loaded event context 2706, e.g., a non-read message notification, before user input is initiated (602). This information can be maintained, for example, using a synchronized cache.

사용자 입력의 시작(602)과 동시 또는 그 이후에 로딩된 이벤트 컨텍스트(2706). 예컨대, 사용자가 가상 비서(1002)와 대화중, 텍스트 메시지가 도착할 수 있다; 비서(1002)에게 이런 이벤트를 통지하는 이벤트 컨텍스트는 비서(1002) 처리에 병렬로 푸시될 수 있다.

Event context 2706 loaded at or after the start (602) of user input. For example, a text message may arrive while the user is talking to the virtual secretary 1002; The event context for notifying the secretary 1002 of such an event can be pushed in parallel to the secretary 1002 process.

In one embodiment, the flexibility to obtain and adjust the context information 1000 is such that, for each source of context information 1000, the balance of communication costs for values with information available to all requests API, and communication policy. For example, device sensor data 1056 or personal acoustic context data 1080 describing the parameters associated with all voice-to-text conversion requests, i.e., the parameters of the microphone, may be loaded for every request. Such a communication policy can be specified, for example, in a configuration table.

Referring to FIG. 9, an example of a configuration table 900 that may be used to specify a caching policy and communication for various sources of context information 1000, according to one embodiment, is shown. For each of a number of different context sources, including a user name, an address book name, an address book number, an SMS event context, and a calendar database, the specific type of context loading is specified in each of the steps of Figure 2: Induce and interpret (100); Interpret natural language (200); Identify a task (300); A dialog response is generated (500). Each entry in the table 900 represents one of the following:

동기화(Sync): 컨텍스트 정보(1000)는 디바이스 상에서 동기화된다.

Sync: The context information 1000 is synchronized on the device.

온 디맨드(On demand): 컨텍스트 정보(1000)는 가상 비서(1002)의 요청에 응답하여 제공된다.

On demand: Context information 1000 is provided in response to a request of the virtual secretary 1002.

푸시(Push): 컨텍스트 정보(1000)는 디바이스에 푸시된다.

Push: Context information 1000 is pushed to the device.

In a fully configurable manner, a large space of potentially related context information 1000 enables streamline natural language interaction between a human and a machine. Rather than loading all of this information at all times that may result in incompatibility, some information is maintained in both the context source and the virtual secretary 1002, while other information is queried on demand. For example, as described above, information such as names used in real-time operations such as voice recognition is maintained locally, while information used only by some possible requests, such as a user's personal calendar, is queried on demand. Data that can not be expected when a user invokes a secretary, such as an incoming SMS event, is pushed when the event occurs.

10, an event diagram illustrating an example of accessing the context information source configured in FIG. 9 during the processing of an interactive sequence in which a secretary 1002 interacts with a user, according to one embodiment 950 are shown.

The sequence shown in Figure 10 represents the following interactive sequence:

T₁: 비서(1002): "안녕 스티브, 제가 무엇을 도와드릴까요?"

T ₁ : Secretary (1002): "Hi Steve, what can I do for you?"

T₂: 사용자: "나의 다음 미팅은 언제인가?"

T ₂ : User: "When is my next meeting?"

T₃: 비서(1002): "너의 다음 미팅은 중역 회의실에서 오후 1시입니다"

T ₃ : Secretary (1002): "Your next meeting is at 1:00 pm in the boardroom."

T₄: [수신 SMS 메시지의 사운드]

T ₄ : [Sound of incoming SMS message]

T₅: 사용자:"그 메시지를 나에게 읽어라"

T ₅ : User: "Read the message to me"

T₆: 비서(1002): "존슨으로부터의 너의 메시지는 '점심 어떻게 할까'"입니다"

T ₆ : Secretary (1002): "Your message from Johnson is" how to do lunch "

T₇: 사용자: "존슨에게 오늘은 점심 같이 할 수 없다고 말해라"

T ₇ : User: "Tell Johnson I can not do lunch today"

T₈: 비서(1002): "OK, 그에게 말할 것이다"

T ₈ : Secretary (1002): "OK, I will tell him"

At time T _0, before the conversation is started, the user name is synchronized (770), and the address book name is synchronized (771). This is an example of a static context that is loaded at initialization time as shown in element 640 of Figure 7d. This enables the secretary 1002 to call the user with his or her last name ("Steve").

At time T ₁ , synchronization steps 770 and 771 are complete. At time T ₂ , the user refers to a request being processed in accordance with steps 100, 200 and 300 of FIG. In the task identification step 300, the virtual secretary 1002 queries 774 the user personal database 1058 as the source of the context 1000: in particular, the virtual secretary 1002 sends an on demand And requests information from the user's calendar database configured for access. At time T _3, the step 500 is performed a dialog response is generated.

At time T ₄ , an SMS message is received: this is an example of an event context 2706. Based on the configuration of the table 900, a notification of the event is pushed to the virtual secretary 1002 (773).

At time T ₅ , the user requests the virtual secretary 1002 to read the SMS message. The presence of the event context 2706 guides the NLP component in step 200 to interpret the "message" as a new SMS message. At time T ₆ , step 300 may be performed by the task component to invoke the API to read the SMS message to the user. At time T ₇ , the user makes a request with an ambiguous verb ("tell") and a name ("Johnny"). The NLP component interprets the natural language 200 by resolving this ambiguity using various sources of the context 1000 including the event context 2706 received at step 773: it speaks to the NLP component, To refer to an SMS message from his personal name Johnney. T in step _7, the execution of a step 771 for matching the name by the numbers used as the search from the received event context object flow step 400 is performed. Secretary 1002, and thus to create a new SMS message, and transmits it to Johnny (Johney) as identified in step T _8.

The present invention will be described in detail with respect to possible embodiments. It will be understood by those skilled in the art that other embodiments are possible. First, the particular naming of a component, the capitalization of a term, an attribute, a data structure, or any other programming or structural aspect is not mandatory or important, and the mechanism implementing the invention or its features may have a different name, format or protocol . Moreover, the system may be implemented in hardware or software combination as described above, or as a whole as a hardware element or as a whole as a software element. In addition, the functionality specific partitioning between the various system components described above is merely exemplary and not mandatory: the functionality performed by a single system component may rather be performed by multiple components, and the functionality performed by multiple components is rather simple Component. &Lt; / RTI &gt;

In various embodiments, the present invention may be implemented in a single or in any combination as a system or method for performing the techniques described above. In another embodiment, the invention is a computer program product comprising a non-transitory computer-readable storage medium and computer program code encoded on the medium, causing the processor in the computing device or other electronic device to perform the above-described techniques Can be implemented.

Reference in the specification to "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase "one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

Some of the foregoing descriptions are presented in terms of algorithms and symbols of operation on in-memory data bits of a computing device. These algorithmic descriptions and representations are those used by those skilled in the art to most effectively convey the substance of an operation to others skilled in the art. The algorithm is generally considered to be a consistent sequence of steps (instructions) that result in a predetermined result. This step is a step requiring physical manipulation of the physical quantity. Typically, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals that can be stored, transferred, combined, compared, and otherwise manipulated. It is always convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. for reasons of common usage. Moreover, it is always convenient to mention a certain configuration of a step requiring physical manipulation of physical quantities as a module or code device without loss of generality.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, the discussion utilizing terms such as "processing", "computing", "computing", "display", or " It should be understood that reference is made to the operations and processes of a computer system or similar electronic computing module and / or device that manipulates and transforms data represented as physical (electronic) quantities within other devices, such as an information storage, transmission or display device .

Certain aspects of the invention include process steps and instructions disclosed in the form of algorithms. It should be noted that process steps and instructions of the present invention may be implemented in software, firmware, and / or hardware, and may be downloaded and operated to reside on different platforms used by various operating systems when implemented in software.

The present invention also relates to an apparatus for performing the above-mentioned operations. Such a device may be specially configured for the required purpose, or it may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored on the computing device. Such computer programs include, but are not limited to, any type of computer readable medium, such as a floppy disk, optical disk, CD-ROM, magneto-optical disk, read only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical card, Readable storage medium, such as any type of disk, including a medium, and a respective medium coupled to a computer system bus. Moreover, the computing devices referred to herein may comprise a single processor, or may have an architecture employing a multiprocessor design for increased computing capability.

The algorithms and displays disclosed herein are not inherently related to any particular computing device, virtualization system, or other device. A variety of general purpose systems may be used as a program in accordance with the teachings disclosed or it may prove more convenient to construct a more specific apparatus to perform the required method steps. The structures required for various such systems will be apparent from the foregoing description. Furthermore, the present invention has not been described with reference to any particular programming language. It should be appreciated that various programming languages may be used to implement the teachings of the invention described above, and that references to particular languages are provided to disclose the best mode possible of the invention.

Thus, in various embodiments, the invention may be implemented as software, hardware, and / or other components for controlling a computer system, a computing device, or other electronic device, or any combination thereof. Such an electronic device may be, for example, a processor, an input device (e.g., a keyboard, a mouse, a touchpad, a trackpad, a joystick, a trackball, a microphone, and / or any combination thereof) in accordance with techniques known in the art. , Output devices (e.g., a screen, speakers, etc.), memory, long-term storage (e.g., magnetic storage, optical storage, etc.), and / or network connectivity. Such electronic devices may be portable or non-portable. Examples of electronic devices that may be used to implement the invention include mobile phones, personal digital assistants, smart phones, kiosks, desktop computers, laptop computers, tablet computers, home appliances, home entertainment devices, music players, , A set-top box, an electronic game unit, and the like. An electronic device embodying the invention may use any operating system, such as iOS or MacOS available from Apple, Inc. of Cupertino, CA, or any other operating system suitable for use with the device.

Although the present invention has been described with a limited number of embodiments, it will be understood by those skilled in the art that other embodiments may be devised without departing from the scope of the invention disclosed herein. Also, the language used herein is selected for readability and tooling purposes, and is not intended to limit or inhibit the subject matter of the present invention. Accordingly, the disclosure of the present invention is intended to be illustrative and not to limit the scope of the invention as hereinafter claimed.

Claims (33)

43. A computer readable storage medium having encoded computer program code for implementing a method for interpreting user input to perform a task on a computing device having at least one processor,
The computer program code comprising at least one processor,
Causing the output device to prompt a user for input;
Receiving spoken user input via an input device;
The method comprising: receiving context information from a context source, the context information including acoustic environment data describing an acoustic environment in which the voice user input is received;
Analyzing the received voice user input to derive a representation of a user's intent, the interpreting step comprising: based on the acoustic environment data of the received context information to derive an expression of the user's intent Disambiguating the received voice user input;
Identifying at least one task and at least one parameter for the task based at least in part on the derived representation of the user intent;
Executing at least one task using the at least one parameter to derive a result;
Generating an interactive response based on the derived result; And
And causing the output device to output the generated dialog response, the method comprising:
Wherein the computer program code further comprises at least one processor configured to cause the output device to prompt a user for input using the received context information, interpret the received voice user input, Identifying at least one parameter for the task and the task, and generating an interactive response. &Lt; Desc / Clms Page number 22 &gt; A system for interpreting user input to perform a task,
An output device configured to provide a prompt to the user for input;
An input device configured to receive a voice user input;
At least one processor communicatively coupled to the output device and the input device,
/ RTI &gt;
Wherein the at least one processor comprises:
Receiving context information from a context source, the context information including acoustic environment data describing an acoustic environment in which the voice user input is received;
Analyzing the received voice user input to derive a representation of the user's intent, the interpretation including interpreting the received voice user input based on the acoustic environment data of the received context information, Including disambiguating input;
Identifying at least one task and at least one parameter for the task based at least in part on the derived representation of the user intent;
Executing at least one task using at least one parameter to derive a result; And
And generate an interactive response based on the derived result,
The output device is further configured to output a generated conversation response,
Using the received context information to provide the user with a prompt for input, interpreting the received voice user input, identifying at least one task and at least one parameter for the task, And generating a conversation response is performed. delete 3. The method of claim 2,
Wherein the output device is configured to provide a prompt to the user via an interactive interface,
Wherein the input device is configured to receive the voice user input via the interactive interface,
Wherein the at least one processor is configured to convert the speech user input to a textual representation. 5. The apparatus of claim 4, wherein the at least one processor is further configured to generate a plurality of candidate text interpretations of the voice user input and at least rank the subset of the generated candidate text interpretations, And configured to convert user input to a textual representation,
Wherein at least one of generating and sequencing is performed using the received context information. 6. The method of claim 5, wherein the received context information used in at least one of generating and sequencing comprises: the acoustic environment data describing an acoustic environment in which the speech user input is received; A vocabulary acquired from a database associated with the user, a vocabulary associated with an application preference, a vocabulary obtained from a usage history, and a current conversation state. 3. The system of claim 2, wherein the output device is configured to provide a prompt to a user by generating at least one prompt based at least in part on the received context information. 3. The method of claim 2, wherein the at least one processor is configured to disambiguate the received voice user input based on the acoustic environment data of the received context information to determine the received context information based at least in part on the received context information A user input interpretation system configured to derive a user's intent by performing natural language processing on a voice user input. 9. The method of claim 8, wherein the received context information used to disambiguate the received voice user input includes data describing an event, an application context, an input previously provided by the user, And at least one selected from the group consisting of location, date, environmental condition, and history. 3. The method of claim 2, wherein the at least one processor is configured to determine at least one task and at least one parameter for the task based at least in part on the received context information to determine at least one task and at least one And to identify parameters of the user input. 11. The method of claim 10, wherein the received context information used to identify at least one task and at least one parameter for the task comprises at least one of: data describing an event; data from a database associated with the user; At least one selected from the group consisting of received data, application context, input previously provided by the user, known information about the user, location, date, environmental conditions, and history. 3. The system of claim 2, wherein the at least one processor is configured to generate an interaction response based at least in part upon the received context information. 13. The method of claim 12, wherein the received context information used to generate the conversation response comprises data from a database associated with the user, an application context, an input previously provided by the user, known information about the user, Condition, and history of the user input. 3. The system of claim 2, wherein the received context information includes at least one selected from the group consisting of context information stored in a server and context information stored in a client. 3. The system of claim 2, wherein the at least one processor is configured to receive context information from a context source by requesting the context information from a context source and receiving the context information in response to the request. 3. The system of claim 2, wherein the at least one processor is configured to receive context information from a context source by receiving at least a portion of the context information prior to receiving the voice user input. 3. The system of claim 2, wherein the at least one processor is configured to receive context information from a context source by receiving at least a portion of the context information after receiving the voice user input. 3. The method of claim 2, wherein the at least one processor receives static context information as part of an initialization step and receives context information from a context source by receiving additional context information after receiving the speech user input A user input interpretation system. 3. The method of claim 2, wherein the at least one processor is configured to receive context information from a context source by receiving a push notification of the change in the context information and updating locally stored context information in response to the push notification Configured user input interpretation system. 3. The method of claim 2, wherein the output device, the input device, and the at least one processor are selected from the group consisting of a phone, a smartphone, a tablet computer, a laptop computer, a personal digital assistant, a desktop computer, a kiosk, Wherein the at least one component is implemented as at least one component selected from the group consisting of a music player, a camera, a television, an electronic game unit, and a set-top box. 3. The system of claim 2, wherein the received context information further comprises an application context. 3. The system of claim 2, wherein the received context information further comprises personal data associated with the user. 3. The system of claim 2, wherein the received context information further comprises data from a database associated with the user. 3. The system of claim 2, wherein the received context information further comprises data obtained from the conversation history. 3. The system of claim 2, wherein the received context information further comprises data received from at least one sensor. 3. The system of claim 2, wherein the received context information further comprises an application preference. 3. The system of claim 2, wherein the received context information further comprises an application usage history. 3. The system of claim 2, wherein the received context information further comprises data describing an event. 3. The system of claim 2, wherein the received context information further comprises a current conversation state. 3. The system of claim 2, wherein the received context information further comprises an input previously provided by the user. 3. The system of claim 2, wherein the received context information further comprises a location. 3. The system of claim 2, wherein the received context information further comprises a local time. 3. The system of claim 2, wherein the received context information further comprises an environmental condition.

Images