US10216832B2 - Underspecification of intents in a natural language processing system - Google Patents

Underspecification of intents in a natural language processing system Download PDF

Info

Publication number
US10216832B2
US10216832B2 US15/384,275 US201615384275A US10216832B2 US 10216832 B2 US10216832 B2 US 10216832B2 US 201615384275 A US201615384275 A US 201615384275A US 10216832 B2 US10216832 B2 US 10216832B2
Authority
US
United States
Prior art keywords
intent
intents
node
natural language
ancestor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US15/384,275
Other versions
US20180174578A1 (en
Inventor
Srinivas Bangalore
John Chen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Runway Growth Finance Corp
Original Assignee
Interactions LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Interactions LLC filed Critical Interactions LLC
Priority to US15/384,275 priority Critical patent/US10216832B2/en
Assigned to INTERACTIONS LLC reassignment INTERACTIONS LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BANGALORE, SRINIVAS, CHEN, JOHN
Priority to PCT/US2017/057867 priority patent/WO2018118202A1/en
Publication of US20180174578A1 publication Critical patent/US20180174578A1/en
Priority to US16/248,433 priority patent/US10796100B2/en
Application granted granted Critical
Publication of US10216832B2 publication Critical patent/US10216832B2/en
Assigned to SILICON VALLEY BANK reassignment SILICON VALLEY BANK INTELLECTUAL PROPERTY SECURITY AGREEMENT Assignors: INTERACTIONS LLC
Assigned to SILICON VALLEY BANK reassignment SILICON VALLEY BANK FIRST AMENDMENT TO AMENDED AND RESTATED INTELLECTUAL PROPERTY SECURITY AGREEMENT Assignors: INTERACTIONS LLC
Assigned to RUNWAY GROWTH FINANCE CORP. reassignment RUNWAY GROWTH FINANCE CORP. INTELLECTUAL PROPERTY SECURITY AGREEMENT Assignors: INTERACTIONS CORPORATION, INTERACTIONS LLC
Assigned to INTERACTIONS LLC reassignment INTERACTIONS LLC RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY RECORDED AT REEL/FRAME: 049388/0082 Assignors: SILICON VALLEY BANK
Assigned to INTERACTIONS LLC reassignment INTERACTIONS LLC RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY RECORDED AT REEL/FRAME: 049388/0152 Assignors: SILICON VALLEY BANK
Assigned to RUNWAY GROWTH FINANCE CORP. reassignment RUNWAY GROWTH FINANCE CORP. CORRECTIVE ASSIGNMENT TO CORRECT THE THE APPLICATION NUMBER PREVIOUSLY RECORDED AT REEL: 060445 FRAME: 0733. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: INTERACTIONS CORPORATION, INTERACTIONS LLC
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06F17/30654
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/10Text processing
    • G06F40/12Use of codes for handling textual entities
    • G06F40/137Hierarchical processing, e.g. outlines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • G06F16/332Query formulation
    • G06F16/3329Natural language query formulation or dialogue systems
    • G06F17/2241
    • G06F17/2785
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/30Semantic analysis
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/1815Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/183Speech classification or search using natural language modelling using context dependencies, e.g. language models

Definitions

  • the present invention generally relates to the field of natural language processing, and more specifically, to determining user intents from natural language utterances.
  • a natural language processing system interprets natural language inputs (hereinafter “utterances”) of users and, in cases where it cannot determine the users' most specific intents, leverages the information that it does understand to determine more general intents and to respond with intelligent prompts and questions related to the general intents to determine more specific intents.
  • domain hierarchy For each domain of interest that a particular company or other organization might have there is an intent ontology (hereinafter “domain hierarchy”).
  • domain hierarchy For each domain of interest that a particular company or other organization might have there is an intent ontology (hereinafter “domain hierarchy”).
  • domain hierarchies may either be separate, or they may be combined into a single hierarchy that encompasses them all.
  • the intents associated with the leaf nodes of a domain hierarchy represent specific user intents. For example, a specific intent of a user may be to “Book a flight.” Ideally, the system recognizes this user intent based on a given utterance.
  • the system determines confidence scores for non-leaf (also referred to as “ancestor”) nodes in the domain hierarchy, which represent more general intents.
  • the nodes that the system evaluates with respect to the utterance are the ancestor nodes of the leaf nodes that represent user intents that the system most confidently associates with the utterance.
  • An ancestor node is selected when the system is sufficiently confident that the ancestor node intent is the same as the intent specified by the utterance.
  • the system is still able to gain information about what the user wants. For example, the system may understand that a user has the general intent “Booking,” even if it cannot understand that the user specifically wants to book a flight. With the information it was able to gain from the general intent, the system can intelligently prompt the user for more specific information about the user intent, without needing to rely on a top-level menu of generic options or on a human intent analyst to determine the user's meaning.
  • FIG. 1 illustrates a high-level block diagram of a computing environment in which users interact with a computer system using natural language, according to one embodiment.
  • FIG. 2 is a high-level block diagram illustrating a detailed view of the natural language processing system from FIG. 1 , according to one embodiment.
  • FIG. 3A is a sample domain hierarchy of intents, according to one embodiment.
  • FIG. 3B is the sample domain hierarchy of intents of FIG. 3A with sample confidence scores, according to one embodiment.
  • FIG. 4 is a high-level block diagram illustrating physical components of a computer used as part or all of the computing environment from FIG. 1 , according to one embodiment.
  • FIG. 1 illustrates a high-level block diagram of a computing environment in which users interact with a computer system using natural language, according to one embodiment.
  • a client device 120 communicates user utterances over a network 140 .
  • a natural language processing system 100 being also connected to the network 140 , provides natural language interpretation services on behalf of a merchant system 110 . Based on its interpretation of user utterances, the natural language processing system 100 can determine an appropriate response to send to the user and can provide rational feedback to the user via the client device 120 . In one embodiment, the natural language processing system 100 provides interpretations of user utterances to the merchant system 110 , which can then determine a response to deliver to the client device 120 .
  • the natural language processing system 100 Upon receipt of an utterance, the natural language processing system 100 determines an intent represented by that utterance. The natural language processing system 100 will be described in more detail with respect to FIG. 2 .
  • the client device 120 is a computing device such as a smart phone, laptop computer, desktop computer, or any other device that can receive user utterances in a natural language form, such as text or voice input.
  • the client device 120 communicates the utterance over the network 140 to the natural language processing system 100 , where the utterance is interpreted.
  • the network 140 may be any suitable communications network for data transmission.
  • the network 140 uses standard communications technologies and/or protocols and can include the Internet.
  • the entities use custom and/or dedicated data communications technologies.
  • FIG. 2 is a high-level block diagram illustrating a detailed view of the natural language processing system 100 from FIG. 1 , according to one embodiment.
  • the natural language processing system 100 comprises a repository of domain hierarchies 205 , a repository of intent understanding models 210 , a repository of hierarchy understanding models 240 , a repository of dialogue scripts 230 , an utterance response module 235 , and an intent determination module 220 .
  • the intent determination module 220 in turn comprises a hierarchy determination module 221 , and a node scoring module 222 .
  • the natural language processing system 100 further comprises a repository of task models and a repository of user models (not illustrated in FIG. 2 ).
  • the dialogue scripts 230 comprise words, phrases, or other prompts for responding to the user utterances (e.g., to the intents determined for the user utterances).
  • a dialogue script can also reference certain task models and user models.
  • a dialogue script can contain a variety of words, phrases and prompts that the utterance response module 235 uses to respond to a user intent.
  • the particular response that the utterance response module 235 uses to respond to a user utterance may depend on the task stored in the repository of task models that is related to the intent and a user model from the repository of user models that may determine which response will be best suited to the particular user.
  • the dialogue scripts 230 might contain the response “OK, it sounds like you'd like to book a flight. Please say the city from which you'd like to depart.”
  • the specific content of the response is determined by the task models.
  • the task models comprise information about what processes must take place to complete certain tasks.
  • the task models may further comprise information about what data must be obtained from the user, or from other sources, to accomplish the tasks.
  • a task model for booking a flight could comprise asking for the destination city and traveling days, determining whether any flights are available on the traveling days, and obtaining the passenger's name, seat preference, and credit card number. Based on a task model, the system determines the best way to prompt a user for the information that is relevant to the specific task.
  • the user models comprise information about the kind of users that are providing user utterances to the client device 120 .
  • the system can recognize a user based on an identifier such as the telephone number of the phone that the user calls from to access the system.
  • the system may have voice recognition models trained to understand the user's voice. This information is stored as part of a user model that the intent determination module 220 can use to better understand the user's meaning.
  • the user models may also contain user-specific preferences, such as prompts that are chosen for that specific user to enhance relevance or understandability for that user.
  • Domain hierarchies stored within the repository of domain hierarchies 205 , are hierarchical graphs having leaf nodes and non-leaf nodes (hereinafter referred to as “ancestor nodes” of the leaf nodes, in that they may be parents of leaf nodes, parents of parents, etc.). Each node represents a possible user intent. Each leaf node represents a different specific user intent (the most specific that the domain hierarchy in question recognizes), and the ancestor nodes represent related but more general user intents.
  • FIG. 3A is a sample domain hierarchy of intents, according to one embodiment.
  • the example of FIG. 3A is a domain hierarchy constructed to represent airline services. It comprises ancestor nodes 360 that represent general user intents. According to this example domain hierarchy, one ancestor node represents a general user intent of “Booking”.
  • the example domain hierarchy of FIG. 3A is a sample domain hierarchy of intents, according to one embodiment.
  • the example of FIG. 3A is a domain hierarchy constructed to represent airline services. It comprises ancestor nodes 360 that represent general user intents. According to this example domain hierarchy, one ancestor node represents a general user intent of “Booking”.
  • 3A further comprises leaf nodes 370 that represent specific user intents such as “Check in”, “Book a flight”, “Book a rental car”, “Book a hotel room”, and “Check flight status.” Some of the leaf nodes 370 are descendants of the ancestor node 360 that represents the general user intent of “Booking” because the specific intents to “Book a flight”, “Book a rental car”, and “Book a hotel room” are all more specific versions of that general intent.
  • the domain hierarchy also has an ancestor node that is the root node of the domain hierarchy. That is, it is the ancestor node 360 of the domain hierarchy from which all other nodes in the hierarchy are descended.
  • the root node represents the most general, i.e. least specific, user intent.
  • the hierarchy understanding models 240 include a model for each domain hierarchy 205 .
  • Hierarchy understanding models 240 map basic information about user utterances to one or more domain hierarchies 205 that are likely to contain information applicable to the user's specific intent.
  • the hierarchy determination module 221 uses the models 240 when determining which of the domain hierarchies 205 is most applicable for a given utterance and hence should be analyzed with respect to the utterance; in some embodiments, each of the hierarchy understanding models 240 is applied to the given utterance, and a most applicable domain hierarchy 205 is selected based on the results (e.g., the domain hierarchy 205 whose corresponding hierarchy understanding model 240 produces the highest score for the utterance).
  • a hierarchy understanding model 240 is trained using supervised machine learning, in which a training set upon which the hierarchy understanding model is trained contains utterances that are labeled with the general user intents they are known to represent.
  • the hierarchy understanding model 240 is trained using multinomial logistic regression.
  • Other training techniques may be employed in other embodiments, such as support vector machines, deep neural networks, na ⁇ ve Bayes, and decision trees.
  • the intent understanding models 210 comprise an understanding model for each intent node in a domain hierarchy 205 .
  • An understanding model for a given intent node takes an utterance as input and produces as output a confidence score indicating whether the utterance represents the intent corresponding to the intent node, e.g., as a scalar value representing a probability value, or the like.
  • the intent understanding model for each intent node is trained using supervised machine learning, in which a training set upon which the intent understanding model is trained contains utterances that are labeled with the user intents the utterances are known to represent.
  • the intent understanding models 210 are trained using multinomial logistic regression. Other training techniques such as support vector machines, deep neural networks, na ⁇ ve Bayes, and decision trees may be employed in other embodiments.
  • the intent determination module 220 associates a user utterance with an output comprising an intent and a confidence score associated with the intent. For example, if the user utterance were “I would like to book a flight”, the output of the intent determination module 220 might be ⁇ BookFlight, 0.85>, indicating the “Book Flight” intent with a confidence score of 0.85.
  • the intent determination module 220 comprises the hierarchy determination module 221 , and the node scoring module 222 .
  • the hierarchy determination module 221 determines which domain hierarchies 205 should be used to determine a user's specific intent.
  • the hierarchy determination module 221 applies hierarchy understanding models 240 to the user utterance.
  • each hierarchy understanding model outputs a score (e.g., a real number value) indicating a likelihood that the domain hierarchy to which the model corresponds contains information that is applicable to a general intent of the user (and should therefore be used to determine the more specific intent of the user).
  • the hierarchy determination module 221 selects a subset of one or more of the applicable domain hierarchies for the intent determination module 220 to use to determine a specific user intent.
  • the hierarchy determination model might select a domain hierarchy of actions related to purchasing products (e.g., buy, return, refund) and a domain hierarchy of products (e.g., ticket, car, gift card).
  • the node scoring module 222 determines a confidence score for some or all of the intents in the one or more domain hierarchies 205 that were selected by the hierarchy determination module 221 .
  • the node scoring module determines confidence scores for leaf nodes by applying the intent understanding models 210 corresponding to nodes in the one or more domain hierarchies.
  • the intent determination module 220 outputs an identification of the intent associated with a chosen node and (optionally) the confidence score associated with the chosen node.
  • the intent determination module 220 outputs information about nodes that are chosen from each of the domain hierarchies (e.g., in the form of a list of the portions of the overall intent) if each node exceeds the minimum confidence threshold.
  • the natural language processing system 100 further comprises an utterance response module 235 that uses the intent determined by the intent determination module 220 to respond to the user.
  • the utterance response module 235 accesses a dialogue script from a dialogue script repository 230 which contains information about how to respond to the intent, such as prompts, phrases, or other responses to make to the user.
  • the utterance response module 235 also accesses task models and user models based on the user intent.
  • the dialogue scripts 230 can include prompts for clarification about the specific intent of the user.
  • a dialogue script might contain instructions for giving the user the prompt “I see that you are interested in booking. Would you like to book a flight, book a hotel room, or book a rental car?” In this way, the system can provide useful prompts that ask for specific clarifications (e.g., what specific type of thing to book), instead of frustrating a user by asking a the most general question about user intent (e.g., “I didn't get that; please say it again”).
  • the utterance response module 235 and the dialogue script repository 230 are hosted by the merchant system 110 instead of by the language processing system 100 .
  • the scores produced by the node scoring module 222 are used by the intent determination module 220 to determine a most likely specific intent that corresponds to a user utterance.
  • the scores produced by the node scoring module 222 are used by the intent determination module 220 to determine a most likely specific intent that corresponds to a user utterance.
  • the node scoring module 222 scores every node in each of the domain hierarchies selected by the hierarchy determination module 221 . If the confidence score associated with at least one of these intents exceeds a minimum confidence threshold, then an output is produced by the natural language processing system 100 comprising the intent with the highest confidence score and (in some embodiments) the confidence score associated with the intent.
  • the intent determination module 220 determines that none of the determined confidence scores exceeds the minimum confidence threshold, the intent determination module 220 tries to identify an ancestor node(s) that represents an intent that most closely corresponds to the given user utterance.
  • the intent determination module 220 examines the relationship between the two nodes corresponding to the two intents with the two highest confidence scores. If the two nodes have a parent child relationship as expressed in their corresponding intent hierarchy, then the intent determination module outputs the parent node, i.e. the more general node. In a situation in which the nodes of multiple domain hierarchies are scored, this procedure is applied to each hierarchy, and the final output is the concatenation of the results from each hierarchy.
  • the output of the intent determination module 220 might be ⁇ book, flight> with a confidence score of 0.25. If this score is less than the minimum confidence threshold, then the intent determination module 220 may ascertain from the two highest scoring verb intents that ⁇ book> should still be output, and conversely determine from the two highest scoring noun intents that ⁇ Root> should be output. Consequently, the intent determination module 220 would output ⁇ book, Root> as the intent (i.e., the user wants to complete a booking of some kind).
  • the confidence scores of only the leaf nodes in a particular hierarchy are computed first. If none of the leaf nodes receive scores of sufficient confidence, then the trained intent understanding models 210 of all or some of the ancestor nodes immediately connected to the leaf nodes are evaluated with respect to the given utterance. This process is repeated, evaluating the next set of nodes up in the hierarchy until either a node is determined to have a score of sufficient confidence, or a root node of the hierarchy is reached, indicating that the utterance does not match any of the possible user intents, even the general intents.
  • the intent understanding models 210 of all ancestor nodes between a leaf node and the root node are used to evaluate the utterance before the ancestor nodes of other leaf nodes are evaluated and assigned scores and typically without reevaluating an ancestor node that was scored previously.
  • the confidence score of an ancestor node is determined based on the scores of other nodes, such as the leaf nodes or other ancestor nodes.
  • Examples of algorithms that can be used to score ancestor nodes include: using the geometric mean of the scores of descendant nodes to compute the score for their ancestor node, computing the maximum score of the descendant nodes, or directly identifying the score of the ancestor node without aggregating from the descendant nodes.
  • the particular way in which the scores are determined differs in different embodiments, some of which are subsequently detailed.
  • the scores of ancestor nodes are determined using the arithmetic mean of the corresponding leaf nodes. That is, for some or all ancestor nodes that are immediately adjacent to at least one leaf node, the average value of the confidence scores of all leaf nodes that are descendants of the ancestor node in question is determined and assigned as the confidence score of that ancestor node.
  • the confidence scores of ancestor nodes are determined using the geometric mean, rather than the arithmetic mean, in which case the descendant leaf node scores are multiplied together then the nth root of the result is taken, where n is the number of descendant leaf nodes.
  • the confidence score of the ancestor node is then compared to a new threshold value, lower than the initial threshold value used for the leaf node confidence scores, and tested against that new threshold value to determine whether the confidence score for the ancestor node is high enough for the ancestor node to be chosen as representing the user intent.
  • the confidence scores of the ancestor nodes may be increased, e.g., by a multiplier, and compared to the initial threshold value.
  • FIG. 3B shows one possible embodiment of determining a user intent for a domain hierarchy for an airline, where the confidence scores of ancestor nodes are computed using an arithmetic mean.
  • the leaf nodes 370 which represent specific user intents, have accompanying confidence scores assigned by the node scoring module 222 in response to an utterance.
  • confidence scores are values between 0 and 100, where a score of 100 for an intent indicates the highest possible confidence that the utterance corresponds to that intent, and a score of 0 indicates the lowest possible confidence that the utterance corresponds to that intent. It is appreciated that confidence scores may be presented in different ways, or with different ranges, in other embodiments.
  • a minimum sufficient confidence score 380 has a value of 80. Since none of the leaf nodes shown in FIG. 3B received confidence scores of at least 80, a confidence score is determined for those ancestor nodes that are immediately adjacent to a leaf node.
  • the “Booking” intent has three descendants that are leaf nodes: “Book a flight” with a score of 77, “Book a rental car” with a score of 67, and “Book a hotel room” with a score of 48. Accordingly, the ancestor node representing the “Booking” intent receives a score of 64, which is the arithmetic mean of the scores of those three leaf node descendants. If, in the example of FIG.
  • the threshold confidence score value becomes 56.
  • the newly scored nodes are examined, and it is found that “Booking” with a score of 64 exceeds this new confidence score threshold.
  • “Booking” is chosen as a general intent that corresponds to the utterance.
  • no leaf nodes receive confidence scores higher than the confidence threshold, a collection of all leaf nodes with scores higher than some predetermined value is found.
  • the nearest common ancestor node to all the nodes in the group of chosen nodes is identified as a general intent of the user. It is appreciated that this technique may also be used by other confidence score determination methods in other embodiments as one possible way to break a tie in the case that two nodes receive identical confidence scores.
  • An example minimum confidence score 380 is established with a value of 80.
  • a predetermined value of 60 is established as a cutoff value for determining which leaf node confidence scores 390 will be considered when finding a nearest common ancestor. Since none of the example confidence scores 390 that are assigned to the leaf nodes is greater than or equal to 80, the example minimum sufficient confidence 380 , the leaf nodes with confidence scores greater than or equal to 60 are found.
  • these leaf nodes represent the intent “Book a flight” with a score of 77, and the intent “Book a rental car” with a score of 67.
  • the nearest common ancestor node of both the intent to “Book a flight” and the intent to “Book a rental car” is the node that represents the intent “Booking”.
  • “Booking” is chosen as a general intent that corresponds to the utterance.
  • some or all ancestor nodes that are adjacent to at least one leaf node receive scores equal to the maximum value of the scores received by some or all leaf nodes that are descendants of the ancestor node in question.
  • the minimum value of scores received by some or all leaf nodes that are descendants of the ancestor node may be used to determine the new scores, rather than the maximum value.
  • the new scores are incremented by some predetermined amount; in other embodiments, the minimum confidence threshold is decreased by some predetermined amount.
  • the newly scored nodes are then compared against the minimum confidence threshold. The process is repeated, propagating up the hierarchy, until a node receives a score that is at least as high as the value of the threshold confidence value, or until the node at the root of the domain hierarchy is reached.
  • the technique of using maximum values of descendant leaf nodes to determine scores for ancestor nodes is shown with respect to the sample domain hierarchy in FIG. 3B . Since all of the leaf nodes in FIG. 3B received confidence scores 390 with values less than 80, the minimum sufficient confidence score 380 , those ancestor nodes adjacent to leaf nodes are scored.
  • the ancestor node that represents the intent “Booking” has three descendant leaf nodes: “Book a flight” with a score of 77, “Book a rental car” with a score of 67, and “Book a hotel room” with a score of 48.
  • the “Booking” ancestor node is assigned a score of 77, the maximum value of the three descendant leaf node scores.
  • the root node is another ancestor node which is adjacent to at least one leaf node.
  • the root node has five descendant leaf nodes. They are “Check in” with a score of 11, “Book a flight” with a score of 77, “Book a rental car” with a score of 67, “Book a hotel room” with a score of 48, and “Check flight status” with a score of 52.
  • the root node receives a confidence score of 77, the maximum of the five descendant leaf node scores.
  • the example minimum sufficient confidence value 380 is decreased by a predetermined amount of 20. Thus, the newly scored nodes are compared to a new minimum sufficient confidence score value of 60.
  • Both newly scored nodes have scores of 77, so both have scores that exceed the cutoff value of 60.
  • the intent “Booking” or the root node are chosen to represent the utterance, according to some predetermined tie-breaking procedure such as determining the nearest common ancestor node of both nodes.
  • two domain hierarchies from the repository of domain hierarchies 205 are used to classify user intents.
  • the user intent is divided into a ⁇ verb, noun> pair, with one intent understanding model 210 corresponding to the “verb” part and another intent understanding model 210 corresponding to the “noun” part.
  • the verb refers to a node in a domain hierarchy that contains user intents related to actions.
  • the noun refers to a node in another domain hierarchy that contains user intents related to objects that are to be acted on. For example, in such an embodiment, if the user wants to ask an airline for information about a flight status, the verb could be “check” and the noun could be “flight status”.
  • the node scoring module 222 determines a confidence score for a ⁇ verb, noun> pair by combining the intent understanding models 210 associated with the verb and the noun, respectively.
  • the node scoring module 222 obtains a confidence score for the ⁇ verb, noun> pair by defining a function that takes as input the verb confidence score and the noun confidence score and outputs a new confidence score for the ⁇ verb, noun> pair.
  • An example of a function for combining a verb confidence score and a noun confidence score is a minimum function.
  • FIG. 4 is a high-level block diagram illustrating physical components of a computer 400 used as part or all of the computing environment from FIG. 1 , according to one embodiment. Illustrated are at least one processor 402 coupled to a chipset 404 . Also coupled to the chipset 404 are a memory 406 , a storage device 408 , a graphics adapter 412 , and a network adapter 416 . A display 418 is coupled to the graphics adapter 412 . In one embodiment, the functionality of the chipset 404 is provided by a memory controller hub 420 and an I/O controller hub 422 . In another embodiment, the memory 406 is coupled directly to the processor 402 instead of the chipset 404 .
  • the storage device 408 is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device.
  • the memory 406 holds instructions and data used by the processor 402 .
  • the graphics adapter 412 displays images and other information on the display 418 .
  • the network adapter 416 couples the computer 400 to a local or wide area network.
  • a computer 400 can have different and/or other components than those shown in FIG. 4 .
  • the computer 400 can lack certain illustrated components.
  • a computer 400 acting as a server may lack a graphics adapter 412 , and/or display 418 , as well as a keyboard or pointing device.
  • the storage device 408 can be local and/or remote from the computer 400 (such as embodied within a storage area network (SAN)).
  • SAN storage area network
  • the computer 400 is adapted to execute computer program modules for providing functionality described herein.
  • module refers to computer program logic utilized to provide the specified functionality.
  • a module can be implemented in hardware, firmware, and/or software.
  • program modules are stored on the storage device 408 , loaded into the memory 406 , and executed by the processor 402 .
  • Embodiments of the entities described herein can include other and/or different modules than the ones described here.
  • the functionality attributed to the modules can be performed by other or different modules in other embodiments.
  • this description occasionally omits the term “module” for purposes of clarity and convenience.
  • Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
  • the present invention also relates to an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer.
  • a computer program may be stored in a non-transitory computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of computer-readable storage medium suitable for storing electronic instructions, and each coupled to a computer system bus.
  • the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
  • the present invention is well suited to a wide variety of computer network systems over numerous topologies.
  • the configuration and management of large networks comprise storage devices and computers that are communicatively coupled to dissimilar computers and storage devices over a network, such as the Internet.

Abstract

A natural language processing system has a hierarchy of user intents related to a domain of interest, the hierarchy having specific intents corresponding to leaf nodes of the hierarchy, and more general intents corresponding to ancestor nodes of the leaf nodes. The system also has a trained understanding model that can classify natural language utterances according to user intent. When the understanding model cannot determine with sufficient confidence that a natural language utterance corresponds to one of the specific intents, the natural language processing system traverses the hierarchy of intents to find a more general user intent that is related to the most applicable specific intent of the utterance and for which there is sufficient confidence. The general intent can then be used to prompt the user with questions applicable to the general intent to obtain the missing information needed for a specific intent.

Description

FIELD OF ART
The present invention generally relates to the field of natural language processing, and more specifically, to determining user intents from natural language utterances.
BACKGROUND
There are many ways to provide natural language as input to a computer system. However, such systems are not necessarily prepared to respond with ease to utterances that are input in a natural language. If a system were not sufficiently confident about the intent of a user, the system could either query a human intent analyst for clarification or present the user with a long list of possible intents. Human analysts are skilled at quickly determining user intents, but it is costly to employ teams of analysts to provide clarification to computer systems in real time. An alternative—responding to the user with a long list of possible options—can frustrate the user by becoming repetitive and by making it seem as though the system completely failed to understand the user input.
SUMMARY
A natural language processing system interprets natural language inputs (hereinafter “utterances”) of users and, in cases where it cannot determine the users' most specific intents, leverages the information that it does understand to determine more general intents and to respond with intelligent prompts and questions related to the general intents to determine more specific intents.
For each domain of interest that a particular company or other organization might have there is an intent ontology (hereinafter “domain hierarchy”). The various domain hierarchies may either be separate, or they may be combined into a single hierarchy that encompasses them all. The intents associated with the leaf nodes of a domain hierarchy represent specific user intents. For example, a specific intent of a user may be to “Book a flight.” Ideally, the system recognizes this user intent based on a given utterance. However, for a given utterance, if the system cannot classify the utterance as corresponding to a specific intent with a confidence that exceeds some minimum threshold of confidence, then the system determines confidence scores for non-leaf (also referred to as “ancestor”) nodes in the domain hierarchy, which represent more general intents. The nodes that the system evaluates with respect to the utterance are the ancestor nodes of the leaf nodes that represent user intents that the system most confidently associates with the utterance. An ancestor node is selected when the system is sufficiently confident that the ancestor node intent is the same as the intent specified by the utterance.
Although the ancestor node represents a general intent that is less specific than that of the leaf node, the system is still able to gain information about what the user wants. For example, the system may understand that a user has the general intent “Booking,” even if it cannot understand that the user specifically wants to book a flight. With the information it was able to gain from the general intent, the system can intelligently prompt the user for more specific information about the user intent, without needing to rely on a top-level menu of generic options or on a human intent analyst to determine the user's meaning.
The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 illustrates a high-level block diagram of a computing environment in which users interact with a computer system using natural language, according to one embodiment.
FIG. 2 is a high-level block diagram illustrating a detailed view of the natural language processing system from FIG. 1, according to one embodiment.
FIG. 3A is a sample domain hierarchy of intents, according to one embodiment.
FIG. 3B is the sample domain hierarchy of intents of FIG. 3A with sample confidence scores, according to one embodiment.
FIG. 4 is a high-level block diagram illustrating physical components of a computer used as part or all of the computing environment from FIG. 1, according to one embodiment.
The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
DETAILED DESCRIPTION
System Environment
FIG. 1 illustrates a high-level block diagram of a computing environment in which users interact with a computer system using natural language, according to one embodiment. A client device 120 communicates user utterances over a network 140. A natural language processing system 100, being also connected to the network 140, provides natural language interpretation services on behalf of a merchant system 110. Based on its interpretation of user utterances, the natural language processing system 100 can determine an appropriate response to send to the user and can provide rational feedback to the user via the client device 120. In one embodiment, the natural language processing system 100 provides interpretations of user utterances to the merchant system 110, which can then determine a response to deliver to the client device 120. These various components are now described in additional detail.
Upon receipt of an utterance, the natural language processing system 100 determines an intent represented by that utterance. The natural language processing system 100 will be described in more detail with respect to FIG. 2.
The client device 120 is a computing device such as a smart phone, laptop computer, desktop computer, or any other device that can receive user utterances in a natural language form, such as text or voice input. The client device 120 communicates the utterance over the network 140 to the natural language processing system 100, where the utterance is interpreted.
The network 140 may be any suitable communications network for data transmission. In an embodiment such as that illustrated in FIG. 1, the network 140 uses standard communications technologies and/or protocols and can include the Internet. In another embodiment, the entities use custom and/or dedicated data communications technologies.
System Architecture
FIG. 2 is a high-level block diagram illustrating a detailed view of the natural language processing system 100 from FIG. 1, according to one embodiment. The natural language processing system 100 comprises a repository of domain hierarchies 205, a repository of intent understanding models 210, a repository of hierarchy understanding models 240, a repository of dialogue scripts 230, an utterance response module 235, and an intent determination module 220. In one embodiment, the intent determination module 220 in turn comprises a hierarchy determination module 221, and a node scoring module 222. In one embodiment, the natural language processing system 100 further comprises a repository of task models and a repository of user models (not illustrated in FIG. 2).
The dialogue scripts 230 comprise words, phrases, or other prompts for responding to the user utterances (e.g., to the intents determined for the user utterances). According to one embodiment, a dialogue script can also reference certain task models and user models. For example, a dialogue script can contain a variety of words, phrases and prompts that the utterance response module 235 uses to respond to a user intent. The particular response that the utterance response module 235 uses to respond to a user utterance may depend on the task stored in the repository of task models that is related to the intent and a user model from the repository of user models that may determine which response will be best suited to the particular user. As one example, if the determined user intent for an utterance were to book a flight, the dialogue scripts 230 might contain the response “OK, it sounds like you'd like to book a flight. Please say the city from which you'd like to depart.” In some embodiments, the specific content of the response (corresponding to the data to be obtained) is determined by the task models.
The task models comprise information about what processes must take place to complete certain tasks. The task models may further comprise information about what data must be obtained from the user, or from other sources, to accomplish the tasks. For example, a task model for booking a flight could comprise asking for the destination city and traveling days, determining whether any flights are available on the traveling days, and obtaining the passenger's name, seat preference, and credit card number. Based on a task model, the system determines the best way to prompt a user for the information that is relevant to the specific task.
The user models comprise information about the kind of users that are providing user utterances to the client device 120. For example, in some embodiments, the system can recognize a user based on an identifier such as the telephone number of the phone that the user calls from to access the system. In some cases, such as for a returning user, the system may have voice recognition models trained to understand the user's voice. This information is stored as part of a user model that the intent determination module 220 can use to better understand the user's meaning. The user models may also contain user-specific preferences, such as prompts that are chosen for that specific user to enhance relevance or understandability for that user.
Domain hierarchies, stored within the repository of domain hierarchies 205, are hierarchical graphs having leaf nodes and non-leaf nodes (hereinafter referred to as “ancestor nodes” of the leaf nodes, in that they may be parents of leaf nodes, parents of parents, etc.). Each node represents a possible user intent. Each leaf node represents a different specific user intent (the most specific that the domain hierarchy in question recognizes), and the ancestor nodes represent related but more general user intents.
For example, FIG. 3A is a sample domain hierarchy of intents, according to one embodiment. The example of FIG. 3A is a domain hierarchy constructed to represent airline services. It comprises ancestor nodes 360 that represent general user intents. According to this example domain hierarchy, one ancestor node represents a general user intent of “Booking”. The example domain hierarchy of FIG. 3A further comprises leaf nodes 370 that represent specific user intents such as “Check in”, “Book a flight”, “Book a rental car”, “Book a hotel room”, and “Check flight status.” Some of the leaf nodes 370 are descendants of the ancestor node 360 that represents the general user intent of “Booking” because the specific intents to “Book a flight”, “Book a rental car”, and “Book a hotel room” are all more specific versions of that general intent. According to the embodiment shown in FIG. 3A, the domain hierarchy also has an ancestor node that is the root node of the domain hierarchy. That is, it is the ancestor node 360 of the domain hierarchy from which all other nodes in the hierarchy are descended. According to one embodiment, the root node represents the most general, i.e. least specific, user intent.
Returning again to FIG. 2, the hierarchy understanding models 240 include a model for each domain hierarchy 205. Hierarchy understanding models 240 map basic information about user utterances to one or more domain hierarchies 205 that are likely to contain information applicable to the user's specific intent. The hierarchy determination module 221 uses the models 240 when determining which of the domain hierarchies 205 is most applicable for a given utterance and hence should be analyzed with respect to the utterance; in some embodiments, each of the hierarchy understanding models 240 is applied to the given utterance, and a most applicable domain hierarchy 205 is selected based on the results (e.g., the domain hierarchy 205 whose corresponding hierarchy understanding model 240 produces the highest score for the utterance). In one embodiment, a hierarchy understanding model 240 is trained using supervised machine learning, in which a training set upon which the hierarchy understanding model is trained contains utterances that are labeled with the general user intents they are known to represent. In one embodiment, the hierarchy understanding model 240 is trained using multinomial logistic regression. Other training techniques may be employed in other embodiments, such as support vector machines, deep neural networks, naïve Bayes, and decision trees.
The intent understanding models 210 comprise an understanding model for each intent node in a domain hierarchy 205. (As a specific example purely for the purposes of illustration, one sample 4-node domain hierarchy 205A is illustrated in FIG. 2, each node depicted with a circle and its corresponding intent understanding model depicted with a square.) An understanding model for a given intent node takes an utterance as input and produces as output a confidence score indicating whether the utterance represents the intent corresponding to the intent node, e.g., as a scalar value representing a probability value, or the like.
In one embodiment, the intent understanding model for each intent node is trained using supervised machine learning, in which a training set upon which the intent understanding model is trained contains utterances that are labeled with the user intents the utterances are known to represent. In one embodiment, the intent understanding models 210 are trained using multinomial logistic regression. Other training techniques such as support vector machines, deep neural networks, naïve Bayes, and decision trees may be employed in other embodiments.
The intent determination module 220 associates a user utterance with an output comprising an intent and a confidence score associated with the intent. For example, if the user utterance were “I would like to book a flight”, the output of the intent determination module 220 might be <BookFlight, 0.85>, indicating the “Book Flight” intent with a confidence score of 0.85. In one embodiment, the intent determination module 220 comprises the hierarchy determination module 221, and the node scoring module 222.
For a given utterance, the hierarchy determination module 221 determines which domain hierarchies 205 should be used to determine a user's specific intent. The hierarchy determination module 221 applies hierarchy understanding models 240 to the user utterance. In one embodiment, each hierarchy understanding model outputs a score (e.g., a real number value) indicating a likelihood that the domain hierarchy to which the model corresponds contains information that is applicable to a general intent of the user (and should therefore be used to determine the more specific intent of the user). The hierarchy determination module 221 selects a subset of one or more of the applicable domain hierarchies for the intent determination module 220 to use to determine a specific user intent. For example, the hierarchy determination model might select a domain hierarchy of actions related to purchasing products (e.g., buy, return, refund) and a domain hierarchy of products (e.g., ticket, car, gift card).
For a given subset of domain hierarchies and a given utterance, the node scoring module 222 determines a confidence score for some or all of the intents in the one or more domain hierarchies 205 that were selected by the hierarchy determination module 221. The node scoring module determines confidence scores for leaf nodes by applying the intent understanding models 210 corresponding to nodes in the one or more domain hierarchies.
Once the intent determination module 220 has identified a node whose confidence score exceeds the minimum confidence threshold for a given utterance, the intent determination module 220 outputs an identification of the intent associated with a chosen node and (optionally) the confidence score associated with the chosen node. In embodiments in which multiple domain hierarchies are used to determine an intent, the intent determination module 220 outputs information about nodes that are chosen from each of the domain hierarchies (e.g., in the form of a list of the portions of the overall intent) if each node exceeds the minimum confidence threshold.
The natural language processing system 100 further comprises an utterance response module 235 that uses the intent determined by the intent determination module 220 to respond to the user. Depending on the user intent, the utterance response module 235 accesses a dialogue script from a dialogue script repository 230 which contains information about how to respond to the intent, such as prompts, phrases, or other responses to make to the user. In some embodiments, the utterance response module 235 also accesses task models and user models based on the user intent. In cases where a general, rather than specific, intent was chosen by the intent determination module 220 (i.e. an ancestor node was selected instead of a leaf node), the dialogue scripts 230 can include prompts for clarification about the specific intent of the user. For example, if the intent determination module 220 determines that a user has a general intent of “Booking,” then a dialogue script might contain instructions for giving the user the prompt “I see that you are interested in booking. Would you like to book a flight, book a hotel room, or book a rental car?” In this way, the system can provide useful prompts that ask for specific clarifications (e.g., what specific type of thing to book), instead of frustrating a user by asking a the most general question about user intent (e.g., “I didn't get that; please say it again”). In one embodiment, the utterance response module 235 and the dialogue script repository 230 (along with the task models and the user models) are hosted by the merchant system 110 instead of by the language processing system 100.
Under-Specification Algorithms
The scores produced by the node scoring module 222 are used by the intent determination module 220 to determine a most likely specific intent that corresponds to a user utterance. Various ways of determining user intents using confidence scores are now described.
In one embodiment, the node scoring module 222 scores every node in each of the domain hierarchies selected by the hierarchy determination module 221. If the confidence score associated with at least one of these intents exceeds a minimum confidence threshold, then an output is produced by the natural language processing system 100 comprising the intent with the highest confidence score and (in some embodiments) the confidence score associated with the intent.
If the intent determination module 220 determines that none of the determined confidence scores exceeds the minimum confidence threshold, the intent determination module 220 tries to identify an ancestor node(s) that represents an intent that most closely corresponds to the given user utterance. The intent determination module 220 examines the relationship between the two nodes corresponding to the two intents with the two highest confidence scores. If the two nodes have a parent child relationship as expressed in their corresponding intent hierarchy, then the intent determination module outputs the parent node, i.e. the more general node. In a situation in which the nodes of multiple domain hierarchies are scored, this procedure is applied to each hierarchy, and the final output is the concatenation of the results from each hierarchy. For example, there may be two domain hierarchies corresponding to verbs and nouns, respectively. In one particular example, the output of the intent determination module 220 might be <book, flight> with a confidence score of 0.25. If this score is less than the minimum confidence threshold, then the intent determination module 220 may ascertain from the two highest scoring verb intents that <book> should still be output, and conversely determine from the two highest scoring noun intents that <Root> should be output. Consequently, the intent determination module 220 would output <book, Root> as the intent (i.e., the user wants to complete a booking of some kind).
In other embodiments, the confidence scores of only the leaf nodes in a particular hierarchy are computed first. If none of the leaf nodes receive scores of sufficient confidence, then the trained intent understanding models 210 of all or some of the ancestor nodes immediately connected to the leaf nodes are evaluated with respect to the given utterance. This process is repeated, evaluating the next set of nodes up in the hierarchy until either a node is determined to have a score of sufficient confidence, or a root node of the hierarchy is reached, indicating that the utterance does not match any of the possible user intents, even the general intents. It is appreciated that in one embodiment, the intent understanding models 210 of all ancestor nodes between a leaf node and the root node are used to evaluate the utterance before the ancestor nodes of other leaf nodes are evaluated and assigned scores and typically without reevaluating an ancestor node that was scored previously.
According to some embodiments, the confidence score of an ancestor node is determined based on the scores of other nodes, such as the leaf nodes or other ancestor nodes. Examples of algorithms that can be used to score ancestor nodes include: using the geometric mean of the scores of descendant nodes to compute the score for their ancestor node, computing the maximum score of the descendant nodes, or directly identifying the score of the ancestor node without aggregating from the descendant nodes. The particular way in which the scores are determined differs in different embodiments, some of which are subsequently detailed.
In one embodiment, if no leaf nodes receive adequately high confidence scores, then the scores of ancestor nodes are determined using the arithmetic mean of the corresponding leaf nodes. That is, for some or all ancestor nodes that are immediately adjacent to at least one leaf node, the average value of the confidence scores of all leaf nodes that are descendants of the ancestor node in question is determined and assigned as the confidence score of that ancestor node. In some embodiments, the confidence scores of ancestor nodes are determined using the geometric mean, rather than the arithmetic mean, in which case the descendant leaf node scores are multiplied together then the nth root of the result is taken, where n is the number of descendant leaf nodes. The confidence score of the ancestor node is then compared to a new threshold value, lower than the initial threshold value used for the leaf node confidence scores, and tested against that new threshold value to determine whether the confidence score for the ancestor node is high enough for the ancestor node to be chosen as representing the user intent. (As an alternative to choosing a new, lower threshold value, the confidence scores of the ancestor nodes may be increased, e.g., by a multiplier, and compared to the initial threshold value.) If the confidence score of the ancestor node is not sufficiently great, the process is repeated (i.e. scoring the ancestor nodes adjacent to the previously scored nodes), until a node of significantly high confidence is chosen or the root of the hierarchical structure is reached.
For example, FIG. 3B shows one possible embodiment of determining a user intent for a domain hierarchy for an airline, where the confidence scores of ancestor nodes are computed using an arithmetic mean. The leaf nodes 370, which represent specific user intents, have accompanying confidence scores assigned by the node scoring module 222 in response to an utterance. In the example embodiment of FIG. 3B, confidence scores are values between 0 and 100, where a score of 100 for an intent indicates the highest possible confidence that the utterance corresponds to that intent, and a score of 0 indicates the lowest possible confidence that the utterance corresponds to that intent. It is appreciated that confidence scores may be presented in different ways, or with different ranges, in other embodiments. For the purposes of the example, a minimum sufficient confidence score 380 has a value of 80. Since none of the leaf nodes shown in FIG. 3B received confidence scores of at least 80, a confidence score is determined for those ancestor nodes that are immediately adjacent to a leaf node. The “Booking” intent has three descendants that are leaf nodes: “Book a flight” with a score of 77, “Book a rental car” with a score of 67, and “Book a hotel room” with a score of 48. Accordingly, the ancestor node representing the “Booking” intent receives a score of 64, which is the arithmetic mean of the scores of those three leaf node descendants. If, in the example of FIG. 3B, there were an additional ancestor node from which the “Booking”, “Check-in”, and “Check flight status” nodes descended, that node would receive a confidence score based on the arithmetic mean of all leaf nodes that are descended from it. Namely, its leaf node descendants would include “Check in” with a score of 11, “Book a flight” with a score of 77, “Book a rental car” with a score of 67, “Book a hotel room” with a score of 48, and “Check flight status” with a score of 52, and it would thus receive a score of 51, the average of the confidence scores of the leaf nodes. In this example, assume that the new threshold value were reduced to 80% of the prior threshold (of 70, in this example). Thus, the threshold confidence score value becomes 56. The newly scored nodes are examined, and it is found that “Booking” with a score of 64 exceeds this new confidence score threshold. As a result, “Booking” is chosen as a general intent that corresponds to the utterance.
According to one embodiment, if no leaf nodes receive confidence scores higher than the confidence threshold, a collection of all leaf nodes with scores higher than some predetermined value is found. The nearest common ancestor node to all the nodes in the group of chosen nodes is identified as a general intent of the user. It is appreciated that this technique may also be used by other confidence score determination methods in other embodiments as one possible way to break a tie in the case that two nodes receive identical confidence scores.
For example, the technique of using nearest common ancestor nodes to choose a general intent that corresponds to the utterance is described with respect to the sample domain hierarchy of FIG. 3B. An example minimum confidence score 380 is established with a value of 80. For the sake of the example, a predetermined value of 60 is established as a cutoff value for determining which leaf node confidence scores 390 will be considered when finding a nearest common ancestor. Since none of the example confidence scores 390 that are assigned to the leaf nodes is greater than or equal to 80, the example minimum sufficient confidence 380, the leaf nodes with confidence scores greater than or equal to 60 are found. In this example, these leaf nodes represent the intent “Book a flight” with a score of 77, and the intent “Book a rental car” with a score of 67. The nearest common ancestor node of both the intent to “Book a flight” and the intent to “Book a rental car” is the node that represents the intent “Booking”. As a result, “Booking” is chosen as a general intent that corresponds to the utterance.
In one embodiment, if none of the leaf nodes receive sufficiently high confidence scores, some or all ancestor nodes that are adjacent to at least one leaf node receive scores equal to the maximum value of the scores received by some or all leaf nodes that are descendants of the ancestor node in question. In some cases, the minimum value of scores received by some or all leaf nodes that are descendants of the ancestor node may be used to determine the new scores, rather than the maximum value. In some embodiments, the new scores are incremented by some predetermined amount; in other embodiments, the minimum confidence threshold is decreased by some predetermined amount. The newly scored nodes are then compared against the minimum confidence threshold. The process is repeated, propagating up the hierarchy, until a node receives a score that is at least as high as the value of the threshold confidence value, or until the node at the root of the domain hierarchy is reached.
For example, the technique of using maximum values of descendant leaf nodes to determine scores for ancestor nodes is shown with respect to the sample domain hierarchy in FIG. 3B. Since all of the leaf nodes in FIG. 3B received confidence scores 390 with values less than 80, the minimum sufficient confidence score 380, those ancestor nodes adjacent to leaf nodes are scored. The ancestor node that represents the intent “Booking” has three descendant leaf nodes: “Book a flight” with a score of 77, “Book a rental car” with a score of 67, and “Book a hotel room” with a score of 48. The “Booking” ancestor node is assigned a score of 77, the maximum value of the three descendant leaf node scores. The root node is another ancestor node which is adjacent to at least one leaf node. The root node has five descendant leaf nodes. They are “Check in” with a score of 11, “Book a flight” with a score of 77, “Book a rental car” with a score of 67, “Book a hotel room” with a score of 48, and “Check flight status” with a score of 52. The root node receives a confidence score of 77, the maximum of the five descendant leaf node scores. For the sake of this example, the example minimum sufficient confidence value 380 is decreased by a predetermined amount of 20. Thus, the newly scored nodes are compared to a new minimum sufficient confidence score value of 60. Both newly scored nodes have scores of 77, so both have scores that exceed the cutoff value of 60. Thus either the intent “Booking” or the root node are chosen to represent the utterance, according to some predetermined tie-breaking procedure such as determining the nearest common ancestor node of both nodes.
According to one embodiment, two domain hierarchies from the repository of domain hierarchies 205 are used to classify user intents. The user intent is divided into a <verb, noun> pair, with one intent understanding model 210 corresponding to the “verb” part and another intent understanding model 210 corresponding to the “noun” part. The verb refers to a node in a domain hierarchy that contains user intents related to actions. The noun refers to a node in another domain hierarchy that contains user intents related to objects that are to be acted on. For example, in such an embodiment, if the user wants to ask an airline for information about a flight status, the verb could be “check” and the noun could be “flight status”. The node scoring module 222 determines a confidence score for a <verb, noun> pair by combining the intent understanding models 210 associated with the verb and the noun, respectively. The node scoring module 222 obtains a confidence score for the <verb, noun> pair by defining a function that takes as input the verb confidence score and the noun confidence score and outputs a new confidence score for the <verb, noun> pair. An example of a function for combining a verb confidence score and a noun confidence score is a minimum function. For example, if a <verb, noun> pair is <book, flight>, the intent understanding model associated with the verb node might assign a confidence score of 0.95 to the verb “book” while the intent understanding model associated with the noun node might assign 0.85 to the noun “flight”. Consequently, the minimum function would assign the confidence score 0.85 to the pair <book, flight> because 0.85 is less than 0.95. Computer System Components
FIG. 4 is a high-level block diagram illustrating physical components of a computer 400 used as part or all of the computing environment from FIG. 1, according to one embodiment. Illustrated are at least one processor 402 coupled to a chipset 404. Also coupled to the chipset 404 are a memory 406, a storage device 408, a graphics adapter 412, and a network adapter 416. A display 418 is coupled to the graphics adapter 412. In one embodiment, the functionality of the chipset 404 is provided by a memory controller hub 420 and an I/O controller hub 422. In another embodiment, the memory 406 is coupled directly to the processor 402 instead of the chipset 404.
The storage device 408 is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. The memory 406 holds instructions and data used by the processor 402. The graphics adapter 412 displays images and other information on the display 418. The network adapter 416 couples the computer 400 to a local or wide area network.
As is known in the art, a computer 400 can have different and/or other components than those shown in FIG. 4. In addition, the computer 400 can lack certain illustrated components. In one embodiment, a computer 400 acting as a server may lack a graphics adapter 412, and/or display 418, as well as a keyboard or pointing device. Moreover, the storage device 408 can be local and/or remote from the computer 400 (such as embodied within a storage area network (SAN)).
As is known in the art, the computer 400 is adapted to execute computer program modules for providing functionality described herein. As used herein, the term “module” refers to computer program logic utilized to provide the specified functionality. Thus, a module can be implemented in hardware, firmware, and/or software. In one embodiment, program modules are stored on the storage device 408, loaded into the memory 406, and executed by the processor 402.
Embodiments of the entities described herein can include other and/or different modules than the ones described here. In addition, the functionality attributed to the modules can be performed by other or different modules in other embodiments. Moreover, this description occasionally omits the term “module” for purposes of clarity and convenience.
Other Considerations
The present invention has been described in particular detail with respect to one possible embodiment. Those of skill in the art will appreciate that the invention may be practiced in other embodiments. First, the particular naming of the components and variables, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Also, the particular division of functionality between the various system components described herein is merely for purposes of example, and is not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead performed by a single component.
Some portions of above description present the features of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or by functional names, without loss of generality.
Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a non-transitory computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of computer-readable storage medium suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present invention is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references to specific languages are provided for invention of enablement and best mode of the present invention.
The present invention is well suited to a wide variety of computer network systems over numerous topologies. Within this field, the configuration and management of large networks comprise storage devices and computers that are communicatively coupled to dissimilar computers and storage devices over a network, such as the Internet.
Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims (10)

What is claimed is:
1. A non-transitory computer-readable storage medium comprising instructions executable by a processor for identifying an intent associated with a natural language utterance, the instructions comprising:
instructions for receiving a natural language utterance specified by a user;
instructions for accessing a domain hierarchy of intents comprising leaf nodes and ancestor nodes, wherein each of a plurality of the leaf nodes has a corresponding intent understanding model trained through supervised machine learning based on respective training sets of utterances that are labeled with user intents that the sets of utterances are known to represent;
instructions for applying a plurality of intent understanding models of the leaf nodes to the natural language utterance, thereby producing a corresponding plurality of leaf node confidence scores;
instructions for determining that none of the leaf node confidence scores exceeds a given minimum confidence threshold;
instructions for, responsive to the determining, determining ancestor node confidence scores for the ancestor nodes wherein each of the ancestor node confidence scores is determined by:
identifying an intent understanding model corresponding to the ancestor node; and
applying the identified intent understanding model corresponding to the ancestor node to the natural language utterance, wherein the intent understanding model corresponding to the ancestor node produces a confidence score for the ancestor node;
instructions for identifying a most applicable ancestor node based on the ancestor node confidence scores; and
instructions for identifying, as a general intent of the natural language utterance, an intent corresponding to the identified most applicable ancestor node.
2. The non-transitory computer-readable storage medium of claim 1, the instructions further comprising:
instructions for identifying, based on the identified general intent, a prompt for additional information; and
instructions for providing the prompt to the user over the computer network.
3. The non-transitory computer-readable storage medium of claim 1, wherein any descendant node in the hierarchy of intents corresponds to a more specific user intent than the intents corresponding to any ancestor nodes from which the descendant node is descended.
4. The non-transitory computer-readable storage medium of claim 1, wherein applying the intent understanding models comprises associating the natural language utterance with a verb from a domain hierarchy of actions and with a noun from a domain hierarchy of intents related to items.
5. The non-transitory computer-readable storage medium of claim 1, wherein applying the intent understanding models comprises associating the natural language utterance with two or more intents, each intent selected from its own intent hierarchy.
6. A computer-implemented method for identifying an intent associated with a natural language utterance comprising:
receiving a natural language utterance specified by a user over a computer network;
accessing a domain hierarchy of intents comprising leaf nodes and ancestor nodes, wherein each of a plurality of the leaf nodes has a corresponding intent understanding model trained through supervised machine learning based on respective training sets of utterances that are labeled with user intents the sets of utterances are known to represent;
applying the intent understanding models corresponding to each of the plurality of leaf nodes to the natural language utterance, thereby producing a corresponding plurality of leaf node confidence scores;
determining that none of the leaf node confidence scores exceeds a given minimum confidence threshold;
responsive to the determining, determining ancestor node confidence scores for the ancestor nodes wherein each of the ancestor node confidence scores is determined by:
identifying an intent understanding model corresponding to the ancestor node; and
applying the identified intent understanding model corresponding to the ancestor node to the natural language utterance, wherein the intent understanding model corresponding to the ancestor node produces a confidence score for the ancestor node;
identifying a most applicable ancestor node based on the ancestor node confidence scores; and
identifying, as a general intent of the natural language utterance, an intent corresponding to the identified most applicable ancestor node.
7. The computer-implemented method of claim 6, further comprising:
identifying, based on the identified general intent, a prompt for additional information; and
providing the prompt to the user over the computer network.
8. The computer-implemented method of claim 6, wherein the leaf nodes in the hierarchy of intents correspond to more specific user intents than the intents corresponding to the ancestor nodes from which the leaf nodes are descended.
9. The computer-implemented method of claim 6, wherein applying the intent understanding models comprises associating the natural language utterance with a verb from a domain hierarchy of actions and with a noun from a domain hierarchy of intents related to items.
10. The computer-implemented method of claim 6, wherein applying the intent understanding models comprises associating the natural language utterance with two or more intents, each intent selected from its own intent hierarchy.
US15/384,275 2016-12-19 2016-12-19 Underspecification of intents in a natural language processing system Active US10216832B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US15/384,275 US10216832B2 (en) 2016-12-19 2016-12-19 Underspecification of intents in a natural language processing system
PCT/US2017/057867 WO2018118202A1 (en) 2016-12-19 2017-10-23 Underspecification of intents in a natural language processing system
US16/248,433 US10796100B2 (en) 2016-12-19 2019-01-15 Underspecification of intents in a natural language processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/384,275 US10216832B2 (en) 2016-12-19 2016-12-19 Underspecification of intents in a natural language processing system

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/248,433 Continuation US10796100B2 (en) 2016-12-19 2019-01-15 Underspecification of intents in a natural language processing system

Publications (2)

Publication Number Publication Date
US20180174578A1 US20180174578A1 (en) 2018-06-21
US10216832B2 true US10216832B2 (en) 2019-02-26

Family

ID=62562620

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/384,275 Active US10216832B2 (en) 2016-12-19 2016-12-19 Underspecification of intents in a natural language processing system
US16/248,433 Active US10796100B2 (en) 2016-12-19 2019-01-15 Underspecification of intents in a natural language processing system

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/248,433 Active US10796100B2 (en) 2016-12-19 2019-01-15 Underspecification of intents in a natural language processing system

Country Status (2)

Country Link
US (2) US10216832B2 (en)
WO (1) WO2018118202A1 (en)

Cited By (59)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190147044A1 (en) * 2016-12-19 2019-05-16 Interactions Llc Underspecification of intents in a natural language processing system
US10685645B2 (en) * 2018-08-09 2020-06-16 Bank Of America Corporation Identification of candidate training utterances from human conversations with an intelligent interactive assistant
US10733375B2 (en) * 2018-01-31 2020-08-04 Apple Inc. Knowledge-based framework for improving natural language understanding
US10978090B2 (en) 2013-02-07 2021-04-13 Apple Inc. Voice trigger for a digital assistant
US10984798B2 (en) 2018-06-01 2021-04-20 Apple Inc. Voice interaction at a primary device to access call functionality of a companion device
US11009970B2 (en) 2018-06-01 2021-05-18 Apple Inc. Attention aware virtual assistant dismissal
US11037565B2 (en) 2016-06-10 2021-06-15 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US11070949B2 (en) 2015-05-27 2021-07-20 Apple Inc. Systems and methods for proactively identifying and surfacing relevant content on an electronic device with a touch-sensitive display
US11087759B2 (en) 2015-03-08 2021-08-10 Apple Inc. Virtual assistant activation
US11120372B2 (en) 2011-06-03 2021-09-14 Apple Inc. Performing actions associated with task items that represent tasks to perform
US11126400B2 (en) 2015-09-08 2021-09-21 Apple Inc. Zero latency digital assistant
US11133008B2 (en) 2014-05-30 2021-09-28 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US11152002B2 (en) 2016-06-11 2021-10-19 Apple Inc. Application integration with a digital assistant
US11169616B2 (en) 2018-05-07 2021-11-09 Apple Inc. Raise to speak
US11237797B2 (en) 2019-05-31 2022-02-01 Apple Inc. User activity shortcut suggestions
US11257504B2 (en) 2014-05-30 2022-02-22 Apple Inc. Intelligent assistant for home automation
US11321116B2 (en) 2012-05-15 2022-05-03 Apple Inc. Systems and methods for integrating third party services with a digital assistant
US11348582B2 (en) 2008-10-02 2022-05-31 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
US11380310B2 (en) 2017-05-12 2022-07-05 Apple Inc. Low-latency intelligent automated assistant
US11388291B2 (en) 2013-03-14 2022-07-12 Apple Inc. System and method for processing voicemail
US11393475B1 (en) 2021-01-13 2022-07-19 Artificial Solutions Iberia S.L Conversational system for recognizing, understanding, and acting on multiple intents and hypotheses
US11405466B2 (en) 2017-05-12 2022-08-02 Apple Inc. Synchronization and task delegation of a digital assistant
US11423886B2 (en) 2010-01-18 2022-08-23 Apple Inc. Task flow identification based on user intent
US11431642B2 (en) 2018-06-01 2022-08-30 Apple Inc. Variable latency device coordination
US11467802B2 (en) 2017-05-11 2022-10-11 Apple Inc. Maintaining privacy of personal information
US11500672B2 (en) 2015-09-08 2022-11-15 Apple Inc. Distributed personal assistant
US11516537B2 (en) 2014-06-30 2022-11-29 Apple Inc. Intelligent automated assistant for TV user interactions
US11526368B2 (en) 2015-11-06 2022-12-13 Apple Inc. Intelligent automated assistant in a messaging environment
US11532306B2 (en) 2017-05-16 2022-12-20 Apple Inc. Detecting a trigger of a digital assistant
US11580990B2 (en) 2017-05-12 2023-02-14 Apple Inc. User-specific acoustic models
US11599331B2 (en) 2017-05-11 2023-03-07 Apple Inc. Maintaining privacy of personal information
US11606463B1 (en) 2020-03-31 2023-03-14 Interactions Llc Virtual assistant architecture for natural language understanding in a customer service system
US11657813B2 (en) 2019-05-31 2023-05-23 Apple Inc. Voice identification in digital assistant systems
US11671920B2 (en) 2007-04-03 2023-06-06 Apple Inc. Method and system for operating a multifunction portable electronic device using voice-activation
US11670289B2 (en) 2014-05-30 2023-06-06 Apple Inc. Multi-command single utterance input method
US11675491B2 (en) 2019-05-06 2023-06-13 Apple Inc. User configurable task triggers
US11675829B2 (en) 2017-05-16 2023-06-13 Apple Inc. Intelligent automated assistant for media exploration
US20230197068A1 (en) * 2021-03-05 2023-06-22 Capital One Services, Llc Systems and methods for dynamically updating machine learning models that provide conversational responses
US11696060B2 (en) 2020-07-21 2023-07-04 Apple Inc. User identification using headphones
US11705130B2 (en) 2019-05-06 2023-07-18 Apple Inc. Spoken notifications
US11710482B2 (en) 2018-03-26 2023-07-25 Apple Inc. Natural assistant interaction
US11727219B2 (en) 2013-06-09 2023-08-15 Apple Inc. System and method for inferring user intent from speech inputs
US11743378B1 (en) 2020-03-31 2023-08-29 Interactions Llc Intelligent agent assistant for natural language understanding in a customer service system
US11755276B2 (en) 2020-05-12 2023-09-12 Apple Inc. Reducing description length based on confidence
US11765209B2 (en) 2020-05-11 2023-09-19 Apple Inc. Digital assistant hardware abstraction
US11783815B2 (en) 2019-03-18 2023-10-10 Apple Inc. Multimodality in digital assistant systems
US11790914B2 (en) 2019-06-01 2023-10-17 Apple Inc. Methods and user interfaces for voice-based control of electronic devices
US11798547B2 (en) 2013-03-15 2023-10-24 Apple Inc. Voice activated device for use with a voice-based digital assistant
US11809483B2 (en) 2015-09-08 2023-11-07 Apple Inc. Intelligent automated assistant for media search and playback
US11809783B2 (en) 2016-06-11 2023-11-07 Apple Inc. Intelligent device arbitration and control
US11838734B2 (en) 2020-07-20 2023-12-05 Apple Inc. Multi-device audio adjustment coordination
US11853536B2 (en) 2015-09-08 2023-12-26 Apple Inc. Intelligent automated assistant in a media environment
US11853647B2 (en) 2015-12-23 2023-12-26 Apple Inc. Proactive assistance based on dialog communication between devices
US11854539B2 (en) 2018-05-07 2023-12-26 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US11888791B2 (en) 2019-05-21 2024-01-30 Apple Inc. Providing message response suggestions
US11886805B2 (en) 2015-11-09 2024-01-30 Apple Inc. Unconventional virtual assistant interactions
US11893992B2 (en) 2018-09-28 2024-02-06 Apple Inc. Multi-modal inputs for voice commands
US11914848B2 (en) 2020-05-11 2024-02-27 Apple Inc. Providing relevant data items based on context
US11924254B2 (en) 2021-05-03 2024-03-05 Apple Inc. Digital assistant hardware abstraction

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10666581B2 (en) * 2017-04-26 2020-05-26 Google Llc Instantiation of dialog process at a particular child node state
US11075982B2 (en) * 2017-07-10 2021-07-27 Nokia Solutions And Networks Oy Scaling hosts in distributed event handling systems
US11385944B2 (en) 2017-07-10 2022-07-12 Nokia Solutions And Networks Oy Event handling in distributed event handling systems
US11551135B2 (en) * 2017-09-29 2023-01-10 Oracle International Corporation Techniques for generating a hierarchical model to identify a class among a plurality of classes
US20200019641A1 (en) * 2018-07-10 2020-01-16 International Business Machines Corporation Responding to multi-intent user input to a dialog system
CN110754948B (en) * 2018-07-27 2021-10-08 九阳股份有限公司 Intention identification method in cooking process and intelligent cooking equipment
US11164562B2 (en) * 2019-01-10 2021-11-02 International Business Machines Corporation Entity-level clarification in conversation services
WO2020166183A1 (en) * 2019-02-13 2020-08-20 ソニー株式会社 Information processing device and information processing method
US20200293874A1 (en) * 2019-03-12 2020-09-17 Microsoft Technology Licensing, Llc Matching based intent understanding with transfer learning
KR102271068B1 (en) * 2019-05-29 2021-06-30 주식회사 카카오 Method and apparatus for controlling learning of a model for estimating the intention of input utterance
CN110457709A (en) * 2019-08-16 2019-11-15 北京一链数云科技有限公司 Outgoing call dialog process method, apparatus and server
US11227127B2 (en) * 2019-09-24 2022-01-18 International Business Machines Corporation Natural language artificial intelligence topology mapping for chatbot communication flow
US11586677B2 (en) 2019-12-12 2023-02-21 International Business Machines Corporation Resolving user expression having dependent intents
US11481442B2 (en) 2019-12-12 2022-10-25 International Business Machines Corporation Leveraging intent resolvers to determine multiple intents
CN111814484B (en) * 2020-07-03 2024-01-26 海信视像科技股份有限公司 Semantic recognition method, semantic recognition device, electronic equipment and readable storage medium
US11551674B2 (en) * 2020-08-18 2023-01-10 Bank Of America Corporation Multi-pipeline language processing platform
CN112115249B (en) * 2020-09-27 2023-11-14 支付宝(杭州)信息技术有限公司 Statistical analysis and result display method and device for user intention
US20230135625A1 (en) * 2021-10-28 2023-05-04 International Business Machines Corporation Automated generation of dialogue flow from documents

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5418717A (en) 1990-08-27 1995-05-23 Su; Keh-Yih Multiple score language processing system
US5434777A (en) * 1992-05-27 1995-07-18 Apple Computer, Inc. Method and apparatus for processing natural language
US5748841A (en) * 1994-02-25 1998-05-05 Morin; Philippe Supervised contextual language acquisition system
US20050086592A1 (en) 2003-10-15 2005-04-21 Livia Polanyi Systems and methods for hybrid text summarization
US20060080101A1 (en) * 2004-10-12 2006-04-13 At&T Corp. Apparatus and method for spoken language understanding by using semantic role labeling
US20060129397A1 (en) * 2004-12-10 2006-06-15 Microsoft Corporation System and method for identifying semantic intent from acoustic information
US20060149544A1 (en) 2005-01-05 2006-07-06 At&T Corp. Error prediction in spoken dialog systems
US20070055529A1 (en) 2005-08-31 2007-03-08 International Business Machines Corporation Hierarchical methods and apparatus for extracting user intent from spoken utterances
US20100083348A1 (en) * 2008-09-29 2010-04-01 Software Ag Method and rule-repository for generating security-definitions for heterogeneous systems
US20100100380A1 (en) * 2006-06-09 2010-04-22 At&T Corp. Multitask Learning for Spoken Language Understanding
US20100179956A1 (en) 2009-01-09 2010-07-15 Ravi Chandra Jammalamadaka Analyzing queries to generate product intention rules
KR20110072847A (en) 2009-12-23 2011-06-29 삼성전자주식회사 Dialog management system or method for processing information seeking dialog
US8315849B1 (en) * 2010-04-09 2012-11-20 Wal-Mart Stores, Inc. Selecting terms in a document
WO2015151157A1 (en) 2014-03-31 2015-10-08 三菱電機株式会社 Device and method for understanding user intent
US9466297B2 (en) * 2014-12-09 2016-10-11 Microsoft Technology Licensing, Llc Communication system
US9633004B2 (en) * 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9009046B1 (en) * 2005-09-27 2015-04-14 At&T Intellectual Property Ii, L.P. System and method for disambiguating multiple intents in a natural language dialog system
US8321220B1 (en) * 2005-11-30 2012-11-27 At&T Intellectual Property Ii, L.P. System and method of semi-supervised learning for spoken language understanding using semantic role labeling
US8700398B2 (en) * 2011-11-29 2014-04-15 Nuance Communications, Inc. Interface for setting confidence thresholds for automatic speech recognition and call steering applications
US10176167B2 (en) * 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US9966065B2 (en) * 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US10614799B2 (en) * 2014-11-26 2020-04-07 Voicebox Technologies Corporation System and method of providing intent predictions for an utterance prior to a system detection of an end of the utterance
EP4030295A1 (en) * 2016-04-18 2022-07-20 Google LLC Automated assistant invocation of appropriate agent
US10268679B2 (en) * 2016-12-02 2019-04-23 Microsoft Technology Licensing, Llc Joint language understanding and dialogue management using binary classification based on forward and backward recurrent neural network
US10216832B2 (en) * 2016-12-19 2019-02-26 Interactions Llc Underspecification of intents in a natural language processing system
US10510336B2 (en) * 2017-06-12 2019-12-17 International Business Machines Corporation Method, apparatus, and system for conflict detection and resolution for competing intent classifiers in modular conversation system
US10452782B1 (en) * 2018-02-20 2019-10-22 Facebook, Inc. Systems and methods for distributing intent models

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5418717A (en) 1990-08-27 1995-05-23 Su; Keh-Yih Multiple score language processing system
US5434777A (en) * 1992-05-27 1995-07-18 Apple Computer, Inc. Method and apparatus for processing natural language
US5625814A (en) * 1992-05-27 1997-04-29 Apple Computer, Inc. Method and apparatus for processing natural language with a hierarchy of mapping routines
US5748841A (en) * 1994-02-25 1998-05-05 Morin; Philippe Supervised contextual language acquisition system
US20050086592A1 (en) 2003-10-15 2005-04-21 Livia Polanyi Systems and methods for hybrid text summarization
US20060080101A1 (en) * 2004-10-12 2006-04-13 At&T Corp. Apparatus and method for spoken language understanding by using semantic role labeling
US7634406B2 (en) * 2004-12-10 2009-12-15 Microsoft Corporation System and method for identifying semantic intent from acoustic information
US20060129397A1 (en) * 2004-12-10 2006-06-15 Microsoft Corporation System and method for identifying semantic intent from acoustic information
US20060149544A1 (en) 2005-01-05 2006-07-06 At&T Corp. Error prediction in spoken dialog systems
US20070055529A1 (en) 2005-08-31 2007-03-08 International Business Machines Corporation Hierarchical methods and apparatus for extracting user intent from spoken utterances
US20100100380A1 (en) * 2006-06-09 2010-04-22 At&T Corp. Multitask Learning for Spoken Language Understanding
US20100083348A1 (en) * 2008-09-29 2010-04-01 Software Ag Method and rule-repository for generating security-definitions for heterogeneous systems
US20100179956A1 (en) 2009-01-09 2010-07-15 Ravi Chandra Jammalamadaka Analyzing queries to generate product intention rules
KR20110072847A (en) 2009-12-23 2011-06-29 삼성전자주식회사 Dialog management system or method for processing information seeking dialog
US8315849B1 (en) * 2010-04-09 2012-11-20 Wal-Mart Stores, Inc. Selecting terms in a document
US20130103389A1 (en) * 2010-04-09 2013-04-25 Wal-Mart Stores, Inc. Selecting Terms in a Document
WO2015151157A1 (en) 2014-03-31 2015-10-08 三菱電機株式会社 Device and method for understanding user intent
US9633004B2 (en) * 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9466297B2 (en) * 2014-12-09 2016-10-11 Microsoft Technology Licensing, Llc Communication system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PCT International Search Report and Written Opinion, PCT Application No. PCT/US2017/057867, dated Jan. 30, 2018, 9 pages.

Cited By (82)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11671920B2 (en) 2007-04-03 2023-06-06 Apple Inc. Method and system for operating a multifunction portable electronic device using voice-activation
US11900936B2 (en) 2008-10-02 2024-02-13 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
US11348582B2 (en) 2008-10-02 2022-05-31 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
US11423886B2 (en) 2010-01-18 2022-08-23 Apple Inc. Task flow identification based on user intent
US11120372B2 (en) 2011-06-03 2021-09-14 Apple Inc. Performing actions associated with task items that represent tasks to perform
US11321116B2 (en) 2012-05-15 2022-05-03 Apple Inc. Systems and methods for integrating third party services with a digital assistant
US11862186B2 (en) 2013-02-07 2024-01-02 Apple Inc. Voice trigger for a digital assistant
US11636869B2 (en) 2013-02-07 2023-04-25 Apple Inc. Voice trigger for a digital assistant
US10978090B2 (en) 2013-02-07 2021-04-13 Apple Inc. Voice trigger for a digital assistant
US11557310B2 (en) 2013-02-07 2023-01-17 Apple Inc. Voice trigger for a digital assistant
US11388291B2 (en) 2013-03-14 2022-07-12 Apple Inc. System and method for processing voicemail
US11798547B2 (en) 2013-03-15 2023-10-24 Apple Inc. Voice activated device for use with a voice-based digital assistant
US11727219B2 (en) 2013-06-09 2023-08-15 Apple Inc. System and method for inferring user intent from speech inputs
US11257504B2 (en) 2014-05-30 2022-02-22 Apple Inc. Intelligent assistant for home automation
US11699448B2 (en) 2014-05-30 2023-07-11 Apple Inc. Intelligent assistant for home automation
US11670289B2 (en) 2014-05-30 2023-06-06 Apple Inc. Multi-command single utterance input method
US11133008B2 (en) 2014-05-30 2021-09-28 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US11810562B2 (en) 2014-05-30 2023-11-07 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US11516537B2 (en) 2014-06-30 2022-11-29 Apple Inc. Intelligent automated assistant for TV user interactions
US11838579B2 (en) 2014-06-30 2023-12-05 Apple Inc. Intelligent automated assistant for TV user interactions
US11842734B2 (en) 2015-03-08 2023-12-12 Apple Inc. Virtual assistant activation
US11087759B2 (en) 2015-03-08 2021-08-10 Apple Inc. Virtual assistant activation
US11070949B2 (en) 2015-05-27 2021-07-20 Apple Inc. Systems and methods for proactively identifying and surfacing relevant content on an electronic device with a touch-sensitive display
US11126400B2 (en) 2015-09-08 2021-09-21 Apple Inc. Zero latency digital assistant
US11550542B2 (en) 2015-09-08 2023-01-10 Apple Inc. Zero latency digital assistant
US11853536B2 (en) 2015-09-08 2023-12-26 Apple Inc. Intelligent automated assistant in a media environment
US11500672B2 (en) 2015-09-08 2022-11-15 Apple Inc. Distributed personal assistant
US11809483B2 (en) 2015-09-08 2023-11-07 Apple Inc. Intelligent automated assistant for media search and playback
US11809886B2 (en) 2015-11-06 2023-11-07 Apple Inc. Intelligent automated assistant in a messaging environment
US11526368B2 (en) 2015-11-06 2022-12-13 Apple Inc. Intelligent automated assistant in a messaging environment
US11886805B2 (en) 2015-11-09 2024-01-30 Apple Inc. Unconventional virtual assistant interactions
US11853647B2 (en) 2015-12-23 2023-12-26 Apple Inc. Proactive assistance based on dialog communication between devices
US11657820B2 (en) 2016-06-10 2023-05-23 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US11037565B2 (en) 2016-06-10 2021-06-15 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US11809783B2 (en) 2016-06-11 2023-11-07 Apple Inc. Intelligent device arbitration and control
US11749275B2 (en) 2016-06-11 2023-09-05 Apple Inc. Application integration with a digital assistant
US11152002B2 (en) 2016-06-11 2021-10-19 Apple Inc. Application integration with a digital assistant
US10796100B2 (en) * 2016-12-19 2020-10-06 Interactions Llc Underspecification of intents in a natural language processing system
US20190147044A1 (en) * 2016-12-19 2019-05-16 Interactions Llc Underspecification of intents in a natural language processing system
US11467802B2 (en) 2017-05-11 2022-10-11 Apple Inc. Maintaining privacy of personal information
US11599331B2 (en) 2017-05-11 2023-03-07 Apple Inc. Maintaining privacy of personal information
US11405466B2 (en) 2017-05-12 2022-08-02 Apple Inc. Synchronization and task delegation of a digital assistant
US11538469B2 (en) 2017-05-12 2022-12-27 Apple Inc. Low-latency intelligent automated assistant
US11580990B2 (en) 2017-05-12 2023-02-14 Apple Inc. User-specific acoustic models
US11862151B2 (en) 2017-05-12 2024-01-02 Apple Inc. Low-latency intelligent automated assistant
US11837237B2 (en) 2017-05-12 2023-12-05 Apple Inc. User-specific acoustic models
US11380310B2 (en) 2017-05-12 2022-07-05 Apple Inc. Low-latency intelligent automated assistant
US11675829B2 (en) 2017-05-16 2023-06-13 Apple Inc. Intelligent automated assistant for media exploration
US11532306B2 (en) 2017-05-16 2022-12-20 Apple Inc. Detecting a trigger of a digital assistant
US10733375B2 (en) * 2018-01-31 2020-08-04 Apple Inc. Knowledge-based framework for improving natural language understanding
US11710482B2 (en) 2018-03-26 2023-07-25 Apple Inc. Natural assistant interaction
US11854539B2 (en) 2018-05-07 2023-12-26 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US11900923B2 (en) 2018-05-07 2024-02-13 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US11907436B2 (en) 2018-05-07 2024-02-20 Apple Inc. Raise to speak
US11169616B2 (en) 2018-05-07 2021-11-09 Apple Inc. Raise to speak
US11487364B2 (en) 2018-05-07 2022-11-01 Apple Inc. Raise to speak
US11009970B2 (en) 2018-06-01 2021-05-18 Apple Inc. Attention aware virtual assistant dismissal
US11630525B2 (en) 2018-06-01 2023-04-18 Apple Inc. Attention aware virtual assistant dismissal
US11431642B2 (en) 2018-06-01 2022-08-30 Apple Inc. Variable latency device coordination
US10984798B2 (en) 2018-06-01 2021-04-20 Apple Inc. Voice interaction at a primary device to access call functionality of a companion device
US11360577B2 (en) 2018-06-01 2022-06-14 Apple Inc. Attention aware virtual assistant dismissal
US10685645B2 (en) * 2018-08-09 2020-06-16 Bank Of America Corporation Identification of candidate training utterances from human conversations with an intelligent interactive assistant
US11893992B2 (en) 2018-09-28 2024-02-06 Apple Inc. Multi-modal inputs for voice commands
US11783815B2 (en) 2019-03-18 2023-10-10 Apple Inc. Multimodality in digital assistant systems
US11675491B2 (en) 2019-05-06 2023-06-13 Apple Inc. User configurable task triggers
US11705130B2 (en) 2019-05-06 2023-07-18 Apple Inc. Spoken notifications
US11888791B2 (en) 2019-05-21 2024-01-30 Apple Inc. Providing message response suggestions
US11657813B2 (en) 2019-05-31 2023-05-23 Apple Inc. Voice identification in digital assistant systems
US11237797B2 (en) 2019-05-31 2022-02-01 Apple Inc. User activity shortcut suggestions
US11790914B2 (en) 2019-06-01 2023-10-17 Apple Inc. Methods and user interfaces for voice-based control of electronic devices
US11606463B1 (en) 2020-03-31 2023-03-14 Interactions Llc Virtual assistant architecture for natural language understanding in a customer service system
US11743378B1 (en) 2020-03-31 2023-08-29 Interactions Llc Intelligent agent assistant for natural language understanding in a customer service system
US11914848B2 (en) 2020-05-11 2024-02-27 Apple Inc. Providing relevant data items based on context
US11765209B2 (en) 2020-05-11 2023-09-19 Apple Inc. Digital assistant hardware abstraction
US11755276B2 (en) 2020-05-12 2023-09-12 Apple Inc. Reducing description length based on confidence
US11838734B2 (en) 2020-07-20 2023-12-05 Apple Inc. Multi-device audio adjustment coordination
US11696060B2 (en) 2020-07-21 2023-07-04 Apple Inc. User identification using headphones
US11750962B2 (en) 2020-07-21 2023-09-05 Apple Inc. User identification using headphones
US11393475B1 (en) 2021-01-13 2022-07-19 Artificial Solutions Iberia S.L Conversational system for recognizing, understanding, and acting on multiple intents and hypotheses
US20230197068A1 (en) * 2021-03-05 2023-06-22 Capital One Services, Llc Systems and methods for dynamically updating machine learning models that provide conversational responses
US11798540B2 (en) * 2021-03-05 2023-10-24 Capital One Services, Llc Systems and methods for dynamically updating machine learning models that provide conversational responses
US11924254B2 (en) 2021-05-03 2024-03-05 Apple Inc. Digital assistant hardware abstraction

Also Published As

Publication number Publication date
WO2018118202A1 (en) 2018-06-28
US20190147044A1 (en) 2019-05-16
US10796100B2 (en) 2020-10-06
US20180174578A1 (en) 2018-06-21

Similar Documents

Publication Publication Date Title
US10796100B2 (en) Underspecification of intents in a natural language processing system
US11087094B2 (en) System and method for generation of conversation graphs
EP3180742B1 (en) Generating and using a knowledge-enhanced model
US8719192B2 (en) Transfer of learning for query classification
US9785717B1 (en) Intent based search result interaction
JP2019503526A (en) Parameter collection and automatic dialog generation in dialog systems
US11449682B2 (en) Adjusting chatbot conversation to user personality and mood
WO2018045646A1 (en) Artificial intelligence-based method and device for human-machine interaction
US10108698B2 (en) Common data repository for improving transactional efficiencies of user interactions with a computing device
CN110770694A (en) Obtaining response information from multiple corpora
US11574144B2 (en) Performance of a computer-implemented model that acts as a multi-class classifier
US20140297574A1 (en) Probabilistic language model in contextual network
US11238111B2 (en) Response generation
US20220337539A1 (en) Personality reply for digital content
US11531821B2 (en) Intent resolution for chatbot conversations with negation and coreferences
CN112417090A (en) Using uncommitted user input data to improve task performance
US11769013B2 (en) Machine learning based tenant-specific chatbots for performing actions in a multi-tenant system
CN117296058A (en) Variant Inconsistent Attacks (VIA) as a simple and effective method of combating attacks
US10942954B2 (en) Dataset adaptation for high-performance in specific natural language processing tasks
WO2022125096A1 (en) Method and system for resume data extraction
US20200118008A1 (en) Building domain models from dialog interactions
CN116186219A (en) Man-machine dialogue interaction method, system and storage medium
US20230108637A1 (en) Generating sorted lists of chat bot design nodes using escalation logs
US20230095036A1 (en) Method and system for proficiency identification
US11775758B2 (en) Computing system for entity disambiguation and not-in-list entity detection in a knowledge graph

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERACTIONS LLC, MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BANGALORE, SRINIVAS;CHEN, JOHN;REEL/FRAME:040751/0374

Effective date: 20161219

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: SILICON VALLEY BANK, MASSACHUSETTS

Free format text: INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNOR:INTERACTIONS LLC;REEL/FRAME:049388/0082

Effective date: 20190603

Owner name: SILICON VALLEY BANK, MASSACHUSETTS

Free format text: FIRST AMENDMENT TO AMENDED AND RESTATED INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNOR:INTERACTIONS LLC;REEL/FRAME:049388/0152

Effective date: 20190603

AS Assignment

Owner name: RUNWAY GROWTH FINANCE CORP., ILLINOIS

Free format text: INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNORS:INTERACTIONS LLC;INTERACTIONS CORPORATION;REEL/FRAME:060445/0733

Effective date: 20220624

AS Assignment

Owner name: INTERACTIONS LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY RECORDED AT REEL/FRAME: 049388/0082;ASSIGNOR:SILICON VALLEY BANK;REEL/FRAME:060558/0474

Effective date: 20220624

AS Assignment

Owner name: INTERACTIONS LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY RECORDED AT REEL/FRAME: 049388/0152;ASSIGNOR:SILICON VALLEY BANK;REEL/FRAME:060558/0719

Effective date: 20220624

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: RUNWAY GROWTH FINANCE CORP., ILLINOIS

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE THE APPLICATION NUMBER PREVIOUSLY RECORDED AT REEL: 060445 FRAME: 0733. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:INTERACTIONS LLC;INTERACTIONS CORPORATION;REEL/FRAME:062919/0063

Effective date: 20220624