US20180189628A1 - Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications - Google Patents
Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications Download PDFInfo
- Publication number
- US20180189628A1 US20180189628A1 US15/394,869 US201615394869A US2018189628A1 US 20180189628 A1 US20180189628 A1 US 20180189628A1 US 201615394869 A US201615394869 A US 201615394869A US 2018189628 A1 US2018189628 A1 US 2018189628A1
- Authority
- US
- United States
- Prior art keywords
- response
- responses
- embedding
- candidate
- electronic communication
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004044 response Effects 0.000 title claims abstract description 504
- 238000004891 communication Methods 0.000 title claims abstract description 172
- 238000000034 method Methods 0.000 claims abstract description 61
- 238000013528 artificial neural network Methods 0.000 claims abstract description 37
- 238000011524 similarity measure Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 12
- 230000015654 memory Effects 0.000 claims description 9
- 230000006403 short-term memory Effects 0.000 claims description 5
- 238000003062 neural network model Methods 0.000 description 18
- 239000000463 material Substances 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006855 networking Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/107—Computer-aided management of electronic mailing [e-mailing]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/55—Rule-based translation
- G06F40/56—Natural language generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
-
- H04L51/36—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/56—Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
Definitions
- Users are often inundated with electronic communications such as emails, SMS (short message service) communications, chat communications, social networking communications, etc.
- Many electronic communications that are sent to a user explicitly solicit a reply or otherwise contain information to which the user may wish to reply. For example, an email that contains “Are you interested in grabbing lunch?” may explicitly solicit a reply. An email that contains “I just got engaged!” may not explicitly solicit a reply, but a user that receives the email may still wish to send a congratulatory reply to the email.
- Users formulate replies to the electronic communications and input the formulated replies at client devices utilizing a keyboard (e.g., a virtual keyboard of a smartphone or tablet), a microphone, and/or other user interface input device(s) of the client device.
- a keyboard e.g., a virtual keyboard of a smartphone or tablet
- microphone e.g., a microphone, and/or other user interface input device(s) of the client device.
- responses such as “Sure, what time?”, “Thanks, but I'm busy”, and “I'll get back to you” may be provided as suggestions based on their contextual relevance to the received chat message of “Are you interested in grabbing lunch?”.
- Selection of one of the provided responses by the user will reduce the number of user inputs that a user must make to formulate the reply, which may reduce the usage of various client device computational resources and/or be of particular benefit to users that have low dexterity (or who otherwise have difficulties making user inputs to a client device).
- a subset of those candidate responses may be selected for initial presentation to the user as suggestions. Implementations disclosed herein seek to select such a subset in a manner that promotes semantic diversity among the responses of the subset.
- some implementations of this specification are directed generally to methods and apparatus related to determining a semantically diverse subset of candidate responses to provide for initial presentation to a user as suggestions for inclusion in a reply to an electronic communication. Some of those implementations determine the semantically diverse subset of candidate responses based on generating, over a response encoder model, embeddings that are each based on one of the plurality of the candidate responses. The embedding based on a given candidate response may be compared to embedding(s) of candidate response(s) already selected for the subset, and the given candidate response added to the subset only if the comparing indicates a difference criterion is satisfied.
- the difference criterion may be satisfaction of a threshold that indicates a sufficient degree of semantic difference between the given candidate response and already selected candidate response(s).
- a “response encoder model” is a model that can be used to process a sequence of tokens (e.g., included in a response) to generate an embedding of that sequence.
- a token of a candidate response may be a textual token (e.g., a word, a phrase) included in the candidate response or a graphical token (e.g., a sticker or an emoji, such as a thumbs-up emoji or a smiley emoji) included in the candidate response.
- a candidate response of “Hope you feel better soon” includes textual tokens of “Hope”, “you”, “feel”, “better”, and “soon”.
- a candidate response of “Can't wait ” includes textual tokens of “Can't” and “wait”, and a graphical token of “ ”.
- the embedding of a sequence of tokens of a candidate response is a dense representation (i.e., a real-valued vector) of the sequence of tokens.
- a response encoder model is a neural network model.
- Determining and providing a semantically diverse subset of responses for initial presentation to a user as suggestions may increase the likelihood that one of the provided responses is sufficient to convey the essence of an intended response of the user. As a result, a likelihood that a user will select one of the provided responses for inclusion in a reply may be increased. As described above, selecting one of the provided responses will reduce the number of user inputs that a user must make to formulate the reply, which may reduce the usage of various client device computational resources and/or be of particular benefit to users that have low dexterity (or who otherwise have difficulties making user inputs to a client device).
- a method implemented by one or more processors includes identifying a candidate set of responses for consideration for inclusion in a reply to a transmitted electronic communication directed to a user.
- Each of the responses of the candidate set include one or more tokens and is determined based on content of the electronic communication.
- the method further includes: selecting a first response, of the responses of the candidate set, for inclusion in a selected set; generating a first embedding over a neural network response encoder model based on applying the first response as input to the neural network response encoder model; and generating a second embedding over the neural network response encoder model based on applying a second response, of the responses of the candidate set, as input to the neural network response encoder model.
- the method further includes determining that the second response satisfies a difference criterion relative to the first response based on comparing the first embedding to the second embedding.
- the method further includes: based on determining that the second response satisfies the difference criterion, selecting the second response for inclusion in the selected set; and providing the responses of the selected set for display in an interface rendered by a client device of the user.
- the interface enables selection of any one of the responses of the selected set for inclusion in a reply electronic communication that is a reply by the user to the electronic communication.
- one or more of the processors are remote from the client device and providing the responses of the selected set for display in the interface includes transmitting content to the client device to cause the client device to visually render the responses of the selected set in the interface a selectable manner.
- the method further includes receiving a selection of a given response of the responses of the selected set, and incorporating the response into the reply communication in response to receiving the selection.
- the selection may be based on user interface input generated by a user interface input device of the client device
- the neural network response encoder model includes long short-term memory cells.
- generating the first embedding includes using a state of one or more of the long short-term memory cells of the neural network response encoder model as the embedding, where the state is the state after applying the first response to the neural network response encoder model.
- generating the first embedding is further based on applying at least part of the electronic communication to the neural network response encoder model prior to applying the first response; and generating the second embedding is further based on applying at least part of the electronic communication to the neural network response encoder model prior to applying the second response to the neural network response encoder model.
- determining that the second response satisfies the difference criterion relative to the first response based on comparing the first embedding to the second embedding includes: calculating a measure between the first embedding and the second embedding, and determining that the measure satisfies a threshold.
- the measure is a cosine similarity measure, a dot product, or a Euclidian distance.
- the candidate set of responses are ranked and selecting the first response is based on it being the most highly ranked.
- the method further includes: identifying an additional response, of the responses of the candidate set, that is the next most highly ranked response following the first response; generating an additional embedding over the neural network response encoder model based on applying the additional response as input to the neural network response encoder model; determining that the additional response fails to satisfy the difference criterion relative to the first response based on comparing the first embedding and the additional embedding; and omitting the additional response from the selected set based on the second response failing to satisfy the difference criterion.
- the method further includes, subsequent to selecting the second response for inclusion in the selected set: identifying an additional response of the responses of the candidate set; generating an additional embedding over the neural network response encoder model based on applying the additional response as input to the neural network response encoder model; determining that the additional response satisfies the difference criterion relative to at least one of: the first response and the second response; and selecting the additional response for inclusion in the selected set based on the second response satisfying the difference criterion relative to at least one of the first response and the second response.
- providing the responses of the selected set occurs independent of any input provided via the client device in generating the reply electronic communication.
- a method implemented by one or more processors includes identifying a ranked candidate set of responses for consideration for inclusion in a reply to an electronic communication transmitted to a user.
- Each of the responses of the candidate set include one or more tokens and is determined based on content of the electronic communication.
- the method further includes, selecting a first response, of the responses of the candidate set, for inclusion in a selected set, based on it being the most highly ranked of the responses of the candidate set.
- the method further includes, for each of a plurality remaining responses of the responses of the candidate set, and until at least one condition has been satisfied: comparing an embedding vector of the remaining response to a first embedding vector of the first response and any embedding vectors of any additional responses selected for inclusion in the selected set, and determining whether to include the remaining response in the selected set based on the comparing.
- the method further includes providing the responses of the selected set for display in an interface rendered by a client device of the user. The interface enables selection of any one of the responses of the selected set for inclusion in a reply electronic communication that is a reply by the user to the electronic communication.
- the at least one condition includes that a threshold quantity of the responses have been selected for inclusion in the selected set.
- one or more of the processors are remote from the client device and providing the responses of the selected set for display in the interface includes transmitting content to the client device to cause the client device to visually render the responses of the selected set in the interface a selectable manner.
- the method further includes receiving a selection of a given response of the responses of the selected set, and incorporating the response into the reply communication in response to receiving the selection.
- implementations may include one or more non-transitory computer readable storage media storing instructions executable by a processor to perform a method such as one or more of the methods described above.
- implementations may include a system including memory and one or more processors operable to execute instructions, stored in the memory, to perform a method such as one or more of the methods described above.
- FIG. 1 illustrates an example environment in which a set of semantically diverse responses may be determined and provided as suggestions for inclusion in a reply electronic communication.
- FIG. 2 illustrates an example of how candidate responses may be determined based on a received electronic communication, and a semantically diverse subset of those candidate responses selected for providing as suggestions for inclusion in a reply to the received electronic communication.
- FIG. 3 is a flow chart illustrating an example method according to implementations disclosed herein.
- FIG. 4 illustrates example responses and minimum similarity scores for those responses that may be determined according to implementations disclosed herein.
- FIG. 5A and FIG. 5B each illustrate example graphical user interfaces for providing a semantically diverse subset of candidate responses as suggestion for inclusion in a reply to a received electronic communication.
- FIG. 6 illustrates an example architecture of a computing device.
- FIG. 1 illustrates an example environment in which a set of semantically diverse responses may be determined and provided as suggestions for inclusion in a reply electronic communication.
- the example environment includes a communication network 101 that facilitates communication between the various components in the environment.
- the communication network 101 may include a wide area network (WAN) such as the Internet, one or more intranets, and/or one or more bus subsystems.
- WAN wide area network
- the communication network 101 may optionally utilize one or more standard communications technologies, protocols, and/or inter-process communication techniques.
- the example environment of FIG. 1 also includes a client device 106 , an electronic communications system 110 , a response system 120 , at least one electronic communications database 152 , response prediction model(s) 154 , and response encoder model(s) 156 .
- Electronic communications system 110 and response system 120 may each be implemented in one or more computing devices that communicate, for example, through a network (e.g., network 101 and/or other network).
- Electronic communications system 110 and response system 120 are example systems in which the systems, components, and techniques described herein may be implemented and/or with which systems, components, and techniques described herein may interface.
- Electronic communications system 110 and response system 120 each include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network.
- electronic communications system 110 and response system 120 may include one or more components of the example computing device of FIG. 6 .
- the operations performed by electronic communications system 110 and response system 120 may be distributed across multiple computer systems.
- one or more aspects of electronic communications system 110 and response system 120 may be combined in a single system and/or one or more aspects may be implemented on the client device 106 .
- the electronic communications database 152 includes one or more storage mediums that include all, or portions of, electronic communications of a plurality of users. In some implementations, the electronic communications database 152 is maintained by the electronic communications system 110 .
- the electronic communications system 110 may include one or more email systems and the electronic communications database 152 may include a plurality of emails that are sent and/or received via the email systems.
- the electronic communications system 110 may include one or more social networking systems and the electronic communications database 152 may include a plurality of messages, posts, or other communications that are sent and/or received via the social networking systems.
- an “electronic communication” or “communication” may refer to an email, a text message (e.g., SMS, MMS), an instant message, a transcribed voicemail, or any other electronic communication that is sent from a first user to a restricted group of one or more additional users.
- an electronic communication may include various metadata and the metadata may optionally be utilized in one or more techniques described herein.
- an electronic communication such as an email may include an electronic communication address such as one or more sender identifiers (e.g., sender email addresses), one or more recipient identifiers (e.g., recipient email addresses, including cc′d and bcc′d recipients), a date sent, one or more attachments, a subject, a type of device that sent and/or received the electronic communication, and so forth.
- sender email addresses e.g., sender email addresses
- recipient identifiers e.g., recipient email addresses, including cc′d and bcc′d recipients
- an electronic communication may include an “initial” message from a first user and an additional message that is a reply message from a second user.
- an electronic communication may include an “initial” message from a first user and an additional message that is also from the first user.
- candidate responses described herein may be generated based on a received communication that includes just a single initial message, or a received communication that includes the initial message and one or more additional messages.
- An electronic communication may be a single document such as an email that includes both an initial message and one or more additional messages, and that can optionally be processed to distinguish the original message and the additional message(s). Processing an electronic communication to distinguish an initial message and an additional message may include “splitting” the message based on presence of metadata, message breaks, header information included in the message, quotes provided around the initial message, etc.
- An electronic communication may also be multiple documents that are mapped to one another based on at least one of the multiple documents being responsive to other of the multiple documents. For example, an electronic communication may include a first email that is an original message received by a user and a second email sent by the user as a reply to that original message, or original and reply SMS messages.
- the mapping of multiple documents to one another may be performed, for example, by the electronic communications system 110 .
- the electronic communications system 110 may map a first email to a second email based on a user selecting a “reply” user interface element when viewing the first email and then drafting the second email responsive to selecting the reply user interface element.
- response system 120 determines and provides suggested responses to include in a reply to an electronic communication, such as an electronic communication provided by client device 106 and/or an electronic communication of electronic communications database 152 to which a user has yet to reply.
- the response system 120 may determine a candidate set of responses based on one or more message features of the electronic communication, and may select a subset of the responses of the candidate set for actually providing as suggestions to include in a reply.
- the response system 120 may provide the subset of the responses as suggestions independent of any textual input provided by the user in generating the reply to the communication and/or independent any other content (e.g., voice input, graphical elements) provided by the user in generating the reply to the communication.
- response system 120 may include a candidate responses engine 122 , a response selection engine 124 , and a presentation engine 128 .
- the response selection engine 124 may include a response embedding module 125 and an embedding comparison module 126 .
- aspects of one or more of engines 122 , 124 , 128 , and/or modules 125 , 126 may be omitted, combined, and/or implemented in a component that is separate from response system 120 , such as client device 106 and/or electronic communications system 110 .
- response system 120 may determine suggested responses for additional electronic communications of the user and/or of additional users.
- the response system 120 may manage determination of suggested responses for each of multiple client devices via different sessions and may manage multiple sessions in parallel.
- the response system 120 in some implementations may be implemented as a cloud-based service employing a cloud infrastructure, e.g., using a server farm or cluster of high performance computers running software suitable for handling high volumes of requests from multiple users.
- many examples herein are described with respect to a single client device 106 and a single electronic communication.
- the candidate responses engine 122 generates a candidate set of responses based on a transmitted electronic communication directed to at least one user (i.e., based at least in part on the content of the electronic communication as that content was sent to the user). In some implementations, the candidate responses engine 122 generates the candidate set of responses based on application of features of the communication to one or more response prediction models 154 .
- the features include tokens (e.g., textual token(s) and/or graphical token(s)) from the communication and the candidate responses engine 122 applies the tokens to a long short-term memory (LSTM) neural network model of the response prediction models 154 to generate output indicative of candidate responses.
- the applied tokens from the communication may include all of the tokens, or one or more subsets of the tokens (e.g., first sentence and/or last sentence, body of the communication only, subject and body, all of the communication with stopwords omitted).
- the candidate responses engine 122 uses the output to determine candidate responses. For example, the candidate responses engine 122 may apply the tokens as input to the LSTM neural network model on a sequential token by token basis to generate, over the model, one or more outputs that each indicate a probability distribution over words.
- the candidate responses engine 122 utilizes the output(s) of the LSTM neural network model to determine a set of candidate responses and to assign scores to (or otherwise rank) the candidate responses.
- the candidate responses are identified from a whitelist of candidate responses, such as a curated list of common responses.
- one or more of the candidate responses may additionally or alternatively be generated in an unrestricted token by token manner (i.e., not identified from a whitelist of common responses).
- the candidate responses engine 122 may identify a number of candidate responses (e.g., 10, 20, 30, or more) and may optionally rank the identified candidate responses based on, for example, scores assigned to the candidate response based on the output generated over the LSTM neural network model.
- the candidate responses engine 122 may communicate the candidate responses and the ranking to the response selection engine 124 .
- the ranking of the candidate responses may be conveyed as scores for the candidate responses, an ordering of the candidate responses (e.g., a list where the first candidate response is the most highly ranked, the second candidate response is the next most highly ranked), etc.
- the candidate responses engine 122 may only identify up to X candidate responses and/or use thresholding to identify only candidate responses that have at least a threshold degree of conformance to the output generated over the LSTM neural network model.
- the LSTM neural network model and/or other model of the one or more response prediction models 154 may be trained based on a large corpus of communications that include original messages and reply messages.
- the neural network model may be trained to maximize the probability of features of observed reply messages given features of their respective original messages.
- the candidate responses engine 122 may utilize additional and/or alternative techniques to generate a set of candidate responses.
- the candidate responses engine 122 may utilize a response prediction model that is not an LSTM neural network model, may utilize feature(s) of a received electronic communication that are in addition to text of the electronic communication, etc.
- the candidate response engine 122 may utilize features that are based on one or more natural language processing tags or other labels applied to text of the electronic communication (e.g., parts of speech, named entities, entity types, tone); features based on text that occurs specifically in the subjects, the first sentences, the last sentences, or other portion of the electronic communication; features based on metadata of the electronic communication such as a time the electronic communication was sent, day of the week the electronic communication was sent, a number of recipients, a type of device that sent the electronic communication, etc.
- the candidate responses engine 122 provides the set of candidate responses to the response selection engine 124 .
- the provided set of candidate responses are ranked. For example, they may be ordered based on the ranking and/or each include a score or other value that indicates its ranking.
- the response selection engine 124 selects the most highly ranked candidate response for inclusion in a selected set to provide to presentation engine 128 for presentation as suggested responses.
- the response embedding module 125 utilizes one or more response encoder models 156 to generate an embedding based on the most highly ranked candidate response and an additional embedding for each of a plurality of additional considered candidate responses of the provided set of candidate responses.
- the response embedding module 125 generates the embeddings “live” (e.g., in response to receiving the candidate responses from the candidate responses engine 122 ).
- the response embedding module 125 may additionally or alternatively pre-generate one or more of the embeddings (e.g., pre-generate embeddings for each of a plurality of common candidate responses).
- An embedding based on a response is a vector of values that are determined based on tokens of the response and that encode at least some of the semantics of the response.
- the embedding may be of the response and may be a transformation of the response into a vector of values that represent the semantics of the response.
- the embedding may be based on the response and also take into account context of an original communication for which a response is a candidate.
- the response embedding module 125 utilizes a trained neural network model of the response encoder models 156 to generate an embedding based on a response.
- the response embedding module 125 may apply the response as input to the neural network model and generate the embedding over the model based on the input.
- the embedding may be the final output of the model, or may be derived from the hidden state of the model.
- the response embedding module 125 may generate an embedding of a response based on an LSTM neural network model.
- the LSTM neural network model may optionally be the same as that optionally utilized by the candidate responses engine 122 and described above.
- the response embedding module 125 may apply the response sequentially on a token by token basis as input to the LSTM neural network model to generate a vector representation of the response in a hidden layer of the LSTM neural network model, and may utilize the vector representation of the response as the embedding of the response.
- the response embedding module 125 may additionally or alternatively utilize other neural network models that generate an embedding of a token (in hidden layer(s) or as end output), such as a GloVE embedding model and/or a Word2Vec embedding model.
- a GloVE embedding model and/or a Word2Vec embedding model.
- the response embedding module 125 may optionally generate the embedding based on the response and also take into account context of a communication for which a response is a candidate. For example, the response embedding module 125 may initially apply tokens (e.g., all of the tokens or a subset) from the communication as input to the LSTM neural network model, then apply the response, and utilize the resulting vector representation in the hidden layer as the embedding of the response. This may effectively encode aspects of the communication in the embedding and result in an embedding that includes semantics of the response in view of the communication. For example, the embedding of “Got it!” can mean “I received it” or “understood” depending on the context of the original communication.
- tokens e.g., all of the tokens or a subset
- an embedding can be generated that more accurately conveys the semantics of the response in the context of the original message. For instance, the embedding of “Got it!” when first applying the original communication text of “Did you get my message?”, may differ significantly from the embedding of “Got it!” when first applying the original communication text of “No, the meeting is tomorrow”.
- the embedding comparison module 126 compares the embedding of each considered candidate response to embedding(s) of candidate response(s) already selected for the subset, and adds the candidate response to the subset only if the comparing indicates a difference criterion is satisfied. For example, assume that only a highest ranked candidate response has so far been selected for inclusion in the subset.
- the embedding comparison module 126 may compare an embedding of a lower ranked candidate response to the embedding of the highest ranked candidate response and only select the lower ranked candidate response for inclusion in the subset if the comparison indicates at least a threshold degree of semantic difference between the two embeddings.
- the embedding comparison module 126 may calculate a cosine similarity between the two embeddings, calculate a Euclidian distance between the two embeddings, and/or calculate a dot product of the two embeddings; and compare the cosine similarity, the Euclidian distance, and/or the dot product to a difference criterion threshold. If the difference criterion threshold is satisfied, the embedding comparison module 126 adds the lower ranked candidate response to the subset. Otherwise, the embedding comparison module 126 does not add the lower ranked candidate response to the subset.
- the embedding comparison module 126 may repeat this process until a threshold quantity of candidate response are selected for the subset, no candidate responses remain for consideration, and/or other condition is satisfied.
- the response selection engine 122 then provides the selected subset of candidate responses to the presentation engine 128 . Additional description of some implementations of the response selection engine 124 is provided below with respect to the method of FIG. 3 .
- the presentation engine 128 provides the selected subset of candidate responses for inclusion in a reply electronic communication that is a reply by the user to the electronic communication.
- the presentation engine 128 may present one or more user interface indications (e.g., graphical, audible) of the responses of the selected subset and, when a user generating the reply selects one of the indications, the corresponding response may be automatically incorporated in the reply.
- the reply may then be automatically sent and/or sent in response to further user interface input of the user (optionally after the user has added additional content to the reply).
- Such automatic incorporation of a selected response into a reply message saves the user from manually entering the selected response, and saves a number of user interaction steps and corresponding input required to establish a reply message that includes the response.
- the presentation engine 128 provides the responses independent of any input and/or other content provided via a computing device of the user in generating a reply to the electronic communication.
- the subset of the responses may be selected by the response selection engine 124 before the user has even viewed or otherwise consumed the communication.
- the candidate responses engine 122 and the response selection engine 124 may process a communication in advance of viewing by the user, and presentation engine 128 may append or otherwise associate the selected responses with the electronic communication before it is even viewed by the user. Hence, if a user views or otherwise consumes the electronic communication, suggested responses to the electronic communication can be provided promptly without delay.
- the presentation engine 128 may provide the selected subset of responses to the client device 106 for presentation to the user as options for including in a reply. In some implementations where the electronic communication is provided to the response system 120 by the electronic communications system 110 via the electronic communications database 152 , the presentation engine 128 may store an association of the selected subset of responses with the electronic communication in the electronic communications database 152 and/or other database. In some implementations, one or more (e.g., all) aspects of the presentation engine 128 may be implemented by the client device 106 and/or the electronic communications system 110 .
- the presentation engine 128 may also provide display prominence information with the selected subset of responses that indicates the prominence (e.g., position, size, color) with which those responses should be presented. Generally, a higher ranking for a particular response, the greater the prominence for that response. As one example, the optional ranking determined by the response embedding module 125 may be utilized to determine the ranking for the responses of the selected subset.
- the presentation engine 128 provides only the selected subset of responses (or associated identifiers), and potentially prominence information, and the client device 106 and/or electronic communications system 110 may generate a display of the selected documents based on the provided data. In some implementations, the presentation engine 128 may additionally provide some or all of the data necessary to generate the display. In some of those implementations, any provided prominence information may be incorporated in the data that indicates how the display should be presented.
- FIG. 2 illustrates an example of how candidate responses may be determined based on a received electronic communication, and a semantically diverse subset of those candidate responses selected for providing as suggestions for inclusion in a reply to the received electronic communication.
- candidate responses engine 122 utilizes one or more response prediction model(s) 154 to determine a set of candidate responses 153 .
- Electronic communication 152 A may be provided by, for example, client device 106 or electronic communications system 110 of FIG. 1 .
- the candidate responses engine 122 provides the set of candidate responses 153 to response selection engine 124 .
- the response selection engine 124 utilizes one or more response encoder models 156 to select a subset of the candidate responses 153 for presentation as suggestions in response to the electronic communication 152 A.
- the response selection engine 124 may utilize the response encoder model(s) 156 to generate embeddings of the candidate responses of the set of candidate responses 153 , and may use the embeddings to select a semantically diverse subset of the set of candidate responses 153 .
- the response selection engine 124 provides the selected responses 155 to the presentation engine 128 .
- the presentation engine 128 provides the selected response 155 for inclusion in a reply electronic communication that is a reply to the electronic communication 152 A.
- FIG. 3 is a flow chart illustrating an example method according to implementations disclosed herein. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems. For instance, some operations may be performed by the response selection engine 124 . Moreover, while operations of the method of FIG. 3 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.
- the system receives candidate responses for consideration in a reply to an electronic communication.
- the system may receive the candidate response from a candidate responses engine that generates the candidate response based on features of the electronic communication, such that the candidate response are contextually relevant to the electronic communication.
- the system selects one of the candidate responses for inclusion in a selected set.
- the candidate responses received at block 300 may be ranked and the system may select the most highly ranked candidate response for inclusion in the selected set.
- the system generates an embedding based on the selected response of block 302 .
- the system may generate the embedding over a neural network response encoder model based on applying the selected response as input to the neural network response encoder model.
- Various response encoder models may be utilized, such as an LSTM neural network model.
- the embedding may be generated as “end” output of the model, or may be generated based on part of the model's hidden state (e.g., based on the LSTM's state after application of the response).
- the system generates the embedding based on the selected response of block 302 and based on one or more features of the electronic communication for which the candidate responses were generated. This may effectively encode aspects of the electronic communication in the embedding and result in an embedding that includes semantics of the selected response in view of the electronic communication.
- the system identifies an additional candidate response of the candidate responses received at block 300 .
- the candidate responses received at block 300 may be ranked and the system may identify the next most highly ranked candidate response.
- the system generates an embedding based on the identified additional response of block 306 .
- the system may generate the embedding over the neural network response encoder model used at block 304 based on applying the identified additional response as input to the neural network response encoder model.
- the system generates the embedding based on the identified additional response of block 306 and based on one or more features of the electronic communication for which the candidate responses were generated. This may effectively encode aspects of the electronic communication in the embedding and result in an embedding that includes semantics of the identified additional response in view of the electronic communication.
- the system determines whether the identified additional response of block 306 satisfies one or more difference criterion relative to response(s) included in the selected set. At one or more initial iterations of block 310 , this involves determining whether the identified additional response satisfies the difference criterion relative to the selected candidate response of block 302 . After additional candidate response(s) have been included in the selected set at block 314 (described below), block 310 may involve determining whether the identified additional response satisfies the difference criterion relative to: the selected candidate response of block 302 ; and additional candidate response(s) included in the selected set 314 at one or more iterations of block 314 .
- Block 310 may include block 311 , where the system compares the embedding of the identified additional response of block 306 to the embedding(s) of the response(s) of the selected set. At one or more initial iterations of block 311 , this involves comparing the embedding of the identified additional response to the embedding of the selected candidate response of block 302 . After additional candidate response(s) have been included in the selected set at block 314 (described below), block 311 may involve comparing the embedding of the identified additional response to: the embedding of the selected candidate response of block 302 ; and additional embedding(s) of candidate response(s) included in the selected set 314 at one or more iterations of block 314 .
- the system calculates a cosine similarity between the two embeddings, calculates a Euclidian distance between the two embeddings, and/or calculates a dot product of the two embeddings.
- determining whether the difference criterion is satisfied may be based on the comparison of block 311 .
- cosine similarity measure(s) between the embedding of the identified additional candidate response and the response(s) of the selected set may each be compared to a difference criterion threshold. If each of the cosine similarity measures satisfies the difference criterion threshold, it may be determined that the identified additional candidate response satisfies the difference criterion. Otherwise, it may be determined that the identified additional candidate response does not satisfy the difference criterion (i.e., if any of the cosine similarity measures fails to satisfy the difference criterion threshold).
- the system determines whether the difference criterion was satisfied at a most recent iteration of block 310 . If not, the system proceeds back to block 306 , identifies yet another additional candidate response, and performs blocks 308 and 310 based on that identified additional candidate response.
- the system determines the difference criterion was satisfied, the system proceeds to block 314 .
- the identified additional candidate response is selected for inclusion in the selected set. The system then proceed to block 316 .
- the system determines whether a threshold number of responses have been selected for inclusion in the selected set.
- Various threshold numbers may be utilized and may optionally be based on characteristics of a client device to which the responses are to be provided as suggestions. For example, for a mobile phone client device the threshold number may be three, whereas for a desktop client device the threshold number may be five.
- Block 318 provides the selected responses for presentation (e.g., audibly and/or for display in a graphical interface) as suggestions for inclusion in a reply electronic communication that is a reply to the electronic communication for which the candidate responses of block 300 were generated.
- Block 318 may include providing the selected responses immediately for presentation on the client device, or may include storing the responses in association with the electronic communication for subsequently providing (e.g., providing in response to the user initially viewing or otherwise initially consuming the electronic communication).
- the system may proceed to block 318 in response to other conditions being satisfied, where those other conditions are in addition to or an alternative to the condition of block 316 . For example, the system may proceed to block 318 in response to determining that no candidate responses are left to process, in response to determining that a threshold number of candidate responses have been processed, and/or in response to a timeout condition.
- the system proceeds back to block 306 , identifies yet another additional candidate response, and performs blocks 308 and 310 based on that identified additional candidate response.
- the partial list of candidate responses displayed in the left column of FIG. 4 may be those received at block 300 and may be generated as candidate responses for an electronic communication of “I'm sick and can't make it”. Further, the list of FIG. 4 may be received with a ranking that is indicated by the order in which they appear in FIG. 4 (i.e., “Hope you feel better soon !” is the most highly ranked, followed by “Hope you feel better soon.”, and so forth).
- the system may select “Hope you feel better soon !” based on it being the most highly ranked. The selection is indicated in FIG. 4 by “Hope you feel better soon !” being bolded.
- the system generates an embedding based on “Hope you feel better soon !”.
- the system selects “Hope you feel better soon.” based on it being the next most highly ranked.
- the system generates an embedding based on “Hope you feel better soon.”
- the system generates a cosine similarity measure of “0.00” based on comparing the two embeddings.
- the cosine similarity measure of “0.00” indicates the two embeddings are semantically the same (e.g., the punctuation marks “.” and “?” may have been ignored in generating the embeddings).
- the system may further compare the measure of “0.00” to a difference criterion threshold (e.g., 3 . 50 ) and determine that the difference criterion threshold is not satisfied.
- the system at block 312 proceeds back to block 306 .
- the system selects “I hope you feel better soon.” based on it being the next most highly ranked.
- the system generates an embedding based on “I hope you feel better soon.”
- the system generates a cosine similarity measure of “1.24” based on comparing the two embeddings.
- the cosine similarity measure of “1.24” indicates the degree of semantic similarity between the two embeddings.
- the system may further compare the measure of “1.24” to a difference criterion threshold (e.g., 3.50) and determine that the difference criterion threshold is not satisfied. Based on the difference criterion not being satisfied, the system at block 312 proceeds back to block 306 .
- a difference criterion threshold e.g., 3.50
- the system continues with iterations of blocks 306 , 308 , 310 , and 312 until the candidate response of “Thanks for letting me know.” is considered. As indicated by the similarity score “5.65” illustrated for that candidate response in FIG. 4 (and the bolding of that candidate response in FIG. 4 ), it satisfies the threshold. Accordingly, at the iteration of block 312 where that candidate response is being considered, the system proceed to block 314 and adds “Thanks for letting me know.” to the selected set. The system then proceeds to block 316 . At block 316 , the threshold number may be three and, as a result, the system may proceed to another iteration of block 306 and select “Oh no ! Feel better !”.
- the system generates an embedding based on “Oh no ! Feel better !”.
- the system generates a cosine similarity measure of “3.16”.
- the generated cosine similarity measure of “3.16” illustrated in FIG. 4 is the minimum of two separate cosine similarity measures generated.
- a first cosine similarity measure may be generated based on comparing the embedding for “Oh no ! Feel better !” to the embedding for “Hope you feel better soon !”; and a second cosine similarity measure may be generated based on comparing the embedding for “Oh no !
- the system may further compare the measure of “3.16” to a difference criterion threshold (e.g., 3.50) and determine that the difference criterion threshold is not satisfied. Based on the difference criterion not being satisfied, the system at block 312 proceeds back to block 306 .
- a difference criterion threshold e.g., 3.50
- the system continues with further iterations of blocks 306 , 308 , 310 , and 312 until the candidate response of “What happened?” is considered.
- the similarity score “7.60” illustrated for that candidate response in FIG. 4 (and the bolding of that candidate response in FIG. 4 )
- it may be selected for inclusion in the set at a corresponding iteration of block 314 .
- the threshold number of block 316 is three
- the method of FIG. 3 may, at a corresponding iteration of block 316 , proceed to block 318 (and not process the subsequent candidate response of “Thank you for letting us know.” and other subsequent candidate responses indicated by the ellipsis of FIG. 4 ).
- the generation of one or more embeddings may be “pre-calculated” and those pre-calculated embeddings utilized at those iterations instead of calculating an embedding “live”.
- FIG. 5A and FIG. 5B each illustrate example graphical user interfaces 500 A and 500 B for providing a semantically diverse subset of candidate responses as suggestion for inclusion in a reply to a received electronic communication.
- the graphical user interfaces 500 A and 500 B may be presented at client device 106 (e.g., in a browser executing at client device 106 and/or in a dedicated electronic communication application executing at client device 106 ).
- an original email 580 A that has been sent to a user is being viewed by the user, before the user has provided any user interface input to indicate a desire to reply to the original email 580 A.
- Selected responses 591 A, 592 A, and 593 A are presented in the graphical user interface 500 A and are presented for inclusion in a reply to the email.
- a user selection of response 591 A may automatically present an editable reply email to the user that includes the text “I'd love to!” automatically incorporated in the reply email—or may automatically generate and send such a reply email (without first presenting an editable version to the user).
- the user selection of a response may be a single user interface input such as a single “tap”, a single “mouse click”, etc. It is noted that in FIG. 5A , the presented responses are all semantically diverse from one another and may be selected according to techniques described herein.
- an original chat message 580 B that has been sent to a user is being viewed by the user, before the user has provided any content for inclusion in a reply to the original chat message 580 B (e.g., possibly after the user has selected a “reply” interface element, but before the user has “typed” anything for the reply).
- Selected responses 591 B, 592 B, and 593 B are presented in the graphical user interface 500 B and are presented for inclusion in a reply to the original chat message 580 B.
- a user selection of response 593 B may automatically present an editable reply chat message to the user that includes the text “What happened?” automatically incorporated in the reply—or may automatically generate and send such a reply chat message (without first presenting an editable version to the user).
- the presented responses are all semantically diverse from one another and may be selected according to techniques described herein.
- responses may additionally or alternatively be provided for presentation to a user audibly via a client device.
- the client device 106 may audibly “read” a received electronic communication, then audibly provide “Speak A to respond with I'd love to, B to respond with what time, C to respond with sorry, I can't”.
- the client device 106 may then cause a corresponding reply to be generated and sent.
- the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and/or how to receive content from the content server that may be more relevant to the user.
- user information e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location
- certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed.
- a user's identity may be treated so that no personal identifiable information can be determined for the user, or a user's geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined.
- geographic location information such as to a city, ZIP code, or state level
- the user may have control over how information is collected about the user and/or used.
- FIG. 6 is a block diagram of an example computing device 610 .
- Computing device 610 typically includes at least one processor 614 which communicates with a number of peripheral devices via bus subsystem 612 .
- peripheral devices may include a storage subsystem 624 , including, for example, a memory subsystem 625 and a file storage subsystem 626 , user interface output devices 620 , user interface input devices 622 , and a network interface subsystem 616 .
- the input and output devices allow user interaction with computing device 610 .
- Network interface subsystem 616 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.
- User interface input devices 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices.
- pointing devices such as a mouse, trackball, touchpad, or graphics tablet
- audio input devices such as voice recognition systems, microphones, and/or other types of input devices.
- use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device 610 or onto a communication network.
- User interface output device 620 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices.
- the display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image.
- the display subsystem may also provide non-visual display such as via audio output devices.
- output device is intended to include all possible types of devices and ways to output information from computing device 610 to the user or to another machine or computing device.
- Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein.
- the storage subsystem 624 may include the logic to perform selected aspects of the method of FIG. 3 .
- Memory 625 used in the storage subsystem can include a number of memories including a main random access memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored.
- a file storage subsystem 626 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges.
- the modules implementing the functionality of certain implementations may be stored by file storage subsystem 626 in the storage subsystem 627 , or in other machines accessible by the processor(s) 617 .
- Bus subsystem 612 provides a mechanism for letting the various components and subsystems of computing device 610 communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.
- Computing device 610 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 610 depicted in FIG. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 610 are possible having more or fewer components than the computing device depicted in FIG. 6 .
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Resources & Organizations (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Economics (AREA)
- Computer Hardware Design (AREA)
- Information Transfer Between Computers (AREA)
- Machine Translation (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- Users are often inundated with electronic communications such as emails, SMS (short message service) communications, chat communications, social networking communications, etc. Many electronic communications that are sent to a user explicitly solicit a reply or otherwise contain information to which the user may wish to reply. For example, an email that contains “Are you interested in grabbing lunch?” may explicitly solicit a reply. An email that contains “I just got engaged!” may not explicitly solicit a reply, but a user that receives the email may still wish to send a congratulatory reply to the email. Users formulate replies to the electronic communications and input the formulated replies at client devices utilizing a keyboard (e.g., a virtual keyboard of a smartphone or tablet), a microphone, and/or other user interface input device(s) of the client device.
- In some situations it may be desirable to suggest one or more responses to a user for inclusion in a reply electronic communication that is a reply to an electronic communication transmitted to the user. For example, if a chat message of “Are you interested in grabbing lunch?” is transmitted to a client device of a user, it may be desirable to provide the user with responses that can be selected by the user, via the client device, (e.g., with a single-tap or other single-selection input) for inclusion in a reply chat message that is responsive to the received chat message. For instance, responses such as “Sure, what time?”, “Thanks, but I'm busy”, and “I'll get back to you” may be provided as suggestions based on their contextual relevance to the received chat message of “Are you interested in grabbing lunch?”. Selection of one of the provided responses by the user will reduce the number of user inputs that a user must make to formulate the reply, which may reduce the usage of various client device computational resources and/or be of particular benefit to users that have low dexterity (or who otherwise have difficulties making user inputs to a client device).
- However, some techniques for generating suggested responses may generate a large quantity of candidate responses that may potentially be provided as suggestions. Due to user interface constraints and/or other considerations, it may be undesirable and/or impossible to provide all such candidate responses for initial presentation to the user as suggestions. Accordingly, a subset of those candidate responses may be selected for initial presentation to the user as suggestions. Implementations disclosed herein seek to select such a subset in a manner that promotes semantic diversity among the responses of the subset.
- In view of these and other considerations, some implementations of this specification are directed generally to methods and apparatus related to determining a semantically diverse subset of candidate responses to provide for initial presentation to a user as suggestions for inclusion in a reply to an electronic communication. Some of those implementations determine the semantically diverse subset of candidate responses based on generating, over a response encoder model, embeddings that are each based on one of the plurality of the candidate responses. The embedding based on a given candidate response may be compared to embedding(s) of candidate response(s) already selected for the subset, and the given candidate response added to the subset only if the comparing indicates a difference criterion is satisfied. For example, the difference criterion may be satisfaction of a threshold that indicates a sufficient degree of semantic difference between the given candidate response and already selected candidate response(s). As used herein, a “response encoder model” is a model that can be used to process a sequence of tokens (e.g., included in a response) to generate an embedding of that sequence. A token of a candidate response may be a textual token (e.g., a word, a phrase) included in the candidate response or a graphical token (e.g., a sticker or an emoji, such as a thumbs-up emoji or a smiley emoji) included in the candidate response. For example a candidate response of “Hope you feel better soon” includes textual tokens of “Hope”, “you”, “feel”, “better”, and “soon”. Also, for example, a candidate response of “Can't wait ” includes textual tokens of “Can't” and “wait”, and a graphical token of “”. The embedding of a sequence of tokens of a candidate response is a dense representation (i.e., a real-valued vector) of the sequence of tokens. In various implementations, a response encoder model is a neural network model.
- Determining and providing a semantically diverse subset of responses for initial presentation to a user as suggestions may increase the likelihood that one of the provided responses is sufficient to convey the essence of an intended response of the user. As a result, a likelihood that a user will select one of the provided responses for inclusion in a reply may be increased. As described above, selecting one of the provided responses will reduce the number of user inputs that a user must make to formulate the reply, which may reduce the usage of various client device computational resources and/or be of particular benefit to users that have low dexterity (or who otherwise have difficulties making user inputs to a client device).
- In some implementations, a method implemented by one or more processors is provided and includes identifying a candidate set of responses for consideration for inclusion in a reply to a transmitted electronic communication directed to a user. Each of the responses of the candidate set include one or more tokens and is determined based on content of the electronic communication. The method further includes: selecting a first response, of the responses of the candidate set, for inclusion in a selected set; generating a first embedding over a neural network response encoder model based on applying the first response as input to the neural network response encoder model; and generating a second embedding over the neural network response encoder model based on applying a second response, of the responses of the candidate set, as input to the neural network response encoder model. The method further includes determining that the second response satisfies a difference criterion relative to the first response based on comparing the first embedding to the second embedding. The method further includes: based on determining that the second response satisfies the difference criterion, selecting the second response for inclusion in the selected set; and providing the responses of the selected set for display in an interface rendered by a client device of the user. The interface enables selection of any one of the responses of the selected set for inclusion in a reply electronic communication that is a reply by the user to the electronic communication.
- This method and other implementations of technology disclosed herein may each optionally include one or more of the following features.
- In some implementations, one or more of the processors are remote from the client device and providing the responses of the selected set for display in the interface includes transmitting content to the client device to cause the client device to visually render the responses of the selected set in the interface a selectable manner.
- In some implementations, the method further includes receiving a selection of a given response of the responses of the selected set, and incorporating the response into the reply communication in response to receiving the selection. The selection may be based on user interface input generated by a user interface input device of the client device
- In some implementations, the neural network response encoder model includes long short-term memory cells. In some of those implementations, generating the first embedding includes using a state of one or more of the long short-term memory cells of the neural network response encoder model as the embedding, where the state is the state after applying the first response to the neural network response encoder model. In some of those implementations: generating the first embedding is further based on applying at least part of the electronic communication to the neural network response encoder model prior to applying the first response; and generating the second embedding is further based on applying at least part of the electronic communication to the neural network response encoder model prior to applying the second response to the neural network response encoder model.
- In some implementations, determining that the second response satisfies the difference criterion relative to the first response based on comparing the first embedding to the second embedding includes: calculating a measure between the first embedding and the second embedding, and determining that the measure satisfies a threshold. In some of those implementations, the measure is a cosine similarity measure, a dot product, or a Euclidian distance.
- In some implementations, the candidate set of responses are ranked and selecting the first response is based on it being the most highly ranked. In some of those implementations, the method further includes: identifying an additional response, of the responses of the candidate set, that is the next most highly ranked response following the first response; generating an additional embedding over the neural network response encoder model based on applying the additional response as input to the neural network response encoder model; determining that the additional response fails to satisfy the difference criterion relative to the first response based on comparing the first embedding and the additional embedding; and omitting the additional response from the selected set based on the second response failing to satisfy the difference criterion.
- In some implementations, the method further includes, subsequent to selecting the second response for inclusion in the selected set: identifying an additional response of the responses of the candidate set; generating an additional embedding over the neural network response encoder model based on applying the additional response as input to the neural network response encoder model; determining that the additional response satisfies the difference criterion relative to at least one of: the first response and the second response; and selecting the additional response for inclusion in the selected set based on the second response satisfying the difference criterion relative to at least one of the first response and the second response. Determining that the additional response satisfies the difference criterion relative to at least one of the first response and the second response may be based on comparing the additional embedding to the first embedding and comparing the additional embedding to the second embedding. Determining that the additional response satisfies the difference criterion relative to at least one of the first response and the second response may include determining that the additional response satisfies the difference criterion relative to both the first response and the second response.
- In some implementations, providing the responses of the selected set occurs independent of any input provided via the client device in generating the reply electronic communication.
- In some implementations, a method implemented by one or more processors is provided and includes identifying a ranked candidate set of responses for consideration for inclusion in a reply to an electronic communication transmitted to a user. Each of the responses of the candidate set include one or more tokens and is determined based on content of the electronic communication. The method further includes, selecting a first response, of the responses of the candidate set, for inclusion in a selected set, based on it being the most highly ranked of the responses of the candidate set. The method further includes, for each of a plurality remaining responses of the responses of the candidate set, and until at least one condition has been satisfied: comparing an embedding vector of the remaining response to a first embedding vector of the first response and any embedding vectors of any additional responses selected for inclusion in the selected set, and determining whether to include the remaining response in the selected set based on the comparing. The method further includes providing the responses of the selected set for display in an interface rendered by a client device of the user. The interface enables selection of any one of the responses of the selected set for inclusion in a reply electronic communication that is a reply by the user to the electronic communication.
- This method and other implementations of technology disclosed herein may each optionally include one or more of the following features.
- In some implementations, the at least one condition includes that a threshold quantity of the responses have been selected for inclusion in the selected set.
- In some implementations, one or more of the processors are remote from the client device and providing the responses of the selected set for display in the interface includes transmitting content to the client device to cause the client device to visually render the responses of the selected set in the interface a selectable manner.
- In some implementations, the method further includes receiving a selection of a given response of the responses of the selected set, and incorporating the response into the reply communication in response to receiving the selection.
- Other implementations may include one or more non-transitory computer readable storage media storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a system including memory and one or more processors operable to execute instructions, stored in the memory, to perform a method such as one or more of the methods described above.
- It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.
-
FIG. 1 illustrates an example environment in which a set of semantically diverse responses may be determined and provided as suggestions for inclusion in a reply electronic communication. -
FIG. 2 illustrates an example of how candidate responses may be determined based on a received electronic communication, and a semantically diverse subset of those candidate responses selected for providing as suggestions for inclusion in a reply to the received electronic communication. -
FIG. 3 is a flow chart illustrating an example method according to implementations disclosed herein. -
FIG. 4 illustrates example responses and minimum similarity scores for those responses that may be determined according to implementations disclosed herein. -
FIG. 5A andFIG. 5B each illustrate example graphical user interfaces for providing a semantically diverse subset of candidate responses as suggestion for inclusion in a reply to a received electronic communication. -
FIG. 6 illustrates an example architecture of a computing device. -
FIG. 1 illustrates an example environment in which a set of semantically diverse responses may be determined and provided as suggestions for inclusion in a reply electronic communication. The example environment includes acommunication network 101 that facilitates communication between the various components in the environment. In some implementations, thecommunication network 101 may include a wide area network (WAN) such as the Internet, one or more intranets, and/or one or more bus subsystems. Thecommunication network 101 may optionally utilize one or more standard communications technologies, protocols, and/or inter-process communication techniques. - The example environment of
FIG. 1 also includes aclient device 106, anelectronic communications system 110, aresponse system 120, at least oneelectronic communications database 152, response prediction model(s) 154, and response encoder model(s) 156. -
Electronic communications system 110 andresponse system 120 may each be implemented in one or more computing devices that communicate, for example, through a network (e.g.,network 101 and/or other network).Electronic communications system 110 andresponse system 120 are example systems in which the systems, components, and techniques described herein may be implemented and/or with which systems, components, and techniques described herein may interface.Electronic communications system 110 andresponse system 120 each include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. In some implementations,electronic communications system 110 andresponse system 120 may include one or more components of the example computing device ofFIG. 6 . The operations performed byelectronic communications system 110 andresponse system 120 may be distributed across multiple computer systems. In some implementations, one or more aspects ofelectronic communications system 110 andresponse system 120 may be combined in a single system and/or one or more aspects may be implemented on theclient device 106. - The
electronic communications database 152 includes one or more storage mediums that include all, or portions of, electronic communications of a plurality of users. In some implementations, theelectronic communications database 152 is maintained by theelectronic communications system 110. For example, theelectronic communications system 110 may include one or more email systems and theelectronic communications database 152 may include a plurality of emails that are sent and/or received via the email systems. As another example, theelectronic communications system 110 may include one or more social networking systems and theelectronic communications database 152 may include a plurality of messages, posts, or other communications that are sent and/or received via the social networking systems. - As used herein, an “electronic communication” or “communication” may refer to an email, a text message (e.g., SMS, MMS), an instant message, a transcribed voicemail, or any other electronic communication that is sent from a first user to a restricted group of one or more additional users. In various implementations, an electronic communication may include various metadata and the metadata may optionally be utilized in one or more techniques described herein. For example, an electronic communication such as an email may include an electronic communication address such as one or more sender identifiers (e.g., sender email addresses), one or more recipient identifiers (e.g., recipient email addresses, including cc′d and bcc′d recipients), a date sent, one or more attachments, a subject, a type of device that sent and/or received the electronic communication, and so forth.
- As used herein, “electronic communication” and “communication” will be utilized, depending on the context, to refer to both an electronic communication that includes only a single “initial” message and an electronic communication that includes an initial message and one or more additional messages that are subsequent to the initial message. For example, an electronic communication may include an “initial” message from a first user and an additional message that is a reply message from a second user. Also, for example, an electronic communication may include an “initial” message from a first user and an additional message that is also from the first user. Accordingly, candidate responses described herein may be generated based on a received communication that includes just a single initial message, or a received communication that includes the initial message and one or more additional messages. An electronic communication may be a single document such as an email that includes both an initial message and one or more additional messages, and that can optionally be processed to distinguish the original message and the additional message(s). Processing an electronic communication to distinguish an initial message and an additional message may include “splitting” the message based on presence of metadata, message breaks, header information included in the message, quotes provided around the initial message, etc. An electronic communication may also be multiple documents that are mapped to one another based on at least one of the multiple documents being responsive to other of the multiple documents. For example, an electronic communication may include a first email that is an original message received by a user and a second email sent by the user as a reply to that original message, or original and reply SMS messages. The mapping of multiple documents to one another may be performed, for example, by the
electronic communications system 110. For example, theelectronic communications system 110 may map a first email to a second email based on a user selecting a “reply” user interface element when viewing the first email and then drafting the second email responsive to selecting the reply user interface element. - Generally,
response system 120 determines and provides suggested responses to include in a reply to an electronic communication, such as an electronic communication provided byclient device 106 and/or an electronic communication ofelectronic communications database 152 to which a user has yet to reply. As described herein, theresponse system 120 may determine a candidate set of responses based on one or more message features of the electronic communication, and may select a subset of the responses of the candidate set for actually providing as suggestions to include in a reply. In some implementations, theresponse system 120 may provide the subset of the responses as suggestions independent of any textual input provided by the user in generating the reply to the communication and/or independent any other content (e.g., voice input, graphical elements) provided by the user in generating the reply to the communication. - In various implementations,
response system 120 may include acandidate responses engine 122, aresponse selection engine 124, and apresentation engine 128. Theresponse selection engine 124 may include aresponse embedding module 125 and an embeddingcomparison module 126. In some implementations, aspects of one or more ofengines modules response system 120, such asclient device 106 and/orelectronic communications system 110. - Description of the components of
response system 120 is provided below with reference to an electronic communication of a user, such as an electronic communication provided byclient device 106 and/or an electronic communication ofelectronic communications database 152 to which the user has yet to reply. Although a single electronic communication is referenced in the examples for clarity, it is understood that theresponse system 120 may determine suggested responses for additional electronic communications of the user and/or of additional users. For example, theresponse system 120 may manage determination of suggested responses for each of multiple client devices via different sessions and may manage multiple sessions in parallel. For instance, theresponse system 120 in some implementations may be implemented as a cloud-based service employing a cloud infrastructure, e.g., using a server farm or cluster of high performance computers running software suitable for handling high volumes of requests from multiple users. However, for the sake of simplicity, many examples herein are described with respect to asingle client device 106 and a single electronic communication. - The
candidate responses engine 122 generates a candidate set of responses based on a transmitted electronic communication directed to at least one user (i.e., based at least in part on the content of the electronic communication as that content was sent to the user). In some implementations, thecandidate responses engine 122 generates the candidate set of responses based on application of features of the communication to one or moreresponse prediction models 154. - In some of those implementations, the features include tokens (e.g., textual token(s) and/or graphical token(s)) from the communication and the
candidate responses engine 122 applies the tokens to a long short-term memory (LSTM) neural network model of theresponse prediction models 154 to generate output indicative of candidate responses. The applied tokens from the communication may include all of the tokens, or one or more subsets of the tokens (e.g., first sentence and/or last sentence, body of the communication only, subject and body, all of the communication with stopwords omitted). Thecandidate responses engine 122 then uses the output to determine candidate responses. For example, thecandidate responses engine 122 may apply the tokens as input to the LSTM neural network model on a sequential token by token basis to generate, over the model, one or more outputs that each indicate a probability distribution over words. - The
candidate responses engine 122 utilizes the output(s) of the LSTM neural network model to determine a set of candidate responses and to assign scores to (or otherwise rank) the candidate responses. In some implementations, the candidate responses are identified from a whitelist of candidate responses, such as a curated list of common responses. In some implementations, one or more of the candidate responses may additionally or alternatively be generated in an unrestricted token by token manner (i.e., not identified from a whitelist of common responses). Regardless of the technique(s) utilized, thecandidate responses engine 122 may identify a number of candidate responses (e.g., 10, 20, 30, or more) and may optionally rank the identified candidate responses based on, for example, scores assigned to the candidate response based on the output generated over the LSTM neural network model. Thecandidate responses engine 122 may communicate the candidate responses and the ranking to theresponse selection engine 124. The ranking of the candidate responses may be conveyed as scores for the candidate responses, an ordering of the candidate responses (e.g., a list where the first candidate response is the most highly ranked, the second candidate response is the next most highly ranked), etc. In some implementations, thecandidate responses engine 122 may only identify up to X candidate responses and/or use thresholding to identify only candidate responses that have at least a threshold degree of conformance to the output generated over the LSTM neural network model. - In some implementations, the LSTM neural network model and/or other model of the one or more
response prediction models 154 may be trained based on a large corpus of communications that include original messages and reply messages. For example, the neural network model may be trained to maximize the probability of features of observed reply messages given features of their respective original messages. - In some implementations, the
candidate responses engine 122 may utilize additional and/or alternative techniques to generate a set of candidate responses. For example, thecandidate responses engine 122 may utilize a response prediction model that is not an LSTM neural network model, may utilize feature(s) of a received electronic communication that are in addition to text of the electronic communication, etc. For example, thecandidate response engine 122 may utilize features that are based on one or more natural language processing tags or other labels applied to text of the electronic communication (e.g., parts of speech, named entities, entity types, tone); features based on text that occurs specifically in the subjects, the first sentences, the last sentences, or other portion of the electronic communication; features based on metadata of the electronic communication such as a time the electronic communication was sent, day of the week the electronic communication was sent, a number of recipients, a type of device that sent the electronic communication, etc. - Regardless of technique(s) utilized, the
candidate responses engine 122 provides the set of candidate responses to theresponse selection engine 124. In some implementations, the provided set of candidate responses are ranked. For example, they may be ordered based on the ranking and/or each include a score or other value that indicates its ranking. In some of those implementations, theresponse selection engine 124 selects the most highly ranked candidate response for inclusion in a selected set to provide topresentation engine 128 for presentation as suggested responses. - The
response embedding module 125 utilizes one or moreresponse encoder models 156 to generate an embedding based on the most highly ranked candidate response and an additional embedding for each of a plurality of additional considered candidate responses of the provided set of candidate responses. In some implementations, theresponse embedding module 125 generates the embeddings “live” (e.g., in response to receiving the candidate responses from the candidate responses engine 122). In some implementations, theresponse embedding module 125 may additionally or alternatively pre-generate one or more of the embeddings (e.g., pre-generate embeddings for each of a plurality of common candidate responses). - An embedding based on a response is a vector of values that are determined based on tokens of the response and that encode at least some of the semantics of the response. In some implementations, the embedding may be of the response and may be a transformation of the response into a vector of values that represent the semantics of the response. In some implementations, the embedding may be based on the response and also take into account context of an original communication for which a response is a candidate.
- In some implementations, the
response embedding module 125 utilizes a trained neural network model of theresponse encoder models 156 to generate an embedding based on a response. For example, theresponse embedding module 125 may apply the response as input to the neural network model and generate the embedding over the model based on the input. For instance, the embedding may be the final output of the model, or may be derived from the hidden state of the model. - As one example, the
response embedding module 125 may generate an embedding of a response based on an LSTM neural network model. In some of those implementations, the LSTM neural network model may optionally be the same as that optionally utilized by thecandidate responses engine 122 and described above. For example, theresponse embedding module 125 may apply the response sequentially on a token by token basis as input to the LSTM neural network model to generate a vector representation of the response in a hidden layer of the LSTM neural network model, and may utilize the vector representation of the response as the embedding of the response. As another example, theresponse embedding module 125 may additionally or alternatively utilize other neural network models that generate an embedding of a token (in hidden layer(s) or as end output), such as a GloVE embedding model and/or a Word2Vec embedding model. - As mentioned above, in some implementations the
response embedding module 125 may optionally generate the embedding based on the response and also take into account context of a communication for which a response is a candidate. For example, theresponse embedding module 125 may initially apply tokens (e.g., all of the tokens or a subset) from the communication as input to the LSTM neural network model, then apply the response, and utilize the resulting vector representation in the hidden layer as the embedding of the response. This may effectively encode aspects of the communication in the embedding and result in an embedding that includes semantics of the response in view of the communication. For example, the embedding of “Got it!” can mean “I received it” or “understood” depending on the context of the original communication. By applying features of a given communication in conjunction with the response “Got it!”, an embedding can be generated that more accurately conveys the semantics of the response in the context of the original message. For instance, the embedding of “Got it!” when first applying the original communication text of “Did you get my message?”, may differ significantly from the embedding of “Got it!” when first applying the original communication text of “No, the meeting is tomorrow”. - The embedding
comparison module 126 compares the embedding of each considered candidate response to embedding(s) of candidate response(s) already selected for the subset, and adds the candidate response to the subset only if the comparing indicates a difference criterion is satisfied. For example, assume that only a highest ranked candidate response has so far been selected for inclusion in the subset. The embeddingcomparison module 126 may compare an embedding of a lower ranked candidate response to the embedding of the highest ranked candidate response and only select the lower ranked candidate response for inclusion in the subset if the comparison indicates at least a threshold degree of semantic difference between the two embeddings. For example, the embeddingcomparison module 126 may calculate a cosine similarity between the two embeddings, calculate a Euclidian distance between the two embeddings, and/or calculate a dot product of the two embeddings; and compare the cosine similarity, the Euclidian distance, and/or the dot product to a difference criterion threshold. If the difference criterion threshold is satisfied, the embeddingcomparison module 126 adds the lower ranked candidate response to the subset. Otherwise, the embeddingcomparison module 126 does not add the lower ranked candidate response to the subset. - The embedding
comparison module 126 may repeat this process until a threshold quantity of candidate response are selected for the subset, no candidate responses remain for consideration, and/or other condition is satisfied. Theresponse selection engine 122 then provides the selected subset of candidate responses to thepresentation engine 128. Additional description of some implementations of theresponse selection engine 124 is provided below with respect to the method ofFIG. 3 . - The
presentation engine 128 provides the selected subset of candidate responses for inclusion in a reply electronic communication that is a reply by the user to the electronic communication. For example, thepresentation engine 128 may present one or more user interface indications (e.g., graphical, audible) of the responses of the selected subset and, when a user generating the reply selects one of the indications, the corresponding response may be automatically incorporated in the reply. The reply may then be automatically sent and/or sent in response to further user interface input of the user (optionally after the user has added additional content to the reply). Such automatic incorporation of a selected response into a reply message saves the user from manually entering the selected response, and saves a number of user interaction steps and corresponding input required to establish a reply message that includes the response. - In some implementations, the
presentation engine 128 provides the responses independent of any input and/or other content provided via a computing device of the user in generating a reply to the electronic communication. In some of those implementations, the subset of the responses may be selected by theresponse selection engine 124 before the user has even viewed or otherwise consumed the communication. For example, thecandidate responses engine 122 and theresponse selection engine 124 may process a communication in advance of viewing by the user, andpresentation engine 128 may append or otherwise associate the selected responses with the electronic communication before it is even viewed by the user. Hence, if a user views or otherwise consumes the electronic communication, suggested responses to the electronic communication can be provided promptly without delay. - In some implementations where the electronic communication is provided to the
response system 120 by theclient device 106, thepresentation engine 128 may provide the selected subset of responses to theclient device 106 for presentation to the user as options for including in a reply. In some implementations where the electronic communication is provided to theresponse system 120 by theelectronic communications system 110 via theelectronic communications database 152, thepresentation engine 128 may store an association of the selected subset of responses with the electronic communication in theelectronic communications database 152 and/or other database. In some implementations, one or more (e.g., all) aspects of thepresentation engine 128 may be implemented by theclient device 106 and/or theelectronic communications system 110. - The
presentation engine 128 may also provide display prominence information with the selected subset of responses that indicates the prominence (e.g., position, size, color) with which those responses should be presented. Generally, a higher ranking for a particular response, the greater the prominence for that response. As one example, the optional ranking determined by theresponse embedding module 125 may be utilized to determine the ranking for the responses of the selected subset. - In some implementations, the
presentation engine 128 provides only the selected subset of responses (or associated identifiers), and potentially prominence information, and theclient device 106 and/orelectronic communications system 110 may generate a display of the selected documents based on the provided data. In some implementations, thepresentation engine 128 may additionally provide some or all of the data necessary to generate the display. In some of those implementations, any provided prominence information may be incorporated in the data that indicates how the display should be presented. -
FIG. 2 illustrates an example of how candidate responses may be determined based on a received electronic communication, and a semantically diverse subset of those candidate responses selected for providing as suggestions for inclusion in a reply to the received electronic communication. - In
FIG. 2 ,candidate responses engine 122 utilizes one or more response prediction model(s) 154 to determine a set of candidate responses 153.Electronic communication 152A may be provided by, for example,client device 106 orelectronic communications system 110 ofFIG. 1 . - The
candidate responses engine 122 provides the set of candidate responses 153 toresponse selection engine 124. Theresponse selection engine 124 utilizes one or moreresponse encoder models 156 to select a subset of the candidate responses 153 for presentation as suggestions in response to theelectronic communication 152A. As described herein, theresponse selection engine 124 may utilize the response encoder model(s) 156 to generate embeddings of the candidate responses of the set of candidate responses 153, and may use the embeddings to select a semantically diverse subset of the set of candidate responses 153. Theresponse selection engine 124 provides the selected responses 155 to thepresentation engine 128. - The
presentation engine 128 provides the selected response 155 for inclusion in a reply electronic communication that is a reply to theelectronic communication 152A. -
FIG. 3 is a flow chart illustrating an example method according to implementations disclosed herein. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems. For instance, some operations may be performed by theresponse selection engine 124. Moreover, while operations of the method ofFIG. 3 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added. - At
block 300, the system receives candidate responses for consideration in a reply to an electronic communication. For example, the system may receive the candidate response from a candidate responses engine that generates the candidate response based on features of the electronic communication, such that the candidate response are contextually relevant to the electronic communication. - At
block 302, the system selects one of the candidate responses for inclusion in a selected set. For example, the candidate responses received atblock 300 may be ranked and the system may select the most highly ranked candidate response for inclusion in the selected set. - At
block 304, the system generates an embedding based on the selected response ofblock 302. For example, the system may generate the embedding over a neural network response encoder model based on applying the selected response as input to the neural network response encoder model. Various response encoder models may be utilized, such as an LSTM neural network model. Further, the embedding may be generated as “end” output of the model, or may be generated based on part of the model's hidden state (e.g., based on the LSTM's state after application of the response). In some implementations, atblock 304 the system generates the embedding based on the selected response ofblock 302 and based on one or more features of the electronic communication for which the candidate responses were generated. This may effectively encode aspects of the electronic communication in the embedding and result in an embedding that includes semantics of the selected response in view of the electronic communication. - At
block 306, the system identifies an additional candidate response of the candidate responses received atblock 300. For example, the candidate responses received atblock 300 may be ranked and the system may identify the next most highly ranked candidate response. - At
block 308, the system generates an embedding based on the identified additional response ofblock 306. For example, the system may generate the embedding over the neural network response encoder model used atblock 304 based on applying the identified additional response as input to the neural network response encoder model. In some implementations, atblock 308 the system generates the embedding based on the identified additional response ofblock 306 and based on one or more features of the electronic communication for which the candidate responses were generated. This may effectively encode aspects of the electronic communication in the embedding and result in an embedding that includes semantics of the identified additional response in view of the electronic communication. - At
block 310, the system determines whether the identified additional response ofblock 306 satisfies one or more difference criterion relative to response(s) included in the selected set. At one or more initial iterations ofblock 310, this involves determining whether the identified additional response satisfies the difference criterion relative to the selected candidate response ofblock 302. After additional candidate response(s) have been included in the selected set at block 314 (described below), block 310 may involve determining whether the identified additional response satisfies the difference criterion relative to: the selected candidate response ofblock 302; and additional candidate response(s) included in the selected set 314 at one or more iterations of block 314. -
Block 310 may include block 311, where the system compares the embedding of the identified additional response ofblock 306 to the embedding(s) of the response(s) of the selected set. At one or more initial iterations ofblock 311, this involves comparing the embedding of the identified additional response to the embedding of the selected candidate response ofblock 302. After additional candidate response(s) have been included in the selected set at block 314 (described below), block 311 may involve comparing the embedding of the identified additional response to: the embedding of the selected candidate response ofblock 302; and additional embedding(s) of candidate response(s) included in the selected set 314 at one or more iterations of block 314. In some implementations, in comparing two embeddings, the system calculates a cosine similarity between the two embeddings, calculates a Euclidian distance between the two embeddings, and/or calculates a dot product of the two embeddings. - When block 310 includes
block 311, determining whether the difference criterion is satisfied may be based on the comparison ofblock 311. For example, cosine similarity measure(s) between the embedding of the identified additional candidate response and the response(s) of the selected set may each be compared to a difference criterion threshold. If each of the cosine similarity measures satisfies the difference criterion threshold, it may be determined that the identified additional candidate response satisfies the difference criterion. Otherwise, it may be determined that the identified additional candidate response does not satisfy the difference criterion (i.e., if any of the cosine similarity measures fails to satisfy the difference criterion threshold). - At
block 312, the system determines whether the difference criterion was satisfied at a most recent iteration ofblock 310. If not, the system proceeds back to block 306, identifies yet another additional candidate response, and performsblocks - If, at
block 312, the system determines the difference criterion was satisfied, the system proceeds to block 314. At block 314, the identified additional candidate response is selected for inclusion in the selected set. The system then proceed to block 316. - At
block 316, the system determines whether a threshold number of responses have been selected for inclusion in the selected set. Various threshold numbers may be utilized and may optionally be based on characteristics of a client device to which the responses are to be provided as suggestions. For example, for a mobile phone client device the threshold number may be three, whereas for a desktop client device the threshold number may be five. - If, at
block 316, the system determines the threshold number of responses have been selected, the system proceeds to block 318 and provides the selected responses for presentation (e.g., audibly and/or for display in a graphical interface) as suggestions for inclusion in a reply electronic communication that is a reply to the electronic communication for which the candidate responses ofblock 300 were generated.Block 318 may include providing the selected responses immediately for presentation on the client device, or may include storing the responses in association with the electronic communication for subsequently providing (e.g., providing in response to the user initially viewing or otherwise initially consuming the electronic communication). In some implementations, the system may proceed to block 318 in response to other conditions being satisfied, where those other conditions are in addition to or an alternative to the condition ofblock 316. For example, the system may proceed to block 318 in response to determining that no candidate responses are left to process, in response to determining that a threshold number of candidate responses have been processed, and/or in response to a timeout condition. - If, at
block 316, the system determines the threshold number of responses have not been selected, the system proceeds back to block 306, identifies yet another additional candidate response, and performsblocks - One non-limiting example of the method of
FIG. 3 is provided with reference toFIG. 4 . The partial list of candidate responses displayed in the left column ofFIG. 4 may be those received atblock 300 and may be generated as candidate responses for an electronic communication of “I'm sick and can't make it”. Further, the list ofFIG. 4 may be received with a ranking that is indicated by the order in which they appear inFIG. 4 (i.e., “Hope you feel better soon !” is the most highly ranked, followed by “Hope you feel better soon.”, and so forth). - At
block 302, the system may select “Hope you feel better soon !” based on it being the most highly ranked. The selection is indicated inFIG. 4 by “Hope you feel better soon !” being bolded. Atblock 304, the system generates an embedding based on “Hope you feel better soon !”. At an initial iteration ofblock 306, the system selects “Hope you feel better soon.” based on it being the next most highly ranked. At an initial iteration ofblock 308, the system generates an embedding based on “Hope you feel better soon.” Atblock 311 ofblock 310, the system generates a cosine similarity measure of “0.00” based on comparing the two embeddings. The cosine similarity measure of “0.00” indicates the two embeddings are semantically the same (e.g., the punctuation marks “.” and “?” may have been ignored in generating the embeddings). Atblock 310, the system may further compare the measure of “0.00” to a difference criterion threshold (e.g., 3.50) and determine that the difference criterion threshold is not satisfied. - Based on the difference criterion not being satisfied, the system at
block 312 proceeds back to block 306. At this next iteration ofblock 306, the system selects “I hope you feel better soon.” based on it being the next most highly ranked. At the next iteration ofblock 308, the system generates an embedding based on “I hope you feel better soon.” Atblock 311 ofblock 310, the system generates a cosine similarity measure of “1.24” based on comparing the two embeddings. The cosine similarity measure of “1.24” indicates the degree of semantic similarity between the two embeddings. Atblock 310, the system may further compare the measure of “1.24” to a difference criterion threshold (e.g., 3.50) and determine that the difference criterion threshold is not satisfied. Based on the difference criterion not being satisfied, the system atblock 312 proceeds back to block 306. - The system continues with iterations of
blocks FIG. 4 (and the bolding of that candidate response inFIG. 4 ), it satisfies the threshold. Accordingly, at the iteration ofblock 312 where that candidate response is being considered, the system proceed to block 314 and adds “Thanks for letting me know.” to the selected set. The system then proceeds to block 316. Atblock 316, the threshold number may be three and, as a result, the system may proceed to another iteration ofblock 306 and select “Oh no ! Feel better !”. based on it being the next most highly ranked. At this iteration ofblock 308, the system generates an embedding based on “Oh no ! Feel better !”. Atblock 311 ofblock 310, the system generates a cosine similarity measure of “3.16”. The generated cosine similarity measure of “3.16” illustrated inFIG. 4 is the minimum of two separate cosine similarity measures generated. In particular, a first cosine similarity measure may be generated based on comparing the embedding for “Oh no ! Feel better !” to the embedding for “Hope you feel better soon !”; and a second cosine similarity measure may be generated based on comparing the embedding for “Oh no ! Feel better !” to the embedding for “Thanks for letting me know.” (since it now also forms part of the selected set). Atblock 310, the system may further compare the measure of “3.16” to a difference criterion threshold (e.g., 3.50) and determine that the difference criterion threshold is not satisfied. Based on the difference criterion not being satisfied, the system atblock 312 proceeds back to block 306. - The system continues with further iterations of
blocks FIG. 4 (and the bolding of that candidate response inFIG. 4 ), it satisfies the threshold. Accordingly, it may be selected for inclusion in the set at a corresponding iteration of block 314. Further, where the threshold number ofblock 316 is three, the method ofFIG. 3 may, at a corresponding iteration ofblock 316, proceed to block 318 (and not process the subsequent candidate response of “Thank you for letting us know.” and other subsequent candidate responses indicated by the ellipsis ofFIG. 4 ). - It is noted that in some implementations, the generation of one or more embeddings (e.g., at
block 304 and/or one or more iterations of block 308) may be “pre-calculated” and those pre-calculated embeddings utilized at those iterations instead of calculating an embedding “live”. -
FIG. 5A andFIG. 5B each illustrate examplegraphical user interfaces graphical user interfaces client device 106 and/or in a dedicated electronic communication application executing at client device 106). - In
FIG. 5A , anoriginal email 580A that has been sent to a user is being viewed by the user, before the user has provided any user interface input to indicate a desire to reply to theoriginal email 580A.Selected responses graphical user interface 500A and are presented for inclusion in a reply to the email. For example, a user selection ofresponse 591A may automatically present an editable reply email to the user that includes the text “I'd love to!” automatically incorporated in the reply email—or may automatically generate and send such a reply email (without first presenting an editable version to the user). The user selection of a response may be a single user interface input such as a single “tap”, a single “mouse click”, etc. It is noted that inFIG. 5A , the presented responses are all semantically diverse from one another and may be selected according to techniques described herein. - In
FIG. 5B , anoriginal chat message 580B that has been sent to a user is being viewed by the user, before the user has provided any content for inclusion in a reply to theoriginal chat message 580B (e.g., possibly after the user has selected a “reply” interface element, but before the user has “typed” anything for the reply).Selected responses graphical user interface 500B and are presented for inclusion in a reply to theoriginal chat message 580B. For example, a user selection ofresponse 593B may automatically present an editable reply chat message to the user that includes the text “What happened?” automatically incorporated in the reply—or may automatically generate and send such a reply chat message (without first presenting an editable version to the user). It is noted that inFIG. 5B , the presented responses are all semantically diverse from one another and may be selected according to techniques described herein. - Although examples of graphical interfaces are presented in
FIGS. 5A and 5B , it is understood that responses may additionally or alternatively be provided for presentation to a user audibly via a client device. For example, theclient device 106 may audibly “read” a received electronic communication, then audibly provide “Speak A to respond with I'd love to, B to respond with what time, C to respond with sorry, I can't”. In response to spoken user interface input of A, B, or C during or after the audible providing, theclient device 106 may then cause a corresponding reply to be generated and sent. - In situations in which the systems described herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed. For example, a user's identity may be treated so that no personal identifiable information can be determined for the user, or a user's geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and/or used.
-
FIG. 6 is a block diagram of anexample computing device 610.Computing device 610 typically includes at least oneprocessor 614 which communicates with a number of peripheral devices viabus subsystem 612. These peripheral devices may include astorage subsystem 624, including, for example, amemory subsystem 625 and afile storage subsystem 626, userinterface output devices 620, userinterface input devices 622, and anetwork interface subsystem 616. The input and output devices allow user interaction withcomputing device 610.Network interface subsystem 616 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices. - User
interface input devices 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information intocomputing device 610 or onto a communication network. - User
interface output device 620 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information fromcomputing device 610 to the user or to another machine or computing device. -
Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, thestorage subsystem 624 may include the logic to perform selected aspects of the method ofFIG. 3 . - These software modules are generally executed by processor 617 alone or in combination with other processors.
Memory 625 used in the storage subsystem can include a number of memories including a main random access memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored. Afile storage subsystem 626 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored byfile storage subsystem 626 in the storage subsystem 627, or in other machines accessible by the processor(s) 617. -
Bus subsystem 612 provides a mechanism for letting the various components and subsystems ofcomputing device 610 communicate with each other as intended. Althoughbus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses. -
Computing device 610 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description ofcomputing device 610 depicted inFIG. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations ofcomputing device 610 are possible having more or fewer components than the computing device depicted inFIG. 6 . - While several implementations have been described and illustrated herein, a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein may be utilized, and each of such variations and/or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.
Claims (16)
Priority Applications (8)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/394,869 US11030515B2 (en) | 2016-12-30 | 2016-12-30 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
DE202017105851.4U DE202017105851U1 (en) | 2016-12-30 | 2017-09-26 | Determine semantically diverse responses to the delivery as suggestions for inclusion in electronic communications |
KR1020197021484A KR102355473B1 (en) | 2016-12-30 | 2017-09-27 | Determination of semantically diverse responses to provide as proposals for inclusion in electronic communications |
CN201780081784.7A CN110121707B (en) | 2016-12-30 | 2017-09-27 | Determining semantically distinct responses to provide suggestions for inclusion in electronic communications |
PCT/US2017/053658 WO2018125340A1 (en) | 2016-12-30 | 2017-09-27 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
JP2019535787A JP7044788B2 (en) | 2016-12-30 | 2017-09-27 | Determining the semantically diverse provision of responses as a proposal to include in electronic communications |
EP17783653.3A EP3563256A1 (en) | 2016-12-30 | 2017-09-27 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
GB1715741.3A GB2558351A (en) | 2016-12-30 | 2017-09-28 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/394,869 US11030515B2 (en) | 2016-12-30 | 2016-12-30 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180189628A1 true US20180189628A1 (en) | 2018-07-05 |
US11030515B2 US11030515B2 (en) | 2021-06-08 |
Family
ID=60081295
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/394,869 Active 2040-03-07 US11030515B2 (en) | 2016-12-30 | 2016-12-30 | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications |
Country Status (8)
Country | Link |
---|---|
US (1) | US11030515B2 (en) |
EP (1) | EP3563256A1 (en) |
JP (1) | JP7044788B2 (en) |
KR (1) | KR102355473B1 (en) |
CN (1) | CN110121707B (en) |
DE (1) | DE202017105851U1 (en) |
GB (1) | GB2558351A (en) |
WO (1) | WO2018125340A1 (en) |
Cited By (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180060308A1 (en) * | 2016-08-31 | 2018-03-01 | Beijing Xiaomi Mobile Software Co., Ltd. | Method and apparatus for message communication |
US20190034412A1 (en) * | 2017-07-31 | 2019-01-31 | Ebay Inc. | Emoji Understanding in Online Experiences |
CN109547843A (en) * | 2019-02-01 | 2019-03-29 | 腾讯音乐娱乐科技(深圳)有限公司 | The method and apparatus that audio-video is handled |
US10374982B2 (en) * | 2017-06-30 | 2019-08-06 | Microsoft Technology Licensing, Llc | Response retrieval using communication session vectors |
US20190348028A1 (en) * | 2018-05-11 | 2019-11-14 | Google Llc | Adaptive interface in a voice-activated network |
US10540694B2 (en) * | 2017-06-29 | 2020-01-21 | Tyler Peppel | Audience-based optimization of communication media |
US20200151773A1 (en) * | 2017-06-29 | 2020-05-14 | Tyler Peppel | Audience-based optimization of communication media |
US10680978B2 (en) * | 2017-10-23 | 2020-06-09 | Microsoft Technology Licensing, Llc | Generating recommended responses based on historical message data |
US10978056B1 (en) * | 2018-04-20 | 2021-04-13 | Facebook, Inc. | Grammaticality classification for natural language generation in assistant systems |
US20210224346A1 (en) | 2018-04-20 | 2021-07-22 | Facebook, Inc. | Engaging Users by Personalized Composing-Content Recommendation |
US11100288B1 (en) * | 2018-06-21 | 2021-08-24 | SoundHound Inc. | Factored neural networks for language modeling |
US20210286851A1 (en) * | 2020-03-11 | 2021-09-16 | Microsoft Technology Licensing, Llc | Guided query recommendations |
US11308450B2 (en) * | 2018-04-27 | 2022-04-19 | Microsoft Technology Licensing, Llc | Generating personalized smart responses |
US11307880B2 (en) | 2018-04-20 | 2022-04-19 | Meta Platforms, Inc. | Assisting users with personalized and contextual communication content |
US20220210107A1 (en) * | 2020-12-31 | 2022-06-30 | Snap Inc. | Messaging user interface element with reminders |
US20220321514A1 (en) * | 2021-03-31 | 2022-10-06 | T-Mobile Usa, Inc. | Image-Based Communication and Response Suggestions |
US11516158B1 (en) * | 2022-04-20 | 2022-11-29 | LeadIQ, Inc. | Neural network-facilitated linguistically complex message generation systems and methods |
US20230016941A1 (en) * | 2021-07-15 | 2023-01-19 | Beijing Zitiao Network Technology Co., Ltd. | Method and device for adding emoji, apparatus and storage medium |
US20230027615A1 (en) * | 2021-07-26 | 2023-01-26 | Microsoft Technology Licensing, Llc | Modeling techniques to classify data sets containing personal identifiable information comprising numerical identifiers |
US11574190B2 (en) * | 2019-10-15 | 2023-02-07 | Samsung Electronics Co., Ltd. | Method and apparatus for determining output token |
WO2023019245A1 (en) * | 2021-08-12 | 2023-02-16 | Yohana Llc | Systems and methods for proposal communication in a task determination system |
US11676220B2 (en) | 2018-04-20 | 2023-06-13 | Meta Platforms, Inc. | Processing multimodal user input for assistant systems |
US11715042B1 (en) | 2018-04-20 | 2023-08-01 | Meta Platforms Technologies, Llc | Interpretability of deep reinforcement learning models in assistant systems |
US11729120B2 (en) * | 2017-03-16 | 2023-08-15 | Microsoft Technology Licensing, Llc | Generating responses in automated chatting |
US11886473B2 (en) | 2018-04-20 | 2024-01-30 | Meta Platforms, Inc. | Intent identification for agent matching by assistant systems |
US11947902B1 (en) | 2023-03-03 | 2024-04-02 | Microsoft Technology Licensing, Llc | Efficient multi-turn generative AI model suggested message generation |
US11962546B1 (en) | 2023-03-03 | 2024-04-16 | Microsoft Technology Licensing, Llc | Leveraging inferred context to improve suggested messages |
US20240296275A1 (en) * | 2023-03-03 | 2024-09-05 | Microsoft Technology Licensing, Llc | Guardrails for efficient processing and error prevention in generating suggested messages |
US12125272B2 (en) | 2023-08-14 | 2024-10-22 | Meta Platforms Technologies, Llc | Personalized gesture recognition for user interaction with assistant systems |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR3096488A1 (en) * | 2019-06-25 | 2020-11-27 | Orange | Process for processing a message and corresponding device. |
US11726656B2 (en) | 2021-02-04 | 2023-08-15 | Keys, Inc. | Intelligent keyboard |
US20230376690A1 (en) * | 2022-05-19 | 2023-11-23 | Apple Inc. | Variable length phrase predictions |
JP7549312B2 (en) | 2022-12-21 | 2024-09-11 | グリー株式会社 | Information processing system, information processing method, and program |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6278996B1 (en) | 1997-03-31 | 2001-08-21 | Brightware, Inc. | System and method for message process and response |
US7496500B2 (en) | 2004-03-01 | 2009-02-24 | Microsoft Corporation | Systems and methods that determine intent of data and respond to the data based on the intent |
US8725513B2 (en) * | 2007-04-12 | 2014-05-13 | Nuance Communications, Inc. | Providing expressive user interaction with a multimodal application |
US8499227B2 (en) * | 2010-09-23 | 2013-07-30 | Micron Technology, Inc. | Memory quality monitor based compensation method and apparatus |
US8589407B2 (en) * | 2011-06-17 | 2013-11-19 | Google Inc. | Automated generation of suggestions for personalized reactions in a social network |
US9117194B2 (en) | 2011-12-06 | 2015-08-25 | Nuance Communications, Inc. | Method and apparatus for operating a frequently asked questions (FAQ)-based system |
KR20140011073A (en) * | 2012-07-17 | 2014-01-28 | 삼성전자주식회사 | Method and apparatus for recommending text |
KR101574796B1 (en) | 2012-07-31 | 2015-12-15 | 진삼순 | Dialog system with Virtual Friend and application method thereof |
JP5763110B2 (en) | 2013-01-08 | 2015-08-12 | ビッグローブ株式会社 | Transmission / reception system, computer, transmission / reception apparatus, transmission / reception method, and program |
US9559651B2 (en) * | 2013-03-29 | 2017-01-31 | Apple Inc. | Metadata for loudness and dynamic range control |
US9213941B2 (en) * | 2014-04-22 | 2015-12-15 | Google Inc. | Automatic actions based on contextual replies |
US20150363696A1 (en) | 2014-06-13 | 2015-12-17 | reMesh | Method and system for generating a collective group response |
KR102232929B1 (en) | 2014-07-31 | 2021-03-29 | 삼성전자주식회사 | Message Service Providing Device and Method Providing Content thereof |
KR101583181B1 (en) * | 2015-01-19 | 2016-01-06 | 주식회사 엔씨소프트 | Method and computer program of recommending responsive sticker |
US10079785B2 (en) | 2015-02-12 | 2018-09-18 | Google Llc | Determining reply content for a reply to an electronic communication |
WO2016167424A1 (en) | 2015-04-16 | 2016-10-20 | 주식회사 플런티코리아 | Answer recommendation device, and automatic sentence completion system and method |
US10965622B2 (en) | 2015-04-16 | 2021-03-30 | Samsung Electronics Co., Ltd. | Method and apparatus for recommending reply message |
US10091140B2 (en) * | 2015-05-31 | 2018-10-02 | Microsoft Technology Licensing, Llc | Context-sensitive generation of conversational responses |
US11550751B2 (en) * | 2016-11-18 | 2023-01-10 | Microsoft Technology Licensing, Llc | Sequence expander for data entry/information retrieval |
US10185763B2 (en) * | 2016-11-30 | 2019-01-22 | Facebook, Inc. | Syntactic models for parsing search queries on online social networks |
-
2016
- 2016-12-30 US US15/394,869 patent/US11030515B2/en active Active
-
2017
- 2017-09-26 DE DE202017105851.4U patent/DE202017105851U1/en active Active
- 2017-09-27 KR KR1020197021484A patent/KR102355473B1/en active IP Right Grant
- 2017-09-27 CN CN201780081784.7A patent/CN110121707B/en active Active
- 2017-09-27 EP EP17783653.3A patent/EP3563256A1/en active Pending
- 2017-09-27 JP JP2019535787A patent/JP7044788B2/en active Active
- 2017-09-27 WO PCT/US2017/053658 patent/WO2018125340A1/en unknown
- 2017-09-28 GB GB1715741.3A patent/GB2558351A/en not_active Withdrawn
Cited By (66)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180060308A1 (en) * | 2016-08-31 | 2018-03-01 | Beijing Xiaomi Mobile Software Co., Ltd. | Method and apparatus for message communication |
US11729120B2 (en) * | 2017-03-16 | 2023-08-15 | Microsoft Technology Licensing, Llc | Generating responses in automated chatting |
US10540694B2 (en) * | 2017-06-29 | 2020-01-21 | Tyler Peppel | Audience-based optimization of communication media |
US11631110B2 (en) * | 2017-06-29 | 2023-04-18 | Tyler Peppel | Audience-based optimization of communication media |
US20200151773A1 (en) * | 2017-06-29 | 2020-05-14 | Tyler Peppel | Audience-based optimization of communication media |
US10708201B2 (en) * | 2017-06-30 | 2020-07-07 | Microsoft Technology Licensing, Llc | Response retrieval using communication session vectors |
US10374982B2 (en) * | 2017-06-30 | 2019-08-06 | Microsoft Technology Licensing, Llc | Response retrieval using communication session vectors |
US10650095B2 (en) * | 2017-07-31 | 2020-05-12 | Ebay Inc. | Emoji understanding in online experiences |
US11928428B2 (en) | 2017-07-31 | 2024-03-12 | Ebay Inc. | Emoji understanding in online experiences |
US20240169151A1 (en) * | 2017-07-31 | 2024-05-23 | Ebay Inc. | Emoji Understanding in Online Experiences |
US20190034412A1 (en) * | 2017-07-31 | 2019-01-31 | Ebay Inc. | Emoji Understanding in Online Experiences |
US11636265B2 (en) | 2017-07-31 | 2023-04-25 | Ebay Inc. | Emoji understanding in online experiences |
US10680978B2 (en) * | 2017-10-23 | 2020-06-09 | Microsoft Technology Licensing, Llc | Generating recommended responses based on historical message data |
US11727677B2 (en) | 2018-04-20 | 2023-08-15 | Meta Platforms Technologies, Llc | Personalized gesture recognition for user interaction with assistant systems |
US11908181B2 (en) | 2018-04-20 | 2024-02-20 | Meta Platforms, Inc. | Generating multi-perspective responses by assistant systems |
US12112530B2 (en) | 2018-04-20 | 2024-10-08 | Meta Platforms, Inc. | Execution engine for compositional entity resolution for assistant systems |
US11231946B2 (en) | 2018-04-20 | 2022-01-25 | Facebook Technologies, Llc | Personalized gesture recognition for user interaction with assistant systems |
US11245646B1 (en) | 2018-04-20 | 2022-02-08 | Facebook, Inc. | Predictive injection of conversation fillers for assistant systems |
US11249773B2 (en) | 2018-04-20 | 2022-02-15 | Facebook Technologies, Llc. | Auto-completion for gesture-input in assistant systems |
US11249774B2 (en) | 2018-04-20 | 2022-02-15 | Facebook, Inc. | Realtime bandwidth-based communication for assistant systems |
US12001862B1 (en) | 2018-04-20 | 2024-06-04 | Meta Platforms, Inc. | Disambiguating user input with memorization for improved user assistance |
US11301521B1 (en) | 2018-04-20 | 2022-04-12 | Meta Platforms, Inc. | Suggestions for fallback social contacts for assistant systems |
US11908179B2 (en) | 2018-04-20 | 2024-02-20 | Meta Platforms, Inc. | Suggestions for fallback social contacts for assistant systems |
US11307880B2 (en) | 2018-04-20 | 2022-04-19 | Meta Platforms, Inc. | Assisting users with personalized and contextual communication content |
US11308169B1 (en) | 2018-04-20 | 2022-04-19 | Meta Platforms, Inc. | Generating multi-perspective responses by assistant systems |
US11368420B1 (en) | 2018-04-20 | 2022-06-21 | Facebook Technologies, Llc. | Dialog state tracking for assistant systems |
US11887359B2 (en) | 2018-04-20 | 2024-01-30 | Meta Platforms, Inc. | Content suggestions for content digests for assistant systems |
US11886473B2 (en) | 2018-04-20 | 2024-01-30 | Meta Platforms, Inc. | Intent identification for agent matching by assistant systems |
US11429649B2 (en) | 2018-04-20 | 2022-08-30 | Meta Platforms, Inc. | Assisting users with efficient information sharing among social connections |
US10978056B1 (en) * | 2018-04-20 | 2021-04-13 | Facebook, Inc. | Grammaticality classification for natural language generation in assistant systems |
US20210224346A1 (en) | 2018-04-20 | 2021-07-22 | Facebook, Inc. | Engaging Users by Personalized Composing-Content Recommendation |
US11721093B2 (en) | 2018-04-20 | 2023-08-08 | Meta Platforms, Inc. | Content summarization for assistant systems |
US11544305B2 (en) | 2018-04-20 | 2023-01-03 | Meta Platforms, Inc. | Intent identification for agent matching by assistant systems |
US11715289B2 (en) | 2018-04-20 | 2023-08-01 | Meta Platforms, Inc. | Generating multi-perspective responses by assistant systems |
US11715042B1 (en) | 2018-04-20 | 2023-08-01 | Meta Platforms Technologies, Llc | Interpretability of deep reinforcement learning models in assistant systems |
US11704899B2 (en) | 2018-04-20 | 2023-07-18 | Meta Platforms, Inc. | Resolving entities from multiple data sources for assistant systems |
US11704900B2 (en) | 2018-04-20 | 2023-07-18 | Meta Platforms, Inc. | Predictive injection of conversation fillers for assistant systems |
US11688159B2 (en) | 2018-04-20 | 2023-06-27 | Meta Platforms, Inc. | Engaging users by personalized composing-content recommendation |
US20230186618A1 (en) | 2018-04-20 | 2023-06-15 | Meta Platforms, Inc. | Generating Multi-Perspective Responses by Assistant Systems |
US11676220B2 (en) | 2018-04-20 | 2023-06-13 | Meta Platforms, Inc. | Processing multimodal user input for assistant systems |
US11308450B2 (en) * | 2018-04-27 | 2022-04-19 | Microsoft Technology Licensing, Llc | Generating personalized smart responses |
US11848009B2 (en) * | 2018-05-11 | 2023-12-19 | Google Llc | Adaptive interface in a voice-activated network |
US20210366469A1 (en) * | 2018-05-11 | 2021-11-25 | Google Llc | Adaptive interface in a voice-activated network |
US11282510B2 (en) * | 2018-05-11 | 2022-03-22 | Google Llc | Adaptive interface in a voice-activated network |
US20190348028A1 (en) * | 2018-05-11 | 2019-11-14 | Google Llc | Adaptive interface in a voice-activated network |
US11908462B2 (en) * | 2018-05-11 | 2024-02-20 | Google Llc | Adaptive interface in a voice-activated network |
US11087748B2 (en) * | 2018-05-11 | 2021-08-10 | Google Llc | Adaptive interface in a voice-activated network |
US20220208183A1 (en) * | 2018-05-11 | 2022-06-30 | Google Llc | Adaptive Interface in a Voice-Activated Network |
US11100288B1 (en) * | 2018-06-21 | 2021-08-24 | SoundHound Inc. | Factored neural networks for language modeling |
CN109547843A (en) * | 2019-02-01 | 2019-03-29 | 腾讯音乐娱乐科技(深圳)有限公司 | The method and apparatus that audio-video is handled |
US11574190B2 (en) * | 2019-10-15 | 2023-02-07 | Samsung Electronics Co., Ltd. | Method and apparatus for determining output token |
US20210286851A1 (en) * | 2020-03-11 | 2021-09-16 | Microsoft Technology Licensing, Llc | Guided query recommendations |
US11924153B2 (en) * | 2020-12-31 | 2024-03-05 | Snap Inc. | Messaging user interface element with reminders |
US20220210107A1 (en) * | 2020-12-31 | 2022-06-30 | Snap Inc. | Messaging user interface element with reminders |
US20220321514A1 (en) * | 2021-03-31 | 2022-10-06 | T-Mobile Usa, Inc. | Image-Based Communication and Response Suggestions |
US11516163B2 (en) * | 2021-03-31 | 2022-11-29 | T-Mobile Usa, Inc. | Image-based communication and response suggestions |
US20230016941A1 (en) * | 2021-07-15 | 2023-01-19 | Beijing Zitiao Network Technology Co., Ltd. | Method and device for adding emoji, apparatus and storage medium |
US12056329B2 (en) * | 2021-07-15 | 2024-08-06 | Beijing Zitiao Network Technology Co., Ltd. | Method and device for adding emoji, apparatus and storage medium |
US20230027615A1 (en) * | 2021-07-26 | 2023-01-26 | Microsoft Technology Licensing, Llc | Modeling techniques to classify data sets containing personal identifiable information comprising numerical identifiers |
US11816246B2 (en) * | 2021-07-26 | 2023-11-14 | Microsoft Technology Licensing, Llc | Modeling techniques to classify data sets containing personal identifiable information comprising numerical identifiers |
WO2023019245A1 (en) * | 2021-08-12 | 2023-02-16 | Yohana Llc | Systems and methods for proposal communication in a task determination system |
US11516158B1 (en) * | 2022-04-20 | 2022-11-29 | LeadIQ, Inc. | Neural network-facilitated linguistically complex message generation systems and methods |
US11962546B1 (en) | 2023-03-03 | 2024-04-16 | Microsoft Technology Licensing, Llc | Leveraging inferred context to improve suggested messages |
US20240296275A1 (en) * | 2023-03-03 | 2024-09-05 | Microsoft Technology Licensing, Llc | Guardrails for efficient processing and error prevention in generating suggested messages |
US11947902B1 (en) | 2023-03-03 | 2024-04-02 | Microsoft Technology Licensing, Llc | Efficient multi-turn generative AI model suggested message generation |
US12125272B2 (en) | 2023-08-14 | 2024-10-22 | Meta Platforms Technologies, Llc | Personalized gesture recognition for user interaction with assistant systems |
Also Published As
Publication number | Publication date |
---|---|
EP3563256A1 (en) | 2019-11-06 |
DE202017105851U1 (en) | 2018-03-06 |
CN110121707A (en) | 2019-08-13 |
KR20190094245A (en) | 2019-08-12 |
WO2018125340A1 (en) | 2018-07-05 |
US11030515B2 (en) | 2021-06-08 |
JP7044788B2 (en) | 2022-03-30 |
KR102355473B1 (en) | 2022-01-24 |
CN110121707B (en) | 2023-10-31 |
GB2558351A (en) | 2018-07-11 |
JP2020504880A (en) | 2020-02-13 |
GB201715741D0 (en) | 2017-11-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11030515B2 (en) | Determining semantically diverse responses for providing as suggestions for inclusion in electronic communications | |
US10534860B2 (en) | Identifying tasks in messages | |
US11727220B2 (en) | Transitioning between prior dialog contexts with automated assistants | |
CN109983430B (en) | Determining graphical elements included in an electronic communication | |
US10917371B2 (en) | Methods and apparatus for determining non-textual reply content for inclusion in a reply to an electronic communication | |
CN110383297B (en) | Cooperatively training and/or using separate input neural network models and response neural network models for determining responses to electronic communications | |
US11238242B2 (en) | Generating output for presentation in response to user interface input, where the input and/or the output include chatspeak | |
US20240103893A1 (en) | Generating content endorsements using machine learning nominator(s) | |
US10733529B1 (en) | Methods and apparatus for determining original electronic messages that contain asks |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GOOGLE INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KAUFMANN, TOBIAS;KANNAN, ANJULI;SIGNING DATES FROM 20161228 TO 20161229;REEL/FRAME:040809/0061 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: GOOGLE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044567/0001 Effective date: 20170929 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PRE-INTERVIEW COMMUNICATION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |