US10783178B2 - Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user - Google Patents

Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user Download PDF

Info

Publication number
US10783178B2
US10783178B2 US15/157,297 US201615157297A US10783178B2 US 10783178 B2 US10783178 B2 US 10783178B2 US 201615157297 A US201615157297 A US 201615157297A US 10783178 B2 US10783178 B2 US 10783178B2
Authority
US
United States
Prior art keywords
natural language
entry
user
input
language input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US15/157,297
Other languages
English (en)
Other versions
US20170337265A1 (en
Inventor
Maryam Garrett
Wan Fen Nicole Quah
Bryan Horling
Ruijie He
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US15/157,297 priority Critical patent/US10783178B2/en
Application filed by Google LLC filed Critical Google LLC
Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HE, Ruijie, QUAH, WAN, GARRETT, MARYAM, HORLING, BRYAN
Priority to EP16826623.7A priority patent/EP3458975A1/en
Priority to CN201680085778.4A priority patent/CN109564578B/zh
Priority to PCT/US2016/068737 priority patent/WO2017200588A1/en
Priority to KR1020187036265A priority patent/KR102196976B1/ko
Priority to JP2018560513A priority patent/JP6978438B2/ja
Priority to KR1020207036980A priority patent/KR102313475B1/ko
Priority to CN202311109978.7A priority patent/CN117235335A/zh
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Publication of US20170337265A1 publication Critical patent/US20170337265A1/en
Priority to US17/026,572 priority patent/US11494427B2/en
Publication of US10783178B2 publication Critical patent/US10783178B2/en
Application granted granted Critical
Priority to JP2021017300A priority patent/JP7222005B2/ja
Priority to US17/964,343 priority patent/US11907276B2/en
Priority to US18/413,846 priority patent/US20240152537A1/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/338—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/103—Formatting, i.e. changing of presentation of documents
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G06F40/169—Annotation, e.g. comment data or footnotes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/029—Location-based management or tracking services

Definitions

  • Some note keeping computer applications and/or other computer applications enable a user to create note entries that include content that is explicitly set by the user. For example, the user may create a note of “call Bob.” Further, some computer applications may enable limited searching of the user created notes.
  • the note keeping computer applications may only enable searching for note entries based on exact keyword matching between terms of a search and terms of the notes. For instance, “call Bob” would only be identified if a search included “call” and/or “Bob”.
  • some note keeping computer applications may not automatically filter and/or rank note entries that are responsive to a search and/or may only do so based on limited user provided content.
  • some note keeping computer applications may require that a user explicitly designate that content provided by the user is to be utilized to create a note entry and therefore do not create any entries for many types of input of the user, such as inputs provided during dialog with a personal assistant and/or during dialog with one or more additional users. Additional and/or alternative drawbacks may be presented.
  • Some implementations of this specification are directed to generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user, such as natural language input provided to an automated personal assistant and/or provided to one or more computing devices of one or more additional users during communication with those additional users.
  • the entry includes one or more terms of the natural language input and optionally includes descriptive metadata that is determined based on one or more terms of the natural language input and/or based on contextual features associated with receiving the natural language input.
  • free-form input is input that is formulated by a user and that is not constrained to a group of options presented for selection by the user.
  • it may be typed input provided by the user via a physical or virtual keyboard of a computing device, spoken input provided by the user to a microphone of the computing device, and/or an image captured by the user via a camera (e.g., a camera of the computing device of the user).
  • a camera e.g., a camera of the computing device of the user
  • Some implementations of this specification are additionally and/or alternatively directed to generating, based on one or more personal database entries of a user, output that is responsive to further free-form natural language input of the user.
  • one or more entries that are responsive to further natural language input of the user can be identified based on matching (soft and/or exact) content of those entries (e.g., descriptive metadata and/or terms of those entries) to one or more search parameters determined based on the further natural language input.
  • content of an identified entry may be utilized in generating the output and/or in ranking multiple identified entries (where multiple entries are identified) relative to one another.
  • the ranking can be utilized to select a subset of entries (e.g., one entry) to utilize in generating the response and/or to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content were derived).
  • Some implementations of this specification are additionally and/or alternatively directed to analyzing given natural language input to determine a measure indicative of a likelihood that the input is intended as input the user desires an automated personal assistant to recall at a later time and/or to determine a measure indicative of a likelihood that the input is intended as a request for personal information from a personal database of the user.
  • Natural language input of the user may be received and processed in various scenarios.
  • natural language input may be input provided by the user during communication with one or more other users, such as communication via chat, SMS, and/or via other message exchange.
  • the natural language input may be provided to an automated personal assistant that engages in dialog with the user via one or more user interface input and output devices.
  • the automated personal assistant may be integrated, in whole or in part, in a computing device of the user (e.g., a mobile phone, a tablet, a device devoted solely to the automated assistant functionality) and may include one or more user interface input devices (e.g., a microphone, a touch-screen) and one or more user interface output devices (e.g., a speaker, a display screen).
  • the automated personal assistant may be implemented, in whole or in part, in one or more computing devices that are separate from, but in communication with, a client computing device of the user.
  • a method performed by one or more processors includes receiving first natural language input that is free-form input formulated by a user via a user interface input device of a computing device of the user.
  • the method further includes generating an entry for the first natural language input in a personal database of the user stored in one or more computer readable media and the generating includes: storing, in the entry, one or more given terms of the terms of the first natural language input or an identifier of the given terms; generating descriptive metadata based on at least one of: one or more of the terms of the first natural language input, and one or more contextual features associated with receiving the first natural language input; and storing the descriptive metadata in the entry.
  • the method further includes receiving second natural language input subsequent to receiving the first natural language input.
  • the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user.
  • the method further includes: determining, based on the second natural language input, at least one search parameter; searching the personal database based on the search parameter; and determining, based on the searching, that the entry is responsive to the second natural language input based at least in part on matching the search parameter to at least some of the descriptive metadata.
  • the method further include generating natural language output that includes one or more natural language output terms that are based on the entry and providing, in response to the second natural language input, the natural language output for presentation to the user via a user interface output device of the computing device or of the additional computing device.
  • generating the descriptive metadata is based on both the given terms of the first natural language input and the one or more contextual features associated with receiving the first natural language input.
  • generating the descriptive metadata is based on one or more contextual features associated with receiving the first natural language input and generating the descriptive metadata based on the one or more contextual features includes: generating temporal metadata that indicates a date or a time that the first natural language input was received.
  • the at least one search parameter is a temporal search parameter and determining that the entry is responsive to the second natural language input includes matching the temporal search parameter to the temporal metadata.
  • the method further includes determining that an additional entry is responsive to the second natural language input and selecting the entry in lieu of the additional entry based on conformance of a current date or time to the temporal metadata of the entry.
  • providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry, and the additional entry is not utilized in generating the natural language output and no output that is based on the additional entry is provided in response to the second natural language input.
  • generating the natural language output includes generating one or more temporal terms of the natural language output based on the temporal metadata of the entry.
  • generating the descriptive metadata is based on one or more contextual features associated with receiving the first natural language input and generating the descriptive metadata includes: generating locational metadata that indicates a location of the user when the first natural language input was received.
  • the at least one search parameter is a locational search parameter and determining that the entry is responsive to the second natural language input includes matching the locational search parameter to the locational metadata.
  • the method further includes: determining that an additional entry is responsive to the second natural language input; and selecting the entry in lieu of the additional entry based on conformance of a current location of the user to the locational metadata of the entry.
  • providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry, and the additional entry is not utilized in generating the natural language output and no output that is based on the additional entry is provided in response to the second natural language input.
  • generating the natural language output includes generating one or more locational terms of the natural language output based on the locational metadata of the entry.
  • the first natural language input is provided during communication between the user and at least one additional user
  • generating the descriptive metadata is based on one or more contextual features associated with receiving the first natural language input
  • generating the descriptive metadata includes generating additional user metadata of the descriptive metadata.
  • the additional user metadata identifies the additional user in communication with the user when the first natural language input was received.
  • the communication between the user and the additional user is via a first message exchange client of the computing device of the user and a second message exchange client of an additional user computing device of the additional user.
  • generating the descriptive metadata is based on one or of the terms of the first natural language input and generating the descriptive metadata based on one or more of the terms of the first natural language input includes generating a semantic label of the descriptive metadata based on one or more of the terms.
  • the semantic label indicates a classification to which the one or more terms belong and to which additional terms not included in the first natural language output also belong.
  • the at least one search parameter is a semantic search parameter and determining that the entry is responsive to the second natural language input includes matching the semantic search parameter to the semantic label.
  • determining the search parameter is based on a prefix of the second natural language input.
  • the method further includes determining, that an additional entry is responsive to the second natural language input and selecting the entry in lieu of the additional entry based on conformance of the second natural language input of the user to the semantic label of the entry.
  • providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry, and the additional entry is not utilized in generating the natural language output and no output that is based on the additional entry is provided in response to the second natural language input.
  • the method further includes determining, based on at least one of the terms of the second natural language input, a measure indicative of a likelihood that the second natural language input is a request for information from the personal database. In some of those implementations, at least one of the searching and the providing are dependent on a magnitude of the measure.
  • a method performed by one or more processors includes: receiving first natural language input that is free-form input formulated by a user via a user interface input device of a computing device of the user and generating an entry for the first natural language input in a personal database of the user stored in one or more computer readable media.
  • the generating includes: storing, in the entry, one or more given terms of the terms of the first natural language input or an identifier of the given terms; generating descriptive metadata based on at least one of: one or more of the terms of the first natural language input, and one or more contextual features associated with receiving the first natural language input; and storing the descriptive metadata in the entry.
  • the method further includes: receiving second natural language input subsequent to receiving the first natural language input; determining at least one search parameter based on the second natural language input; searching the personal database based on the search parameter; determining, based on the searching, that the entry is responsive to the second natural language input; determining, based on the searching, that an additional entry is also responsive to the second natural language input; ranking the entry relative to the additional entry based on at least some of the descriptive metadata; generating natural language output that includes one or more natural language output terms that are based on the entry; and providing, in response to the second natural language input, the natural language output for presentation to the user via a user interface output device, wherein providing the natural language output is based on the ranking of the entry relative to the additional entry.
  • providing the natural language output based on the ranking of the entry relative to the additional entry includes providing the natural language output without providing any output that is based on the additional entry in response to the second natural language input.
  • providing the natural language output based on the ranking of the entry relative to the additional entry includes providing the natural language output and providing additional output based on the additional entry, and providing the natural language output and the additional output for presentation in an order that is based on the ranking.
  • some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods.
  • Some implementations also include a non-transitory computer readable storage medium storing computer instructions executable by one or more processors to perform any of the aforementioned methods.
  • FIG. 1 is a diagram of an example environment in which implementations disclosed herein may be implemented.
  • FIG. 2A illustrates an example of generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user.
  • FIG. 2B illustrates an example of generating, based on one or more personal database entries of a user, natural language output that is responsive to further free-form natural language input of the user.
  • FIG. 3A illustrates an example client computing device with a display screen displaying an example of dialog that may occur between a user of the client computing device and an automated personal assistant according to implementations described herein.
  • FIGS. 3B, 3C, 3D, 3E, 3F, and 3G each illustrates a portion the example client computing device of FIG. 3A , with the display screen displaying dialog that may occur subsequent to the dialog of FIG. 3A according to implementations described herein.
  • FIG. 4A illustrates the client computing device of FIG. 3A with the display screen displaying another example of dialog that may occur between the user of the client computing device and the automated personal assistant according to implementations described herein.
  • FIGS. 4B and 4C each illustrates a portion the example client computing device of FIG. 4A , with the display screen displaying dialog that may occur subsequent to the dialog of FIG. 4A according to implementations described herein.
  • FIGS. 5A and 5B present a flow chart illustrating an example method of generating personal database entries for a user based on free-form natural language input formulated by the user and generating, based on one or more of the personal database entries, natural language output that is responsive to further free-form natural language input of the user.
  • FIG. 6 illustrates an example architecture of a computing device.
  • Implementations of this specification are directed to generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user.
  • the personal database entry may be generated by an automated personal assistant in response to natural language input provided to the automated personal assistant.
  • a personal database entry is an entry in a personal database that is accessible to the user and non-accessible to a plurality of additional users that differ from the user.
  • the personal database may be accessible to only the user and one or more applications and/or systems specified by the user and non-accessible by any other users, applications, and/or systems.
  • the personal database may be accessible to the user and certain additional users designated by the user.
  • any personal database may be stored on one or more computer readable media and may be associated with access information to allow access only to users, applications, and/or systems that are authorized to access such content and prevent access to all other users, applications, and/or systems.
  • a generated personal database entry includes one or more terms of the natural language input and optionally includes descriptive metadata that is determined based on one or more terms of the natural language input and/or based on contextual features associated with receiving the natural language input.
  • Descriptive metadata that is determined based on the terms of the natural language input may include, for example, a semantic label indicating a classification to which one or more of the terms belong and to which additional terms not included in the natural language output also belong. For example, a semantic label of “person” may be determined based on the term “Bob”, a semantic label of “place” may be determined based on the term “home”, etc.
  • Another example of descriptive metadata determined based on the terms of the natural language input may include “memoryness” metadata that indicates the likelihood (e.g., true/false, or a range of values) that the natural language input is intended as input the user desires an automated personal assistant to be able to recall at a later time.
  • descriptive memoryness metadata that indicates a high likelihood the user desires the ability for future recall may be generated based on the presence of certain key terms in the input (e.g., “remember”, “don't forget”) and/or based on output received in response to providing the input to a classifier trained to predict whether and/or to what degree natural language input indicates a desire for the natural language input to be “remembered”.
  • the descriptive metadata that is determined based on the contextual features associated with receiving the natural language input may include, for example, temporal metadata, locational metadata, and/or additional user metadata.
  • the temporal metadata may indicate a date (e.g., May 1, 2016; May; 2016; last week; Sunday) and/or a time (e.g., 8:00 AM, morning, between 7:00 and 9:00) that the natural language input was received.
  • the locational metadata may indicate a location of the user (e.g., a ZIP code, “home”, “work”, a City, a Neighborhood) when the natural language input was received.
  • the additional user metadata may indicate one or more additional users to whom the natural language input was directed and/or one or more additional users who were engaged in communication with the user when the natural language input was received. For example, when the natural language input is received from an ongoing message exchange thread (e.g., chat, SMS) between the user and an additional user, the additional user metadata may indicate the additional user.
  • an ongoing message exchange thread e.g., chat, SMS
  • Implementations of this specification are additionally and/or alternatively directed to generating, based on one or more personal database entries of a user, output that is responsive to further free-form natural language input of the user.
  • one or more entries that are responsive to further natural language input of the user can be identified based on matching (soft and/or exact) content of those entries (e.g., descriptive metadata and/or natural language input terms of those entries) to one or more search parameters determined based on the further natural language input.
  • content of an identified entry may be utilized in generating the output and/or in ranking multiple identified entries (where multiple entries are identified) relative to one another. The ranking can be utilized to select a subset of entries to utilize in generating the response and/or to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content were derived).
  • the example environment includes a plurality of client computing devices 106 1-N and an automated personal assistant 120 .
  • automated personal assistant 120 is illustrated in FIG. 1 as separate from the client computing devices 106 1-N , in some implementations all or aspects of the automated personal assistant 120 may be implemented by one or more of the client computing devices 106 1-N .
  • client computing device 106 1 may implement one instance or more aspects of automated personal assistant 120 and client computing device 106 N may also implement a separate instance of those one or more aspects of automated personal assistant 120 .
  • the client computing devices 106 1-N and those aspects of automated personal assistant 120 may communicate via one or more networks such as a local area network (LAN) and/or wide area network (WAN) (e.g., the Internet).
  • LAN local area network
  • WAN wide area network
  • the client computing devices 106 1-N may include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a dedicated computing device that enables dialog with users (e.g., a stand-alone device that includes a microphone and speaker, but no display), a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.
  • a desktop computing device e.g., a laptop computing device, a tablet computing device, a dedicated computing device that enables dialog with users (e.g., a stand-alone device that includes a microphone and speaker, but no display),
  • a given user may communicate with automated personal assistant 120 utilizing a plurality of client computing devices of the user that collectively form a coordinated “ecosystem” of computing devices.
  • a personal database entry may be generated based on natural language input of a user via a first client computing device and output generated based on that entry in response to further natural language input of the user via a second client computing device.
  • many examples described in this disclosure will focus on a given user operating a single one of the client computing devices 106 1-N .
  • Each of the client computing devices 106 1-N may operate a variety of different applications, such as a corresponding one of the message exchange clients 107 1-N .
  • Message exchange clients 107 1-N may come in various forms and the forms may vary across the client computing devices 106 1-N and/or multiple forms may be operated on a single one of the client computing devices 106 1-N .
  • one or more of the message exchange clients 107 1-N may come in the form of a short messaging service (“SMS”) and/or multimedia messaging service (“MMS”) client, an online chat client (e.g., instant messenger, Internet relay chat, or “IRC,” etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to dialog with automated personal assistant 120 (e.g., implemented on a dedicated computing device that enables dialog with a user), and so forth.
  • SMS short messaging service
  • MMS multimedia messaging service
  • IRC Internet relay chat
  • one or more of the message exchange clients 107 1-N may be implemented via a webpage or other resources rendered by a web browser (not depicted) or other application of client computing device 106 .
  • the automated personal assistant 120 receives input from and/or provides output to one or more users via user interface input and output devices of one or more client devices 106 1-N .
  • Implementations described herein relate to the automated personal assistant 120 generating personal database entries for a user based on natural language input provided by the user and generating output for presentation to the user based on one or more generated personal database entries of the user.
  • the automated personal assistant 120 may also perform one or more additional functions.
  • the automated personal assistant 120 may respond to some natural language interface input of a user with output that is generated, in whole or in part, based on information that is not generated based on a personal database entry generated based on past natural language input of the user.
  • the automated personal assistant may utilize one or more documents of public database 154 in generating a response to some natural language interface input of the user. For example, in response to user natural language input of “what does ambidextrous mean?”, the automated personal assistant may generate output of “ambidextrous is an adjective that describes the ability to use the right and left hands equally well” based on a definition for “ambidextrous” that is not personal to the user.
  • user interface input described herein is explicitly directed to the automated personal assistant 120 .
  • one of the message exchange clients 107 1-N may be a personal assistant messaging service dedicated to conversations with automated personal assistant 120 and user interface input provided via that personal assistant messaging service may be automatically provided to automated personal assistant 120 .
  • one of the message exchange clients 107 1-N and an instance of at least aspects of the personal assistant 120 may both be implemented at least in part on a stand-alone client computing device that at least selectively monitors for voice user interface input and responds with audible user interface output.
  • the user interface input may be explicitly directed to the automated personal assistant 120 via one or more of the message exchange clients 107 1-N based on particular user interface input that indicates the automated personal assistant 120 is to be invoked.
  • the particular user interface input may be one or more typed characters (e.g., @PersonalAssistant), user interaction with a virtual button (e.g., a tap, a long tap), a verbal command (e.g., “Hey Personal Assistant”), etc.
  • the automated personal assistant 120 may perform one or more actions in response to user interface input, even when that user interface input is not explicitly directed to the automated personal assistant 120 .
  • the automated personal assistant 120 may examine the contents of user interface input and perform one or more actions in response to certain terms being present in the user interface input and/or based on other cues.
  • Each of the client computing devices 106 1-N and automated personal assistant 120 may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network.
  • the operations performed by one or more of the client computing devices 106 1-N and/or by the automated personal assistant 120 may be distributed across multiple computer systems.
  • Automated personal assistant 120 may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network.
  • Automated personal assistant 120 may include an input processing engine 122 , a contextual engine 124 , an entry generation engine 126 , a searching engine 128 , a ranking engine 130 , and an output generation engine 132 .
  • one or more of engines 122 , 124 , 126 , 128 , 130 , and/or 132 may be omitted.
  • all or aspects of one or more of engines 122 , 124 , 126 , 128 , 130 , and/or 132 may be combined.
  • one or more of engines 122 , 124 , 126 , 128 , 130 , and/or 132 may be implemented in a component that is separate from automated personal assistant 120 .
  • one or more of engines 122 , 124 , 126 , 128 , 130 , and/or 132 , or any operative portion thereof may be implemented in a component that is executed by of the client computing devices 106 1-N .
  • Input processing engine 122 processes natural language input generated by users via client computing devices 106 1-N and generates annotated output for use by one or more other components of the automated personal assistant 120 .
  • the input processing engine 122 may process natural language free-form input that is generated by a user via one or more user interface input devices of client device 106 1 .
  • the generated annotated output includes one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input.
  • the input processing engine 122 is configured to identify and annotate various types of grammatical information in natural language input.
  • the input processing engine 122 may include a part of speech tagger configured to annotate terms with their grammatical roles.
  • the part of speech tagger may tag each term with its part of speech such as “noun,” “verb,” “adjective,” “pronoun,” etc.
  • the input processing engine 122 may additionally and/or alternatively include a dependency parser configured to determine syntactic relationships between terms in natural language input.
  • the dependency parser may determine which terms modify other terms, subjects and verbs of sentences, and so forth (e.g., a parse tree)—and may make annotations of such dependencies.
  • the input processing engine 122 may additionally and/or alternatively include an entity tagger configured to annotate entity references in one or more segments such as references to people, organizations, locations, and so forth.
  • entity tagger may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person).
  • the entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.
  • the input processing engine 122 may additionally and/or alternatively include a coreference resolver configured to group, or “cluster,” references to the same entity based on one or more contextual cues.
  • the coreference resolver may be utilized to resolve the term “it” to “bike helmet” in the natural language input “I'm looking for my bike helmet. Do you know where it is?”
  • one or more components of the input processing engine 122 may rely on annotations from one or more other components of the input processing engine 122 .
  • the named entity tagger may rely on annotations from the coreference resolver and/or dependency parser in annotating all mentions to a particular entity.
  • the coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity.
  • one or more components of the input processing engine 122 may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.
  • a first user in a message exchange thread may provide input of “I'll leave the keys under the doormat for you” and a second user may provide, via the message exchange thread, input to the automated messaging system of “Remember that”.
  • the coreference resolver may resolve “that” to “I'll leave the keys under the doormat for you” and may further resolve “I'll” to the first user and “you” to the second user.
  • the contextual engine 124 determines one or more contextual features that are associated with natural language input received from one of the client computing devices 106 1-N .
  • the contextual engine 124 determines one or more of the contextual features independent of terms of the natural language input.
  • contextual feature may indicate a date and/or a time that the natural language input was received and may be determined independent of terms of the natural language input.
  • a contextual feature may indicate a location of the user (e.g., based on GPS or other location data) when the natural language input was received and may be determined independent of terms of the natural language input.
  • a contextual feature may indicate one or more additional users to whom the natural language input was directed and/or one or more additional users who were engaged in communication with the user when the natural language input was received—and may be determined independent of terms of the natural language input.
  • the contextual engine 124 determines one or more of the contextual features independent of any explicit input of the user that provided the natural language input, such as explicit user interface input that defines the contextual features and/or explicit input that confirms automatically determined contextual features are correct.
  • the entry generation engine 126 generates personal database entries for a user in response to at least some instances of natural language input formulated by the user via a corresponding one of the client computing devices 106 1-N .
  • the entry generation engine 126 may utilize one or more terms of the natural language input, annotations of the natural language input provided by input processing engine 122 , and/or contextual features provided by contextual engine 124 .
  • natural language input of “remember my locker combination is 10, 20, 30” that is formulated by a user via one of the client computing devices 106 1-N and provided to the automated personal assistant 120 .
  • the contextual engine 124 determines the natural language input is provided at 6:00 AM on May 1, 2016 while the user's location is “at the gym”.
  • the user's location of “at the gym” may be defined with any one or more of various levels of granularity, such as a specific latitude/longitude, a street address, an entity identifier that identifies the particular gym, an entity class identifier of “gym”, etc.
  • the entry generation engine 126 may generate an entry that includes the terms “locker combination is 10, 20, 30”; temporal metadata of: 6:00 AM, “early morning”, and/or some other indication of 6:00 AM and temporal metadata of “May 1 st ”, “Sunday in May”, and/or some other identifier of “May 1, 2016”; and locational metadata that is an indication of the user's location of “at the gym.”
  • the entry generation engine 126 may generate a personal database entry based on received natural language input in response to that natural language input having one or more characteristics. For example, in some of those implementations, the entry generation engine 126 may generate a personal database entry in response to natural language input that includes one or more key terms (e.g., “remember”, “remember this”, and/or “don't forget”) and/or that includes those key terms in certain positions of the natural language input (e.g., as a prefix).
  • key terms e.g., “remember”, “remember this”, and/or “don't forget”
  • the entry generation engine 126 may additionally and/or alternatively generate a personal database entry in response to providing at least some of natural language input to a classifier trained to predict whether and/or to what degree natural language input indicates a desire for the natural language input to be “remembered”—and receiving, as output from the classifier, an indication that the natural language input indicates a desire to be remembered and/or indicates the desire to at least a threshold degree.
  • the entry generation engine 126 may also generate “memoryness” metadata for an entry that indicates a likelihood that the natural language input utilized to generate the entry indicates a desire for the automated personal assistant 120 to recall at least some of the natural language input at a later time. For example, the entry generation engine 126 may generate memoryness metadata that is a first value (e.g., “1”) when the natural language input includes one or more key terms (e.g., “remember”, “remember this”, and/or “don't forget”) and is a second value (e.g., “0”) when the natural language input does not include any of those key terms.
  • a first value e.g., “1”
  • key terms e.g., “remember”, “remember this”, and/or “don't forget”
  • the entry generation engine 126 may generate memoryness metadata that is selected from one of three or more values based on one or more characteristics of the natural language input and/or of associated contextual features. For example, the entry generation engine 126 may provide at least some of received natural language input and/or one or more contextual features to a classifier trained to predict whether and/or to what degree natural language input indicates a desire for the natural language input to be “remembered”—and receive, as output from the classifier, a measure of the likelihood that the natural language input indicates a desire for the natural language input to be remembered. The engine 126 may utilize that measure, or another indication based on that measure, as the memoryness metadata.
  • the personal database entries generated by entry generation engine 126 may be stored in personal database 152 .
  • Personal database 152 may be provided on one or more non-transitory computer readable media, such as one or more media that are local to one of the client computing devices 106 1-N , local to the automated personal assistant 120 , and/or remote from the client computing devices 106 1-N and/or the automated personal assistant 120 .
  • Each personal database entry of personal database 152 may include the underlying content itself and/or one or more index items that link to the underlying content.
  • the personal database 152 may be local to one of the client computing devices 106 1-N , and that client computing device may implement an instance of one or more (e.g., all) aspects of the automated personal assistant 120 .
  • that client computing device may include one or more user interface input devices (e.g., a microphone), one or more user interface output devices (e.g., a speaker), and the personal database 152 .
  • Personal database 152 enables searching of personal database entries to determine whether all or aspects of an entry are relevant to further natural language input of the user as described herein.
  • multiple iterations of personal database 152 may be provided with each iteration being specific to a particular user and may include personal database entries that are accessible to the user and/or system(s) designated by the user and non-accessible to a plurality of additional users and/or systems that differ from the user.
  • the personal database 152 may be accessible only to the user and non-accessible by any other users.
  • the personal database 152 may be accessible to the user and certain additional users designated by the user.
  • the personal database 152 may be associated with access information to allow access to users who are authorized to access such content and prevent access to all other users.
  • the personal database 152 includes personal database entries of a plurality of users, with each entry and/or a group of entries associated with access information to allow access only to users and/or systems who are authorized to access such index items and prevent access to all other users and/or systems.
  • the personal database 152 may include entries for a plurality of users, but each entry and/or a group of entries may include access information to prevent access by any users not authorized to access such entries. Additional and/or alternative techniques for restricting access to entries of personal database 152 may be utilized.
  • the searching engine 128 searches entries of personal database 152 that are associated with a user in response to at least some instances of natural language input formulated by the user via a corresponding one of the client computing devices 106 1-N .
  • the searching engine 128 generates one or more search parameters based on the formulated natural language input, and optionally based on annotations of that input provided by input processing engine 122 , and utilizes those search parameters to determine one or more entries of the personal database 152 that include content (e.g., descriptive metadata and/or natural language input terms of those entries) that matches (soft match and/or exact match) one or more (e.g., all) of the search parameters.
  • content e.g., descriptive metadata and/or natural language input terms of those entries
  • a search parameter generated by the searching engine 128 may include, for example, one or more terms of the natural language input, a synonym of one or more terms of the natural language input, a semantic label for one or more terms of the natural language input, a location restriction based on one or more terms of the natural language input, a temporal restriction based on one or more terms of the natural language input, etc.
  • the searching engine 128 may determine search parameters of “bike helmet” (based on terms of the natural language input), “bicycle helmet” (based on a synonym of the natural language input), a semantic label indicative of “person” (based on presence of “who” in the natural language input), and a temporal restriction corresponding to “last month”.
  • the searching engine 128 searches the personal database 152 based on received natural language input in response to that natural language input having one or more characteristics. For example, in some of those implementations, the searching engine 128 may search the personal database 152 in response to natural language input that includes one or more key terms (e.g., “What”, “Who”, “Where”, “When”, “my”, and/or “I”) and/or that includes those key terms in certain positions of the natural language input.
  • key terms e.g., “What”, “Who”, “Where”, “When”, “my”, and/or “I”
  • the search engine may search the personal database 152 in response to natural language input that includes an inquiry term such as “what”, “who”, or “where” with a syntactical relationship (e.g., as indicated by annotations of the input processing engine 122 ) to one or more terms such as “I” or “my”.
  • the searching engine 128 may additionally and/or alternatively search the personal database 152 in response to providing at least some of natural language input to a classifier trained to predict whether and/or to what degree natural language input is intended as a request for personal information from a personal database of the user—and receiving, as output from the classifier, an indication that the natural language input indicates a desire and/or indicates a desire to at least a threshold degree.
  • the searching engine 128 may also search the public database 154 in response to at least some instances of natural language input formulated by the user via a corresponding one of the client computing devices 106 1-N .
  • the searching engine 128 may search both the public database 154 and the personal database 152 in response to some natural language input and/or may search only one of the public database 154 and the personal database 152 .
  • the public database 154 may include various types of content that are not restricted to the user and/or to other user's and/or systems designated by the user, and that may optionally be utilized by the searching engine 128 to respond to various types of natural language input.
  • the public database 154 may include content items from publicly accessible documents on the World Wide Web, scripted responses to certain natural language input, etc.
  • the public database 154 may include a definition for “ambidextrous” to enable the searching engine 128 to identify that definition in response to natural language input of “what does ambidextrous mean?”.
  • the public database 154 may include a scripted response of “Hello, how are you” that is identified by the searching engine 128 in response to natural language input of “Hi” or “Hello”.
  • the ranking engine 130 optionally ranks one or more entries (personal and/or public) identified by the searching engine 128 based on one or more criteria.
  • the ranking engine 130 optionally calculates a score for each of one or more entries that are responsive to the query using the one or more criteria.
  • the score of an entry may itself be used as the ranking for the entry, or may be compared to scores of other entries (where other entries are determined) to determine the ranking of the entry.
  • Each criteria used to rank an entry provides information about the entry itself and/or the relationship between the entry and the natural language input.
  • the ranking of a given entry may be based on conformance of a semantic label of descriptive metadata of the entry to a semantic label generated for the natural language input. For example, if a semantic label for one or more terms of the natural language input matches a semantic label in descriptive metadata of the given entry, the ranking of the given entry may be positively influenced.
  • the ranking of the given entry may additionally and/or alternatively be based on conformance of locational metadata of the given entry to a current location of the user (e.g., determined by contextual engine 124 ) when the natural language input was received. For example, if a current location of the user is the same as, or within a threshold distance of, a location indicated by locational metadata of the given entry, the ranking of the given entry may be positively influenced.
  • the ranking of the given entry may additionally and/or alternatively be based on conformance of temporal metadata of the given entry to a current date and/or time (e.g., determined by contextual engine 124 ) when the natural language input was received. For example, if a current date is within a threshold of a date indicated by temporal data of the given entry, the given entry may be promoted more than if the current date was not within the threshold.
  • the ranking of the given entry may additionally and/or alternatively be based on memoryness metadata for the given entry. For example, if a measure included in or defined by the memoryness metadata satisfies a threshold, the ranking of the given entry may be promoted more than if the measure did not satisfy the threshold.
  • the searching engine 128 identifies a first entry from the public database 154 and a second entry from the personal database 152
  • the rankings of the entries may be based on whether and/or to what degree the natural language input is intended as a request for personal information from a personal database of the user (e.g., as described above with respect to searching engine 128 ). For example, if characteristics of the natural language input indicate it is highly likely to be a request for personal information from a personal database of the user, any identified entries from the public database 154 may be demoted.
  • the output generation engine 132 utilizes content of one or more entries identified by searching engine 128 , and optionally the rankings of those entries determined by ranking engine 130 , to generate output for presentation to the user as a response to the provided natural language input. For example, the output generation engine 132 may select a subset (e.g., one) of identified entries based on the ranking and may generate output based on the selected subset. Also, for example, the output generation engine 132 may additionally and/or alternatively utilize the ranking to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content were derived).
  • the output generation engine 132 may include one or more of the terms of the given entry that are from the natural language input for which the given entry was generated and/or may include output that is based on generated descriptive metadata for the entry.
  • FIGS. 2A and 2B additional description is provided of various components of automated personal assistant 120 .
  • a user utilizes one or more user interface input devices of the computing device 106 1 to provide natural language input 201 A to message exchange client 107 1 , which transmits the natural language input 201 A to input processing engine 122 .
  • the natural language input 201 A may be free-from input as described herein and may be, for example, typed input provided by the user via a physical or virtual keyboard of a computing device or spoken input provided by the user to a microphone of the computing device.
  • the natural language input 201 B is spoken input, it may optionally be transcribed to textual input by the computing device and/or by the input processing engine 122 .
  • the input processing engine 122 processes the natural language input 201 A and generates various annotations of the natural language input.
  • the input processing engine 122 provides annotated input (e.g., terms of the natural language input 201 A and generated annotations) 202 A to entry generation engine 126 .
  • the contextual engine 124 determines contextual features associated with receiving the natural language input 201 A. For example, the contextual engine 124 may determine the contextual features based on data provided by message exchange client 107 1 (e.g. a timestamp, locational data) and/or other applications and/or the operating system of the client computing device 106 1 . In some implementations, the contextual features 204 may be utilized by the entry generation engine 126 as described below and/or may be utilized by the input processing engine 122 (e.g., to improve entity resolution and/or other determinations based on user location and/or other contextual features).
  • message exchange client 107 1 e.g. a timestamp, locational data
  • the contextual features 204 may be utilized by the entry generation engine 126 as described below and/or may be utilized by the input processing engine 122 (e.g., to improve entity resolution and/or other determinations based on user location and/or other contextual features).
  • the entry generation engine 126 generates a personal database entry 203 for a user and stores it in personal database 152 .
  • the entry generation engine 126 may utilize one or more terms of the natural language input 201 A, annotated input 202 A of the natural language input provided by input processing engine 122 , and/or contextual features 204 provided by contextual engine 124 .
  • FIG. 2B illustrates an example of generating, based on one or more personal database entries of a user, natural language output that is responsive to further free-form natural language input of the user.
  • the example of FIG. 2B may occur subsequent to the example of FIG. 2A and may utilize the entry 203 generated in FIG. 2B .
  • a user utilizes one or more user interface input devices of the computing device 106 1 to provide further natural language input 201 B to message exchange client 107 1 , which transmits the natural language input 201 B to input processing engine 122 .
  • the natural language input 201 B may be free-from input as described herein and may be, for example, typed input provided by the user via a physical or virtual keyboard of a computing device or spoken input provided by the user to a microphone of the computing device.
  • the input processing engine 122 processes the natural language input 201 A and generates various annotations of the natural language input.
  • the input processing engine 122 provides annotated output (e.g., terms of the natural language input 201 B and generated annotations) 202 B to searching engine 128 .
  • the searching engine 128 generates one or more search parameters based on the annotated input 202 B, and utilizes those search parameters to determine one or more entries of the personal database 152 and/or public database 154 that include content that matches one or more of the search parameters. As described herein, in some implementations the searching engine 128 may search only personal database 152 in response to the natural language input 201 B based on one or more characteristics of the natural language input 201 B. The searching engine 128 determines that entry 203 of personal database 152 is responsive to the search based on matching one or more search parameters to content of the entry 203 .
  • the searching engine 128 provides the entry 203 to the ranking engine 130 , which optionally generates a ranking for the entry 203 .
  • the ranking engine 130 may determine if a ranking score for the entry 203 satisfies a threshold that warrants the output generation engine 132 utilizing the entry 203 to generate natural language output 205 .
  • the contextual engine 124 determines contextual features associated with receiving the natural language input 201 B and may provide those contextual features to the ranking engine 130 for use in generating a ranking as described herein. In some implementations where the searching engine 128 determines multiple responsive entries, the ranking engine 130 may rank those relative to one another.
  • the output generation engine 132 utilizes content of the entry 203 to generate natural language output 205 for presentation to the user as a response to the natural language input 201 B.
  • the output generation engine 132 may generate output 205 that includes one or more of the terms of the entry 203 that are from the natural language input 201 A and/or that includes one or more terms that are based on generated descriptive metadata for the entry 203 .
  • the natural language output 205 is provided to the message exchange client 107 1 for audible and/or visual presentation to the user via a user interface output device of the client computing device 106 1 .
  • the output generation engine 132 may select a subset (e.g., one) of identified entries based on the ranking and may generate output based on the selected subset. Also, for example, the output generation engine 132 may additionally and/or alternatively utilize the ranking to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content were derived).
  • FIG. 3A illustrates the client computing device 106 1 of FIG. 1 with a display screen 140 displaying an example of dialog that may occur between a user of the client computing device 106 1 and the automated personal assistant 120 according to implementations described herein.
  • the dialog includes natural language input 380 A, 380 B, 380 C provided by the user to an automated personal assistant and responses 390 A, 390 B, 390 C that may optionally be provided by the automated personal assistant 120 to confirm the assistant has generated an entry in response to the inputs.
  • alternative response may be provided by the automated personal assistant 120 .
  • the automated personal assistant may respond with a prompt to seek such further clarification.
  • the automated personal assistant 120 may identify both a “Bob Smith” and “Bob White” in contact entries of the user and may generate a prompt such as “Do you mean Bob Smith or Bob White?” to solicit further input from the user to enable determination of a particular contact and further tailoring of an entry based on the input 380 A.
  • the automated personal assistant 120 may determine that the input 380 D conflicts with previous input 380 B, and the assistant 120 might prompt “Should this replace the bike lock combination you provided on May 5, 2016?” to determine if an entry generated based on the input 380 D should replace an entry generated based on the input 380 B.
  • FIG. 3A also shows, in parentheses, a date and a time that the user provided each of the various inputs 380 A-D.
  • the dates and times may be utilized in generating descriptive metadata.
  • the dates and/or times may optionally not be shown in the display provided to the user, but are illustrated in FIG. 3A for purposes of clarity.
  • the display screen 140 further includes a textual reply interface element 384 that the user may select to generate user input via a virtual keyboard and a voice reply interface element 385 that the user may select to generate user input via a microphone.
  • the display screen 140 also includes system interface elements 381 , 382 , 383 that may be interacted with by the user to cause the computing device 106 1 to perform one or more actions.
  • FIGS. 3A, 3B, 3C, 3D, 3E, 3F, and 3G each illustrates a portion the example client computing device 106 1 , with the display screen 140 displaying dialog that may occur subsequent to the dialog of FIG. 3A according to implementations described herein.
  • the user provides, at a later time, further natural language input 380 E of “who has my bike helmet?”.
  • the automated personal assistant 120 provides a response 390 E of “On May 1 st you told me that Bob had your bike helmet.”
  • the response 390 E may be generated by: issuing a search based on “where is my bike helmet”; identifying, as responsive to the search, an entry that was created based on the user input 380 A of FIG. 3A ; and using that entry to generate the response 390 E.
  • an entry generated for user input 380 A may include the terms “Bob has my bike helmet” and descriptive metadata of: a semantic label of “person” associated with “Bob”, a semantic label of “object” associated with “bike helmet”, and temporal metadata indicating May 1 st and/or 8:00 AM.
  • a search may be issued based on the user input 380 E, such as a search that includes a first search parameter of “bike helmet” and a second search parameter of a semantic label of “person” (based on presence of “who” in the user input 380 E).
  • the entry for input 380 A may be identified based on having terms that match the first search parameter and a sematic label that matches the second search parameter.
  • the response 390 E may be generated to include temporal metadata from the entry for input 380 A (“On May 1 st ”) and the text from the entry for input 380 A (“Bob has your Bike helmet”).
  • the automated personal assistant provides a response 390 F of “May 7, 2016”
  • the response 390 F may be generated by: issuing a search based on “when did I put tiers on my bike”; identifying, as responsive to the search, an entry that was created based on the user input 380 B of FIG. 3A ; and using that entry to generate the response 390 F.
  • an entry generated for user input 380 B may include the terms “put new tires on my bike on May 7, 2016”.
  • the entry may include the terms “May 7, 2016” instead of “today” (which is provided in the user input 380 B) based on entry generation engine 126 replacing “today” with the date the user input 380 B was received (e.g., based on annotations provided by input processing engine 122 ).
  • the entry generated for user input 380 B may also include descriptive metadata of: temporal metadata indicating May 7, 2016 and a semantic label of “when” associated with the terms “May 7, 2016” of the entry.
  • a search may be issued based on the user input 380 F, such as a search that includes a first search parameter of “tires”, a second search parameter of “bike”, and a third search parameter of a semantic label of “when” (based on presence of “when” in the user input 380 F).
  • the entry generated for input 380 B may be identified based on having terms that match the first search parameter and the second search parameter and having semantic label metadata that matches the third search parameter.
  • the response 390 F may be generated to include only the terms “May 7, 2016” from the entry for input 380 B based on those terms being associated with the semantic label metadata that matches the third search parameter.
  • FIGS. 3D and 3E illustrate further natural language inputs 380 G 1 and 380 G 2 of “What is my bike lock combination”.
  • a first entry generated for user input 380 B ( FIG. 3A ) and a second entry generated for user input 380 D ( FIG. 3A ) may both identified based on a search for natural language inputs 380 G 1 , 380 G 2 .
  • a response 390 G that is generated based on the second entry for user input 380 D is provided without providing any output that is generated based on the first entry for user input 380 B.
  • This may be based on ranking the first entry and the second entry based on one or more criteria and determining the second entry is ranked higher than the first entry.
  • the second entry may be ranked higher based on temporal metadata for the second entry being more proximal to a current time than temporal metadata for the first entry.
  • the response 390 G is generated based on the second entry for user input 380 D without providing any output that is generated based on the first entry for user input 380 B.
  • Providing the content from only one of the two answers may be based on various factors such as system settings, size of the display screen 140 , and/or based on processing of input 380 G 1 to understand that only a single answer is desired.
  • a response may include output generated based on both the first entry and the second entry.
  • a response 390 H is generated that is based on both the second entry for user input 380 D and the first entry for user input 380 B.
  • the portion of the response 390 H that corresponds to the second entry is presented before the portion based on the first entry. This may be based on, for example, the ranking of the second entry being higher than the ranking of the first entry (e.g., based on temporal metadata for the second entry being more proximal to a current time than temporal metadata for the first entry).
  • the user provides, at a later time, further natural language input 380 H of “How do I put tires on my bike?”.
  • the automated personal assistant 120 provides a response 390 I that includes part of the steps for putting new tires on a bike.
  • the response 390 I is an example of a response that may be generated by the automated personal assistant 120 based on a public entry of public database 154 .
  • the further natural language input 380 H includes the terms “tires” and “bikes” that may also be included in a personal entry generated based on the user input 380 C ( FIG. 3A )
  • the automated personal assistant 120 does not provide responsive content that is based on the personal entry. This may be based on one or more of various factors.
  • a semantic label for the input 380 H may not match a semantic label for the personal entry generated based on the user input 380 C. This may lead to that entry not being determined and/or may negatively influence the ranking of that entry.
  • the ranking for the public entry utilized to generate response 390 I may be higher than the ranking of the personal entry—and the response 390 I generated based on only the public entry based on a rule that dictates responses are to be generated based on only the highest ranked entry and/or based on only the highest ranked entry when its ranking score satisfies a threshold relative to the next highest ranked entry.
  • the automated personal assistant 120 provides a response 390 J that includes content from a private database entry and content from a public database.
  • the response 390 J includes content (“You put new tires on your bike on May 7, 2016”) that is based on a personal entry generated for the input 380 C ( FIG. 3A ).
  • the response 390 J also includes content (“Do you want to learn how to install new bike tires?”) that may be based on a public entry of public database 154 .
  • the automated personal assistant 120 may include content that is based on an entry generated for the input 380 C based on matching search parameters of “new” “tires” and “bike” to content of the entry.
  • the content from the private database entry may be presented before the content from the public database based on a ranking of the entries underlying the content.
  • FIG. 4A illustrates the client computing device 106 1 with the display screen 140 displaying another example of dialog that may occur between the user of the client computing device 106 1 and the automated personal assistant 120 according to implementations described herein.
  • the dialog includes natural language inputs 480 A and 480 B provided by the user to the automated personal assistant 120 and responses 490 A and 490 B that may optionally be provided by the automated personal assistant 120 to confirm the assistant 120 has generated an entry in response to the inputs.
  • FIG. 4A also shows, in parentheses, a date and a time that the user provided each of the inputs 480 A and 480 B.
  • the dates and times may be utilized in generating descriptive metadata. The dates and/or times may optionally not be shown in the display provided to the user, but are illustrated in FIG. 4A for purposes of clarity.
  • the display screen 140 further includes a textual reply interface element 484 that the user may select to generate user input via a virtual keyboard and a voice reply interface element 485 that the user may select to generate user input via a microphone.
  • the display screen 140 also includes system interface elements 481 , 482 , 483 that may be interacted with by the user to cause the computing device 106 1 to perform one or more actions.
  • the user provides first user input 480 A that includes “my locker combination is 7, 22, 40” and provides it at the “Gym” on “May 1, 2016” at “6:00 AM”.
  • a first entry may be created for the first user input that includes one or more terms from the input, includes locational metadata indicative of the “Gym”, and that includes temporal metadata indicative of May 1, 2016 and/or 6:00 AM.
  • the user provides second user input 480 B that includes “my locker combination is 40, 20, 10” and provides it at “Work” on “May 10, 2016” at “8:00 AM”.
  • a second entry may be created for the first user input that includes one or more terms from the input, includes locational metadata indicative of “Work”, and that includes temporal metadata indicative of May 10, 2016 and/or 8:00 AM.
  • FIGS. 4B and 4C each illustrates a portion the example client computing device 106 1 with the display screen 140 displaying dialog that may occur subsequent to the dialog of FIG. 4A according to implementations described herein.
  • the user provides further input 480 C of “What is my locker combination?” and provides it at 5:45 AM while the user is at the Gym.
  • the automated personal assistant 120 provides a response 490 C that is generated based on the first entry for the user input 480 A of FIG. 4A —and that does not include any content generated based on the second entry for the user input 480 B. This may be based on, for example, conformance of the locational metadata of the first entry to the current location of the user and/or conformance of the temporal metadata to a current time. Compare this to a situation where the user was at “Work” at “8:00 AM”. In such a situation the response may be generated based on the second entry and independent of the first entry.
  • the user provides further input 480 D of “What is my locker combination?” that is the same as the input 480 C of FIG. 4B .
  • the automated personal assistant 120 provides the response 490 D that is generated based on both the first entry for the user input 480 A of FIG. 4A —and the second entry for the user input 480 B.
  • Generating a response based on both entries instead of just one may be based on, for example, the user's current location not conforming strongly to the locational metadata of either of the entries and/or the current time not conforming strongly to the temporal metadata of either of the entries.
  • content generated based on the second entry may be presented before content generated based on the first entry based on the date of temporal metadata of the second entry being more proximal to the current date than that of the first entry.
  • FIGS. 3A-3G and 4A-4C illustrate example graphical user interfaces that may be provided to a user
  • responses provided by automated personal assistant 120 may be provided for presentation audibly (e.g., via a speaker), without necessarily providing graphical indications of those responses.
  • a user may provide natural language input via a microphone and the automated personal assistant 120 may respond to such input with output that is configured for presentation audibly.
  • FIGS. 5A and 5B present a flowchart illustrating an example method 500 of generating personal database entries for a user based on free-form natural language input formulated by the user and generating, based on one or more of the personal database entries, natural language output that is responsive to further free-form natural language input of the user.
  • the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as automated personal assistant 120 .
  • operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.
  • the system awaits natural language input.
  • the system receives natural language input that is formulated by a user via one or more user interface input devices.
  • the system determines whether the natural language input of block 552 indicates a request for personal information, such as a request for personal information from a personal database of the user. For example, in some implementations, the system may determine the input indicates a request for personal information if it includes one or more key terms (e.g., “What”, “Who”, “Where”, “my”, “I” and/or “don't forget”) and/or that includes those key terms in certain positions of input. Also, for example, in some implementations the system may additionally and/or alternatively y determine the input indicates a request for personal information based on output received in response to providing at least some of input to a classifier trained to predict whether and/or to what degree natural language input is intended as a request for personal information.
  • key terms e.g., “What”, “Who”, “Where”, “my”, “I” and/or “don't forget”
  • the system may additionally and/or alternatively y determine the input indicates a request for personal information based on output received in response to providing at least some of
  • block 554 may be omitted (e.g., one or more of blocks 562 - 570 may be performed for each input) or instantiated (for the first time or again) after one or more of blocks 562 - 570 (e.g., the decision of block 554 may be based on whether any entries are determined at block 570 and/or may be based on the content and/or ranking of any returned entries).
  • block 556 the system determines whether the input indicates a desire for recall. For example, in some implementations the system may determine the input indicates a desire for recall based on the presence of certain terms in the input (e.g., “remember”, “don't forget”) and/or based on output received in response to providing the input to a classifier trained to predict whether and/or to what degree natural language input indicates a desire for the natural language input to be “remembered.” In some implementations, block 556 may be omitted. For example, one or more of blocks 558 and 560 may be performed for each input or for each input that is determined not to indicate a request for personal information at block 554 and/or that satisfies one or more other criteria.
  • certain terms in the input e.g., “remember”, “don't forget”
  • block 556 may be omitted. For example, one or more of blocks 558 and 560 may be performed for each input or for each input that is determined not to indicate a request for personal information at block 554 and
  • the system determines at block 556 the input does not indicate a desire for recall, the system proceeds back to block 550 and awaits further natural language output, optionally after providing an error output (e.g., “I don't understand your request”) and/or proceeding to other unillustrated blocks (e.g., blocks for searching a public database and/or providing other output).
  • an error output e.g., “I don't understand your request”
  • other unillustrated blocks e.g., blocks for searching a public database and/or providing other output.
  • the system determines at block 556 that the natural language input of block 552 does indicate a desire for recall, the system proceeds to block 558 where the system generates an entry for the input of block 552 in a personal database of the user.
  • the system in performing block 558 , the system generates descriptive metadata for the entry at block 558 A (e.g., based on terms of the input and/or contextual features associated with receiving the input) and stores one or more terms of the input of block 552 and the generated descriptive metadata in the entry at block 558 B.
  • the system may then optionally perform one or more other actions at block 560 , such as providing output that confirms an entry has been generated for the input, then proceeds back to block 550 .
  • the system determines one or more search parameters based on the input of block 552 .
  • the system searches a personal database of the user based on the search parameters determined at block 562 .
  • the system determines at least one personal entry of the personal database that is responsive to the search.
  • the system generates natural language output based on content of the at least one personal entry determined at block 566 .
  • the system may generate output based on a personal entry only when a ranking for that personal entry satisfies a threshold. The ranking may be for the personal entry itself, or for the personal entry relative to one or more other personal entries that may be determined at block 566 to be responsive to the search.
  • the system provides the output of block 568 for presentation to the user via one or more user interface output devices.
  • the system then returns to block 550 ( FIG. 5A ) and awaits further natural language input.
  • the system may optionally search one or more public databases in the method 500 and may optionally generate output with content from the public databases.
  • Such public database content may be included in output in combination with content from a personal entry and/or in lieu of content from the personal entry.
  • FIG. 6 is a block diagram of an example computing device 610 that may optionally be utilized to perform one or more aspects of techniques described herein.
  • one or more of the client computing devices 160 1-N , automated personal assistant 120 , and/or other component(s) may comprise one or more components of the example computing device 610 .
  • Computing device 610 typically includes at least one processor 614 which communicates with a number of peripheral devices via bus subsystem 612 .
  • peripheral devices may include a storage subsystem 624 , including, for example, a memory subsystem 625 and a file storage subsystem 626 , user interface output devices 620 , user interface input devices 622 , and a network interface subsystem 616 .
  • the input and output devices allow user interaction with computing device 610 .
  • Network interface subsystem 616 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.
  • User interface input devices 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices.
  • pointing devices such as a mouse, trackball, touchpad, or graphics tablet
  • audio input devices such as voice recognition systems, microphones, and/or other types of input devices.
  • use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device 610 or onto a communication network.
  • User interface output devices 620 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices.
  • the display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image.
  • the display subsystem may also provide non-visual display such as via audio output devices.
  • output device is intended to include all possible types of devices and ways to output information from computing device 610 to the user or to another machine or computing device.
  • Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein.
  • the storage subsystem 624 may include the logic to perform selected aspects of the method of FIGS. 5A and 5B .
  • Memory 625 used in the storage subsystem 624 can include a number of memories including a main random access memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored.
  • a file storage subsystem 626 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges.
  • the modules implementing the functionality of certain implementations may be stored by file storage subsystem 626 in the storage subsystem 624 , or in other machines accessible by the processor(s) 614 .
  • Bus subsystem 612 provides a mechanism for letting the various components and subsystems of computing device 610 communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.
  • Computing device 610 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 610 depicted in FIG. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 610 are possible having more or fewer components than the computing device depicted in FIG. 6 .
  • the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and/or how to receive content from the content server that may be more relevant to the user.
  • user information e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location
  • certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed.
  • a user's identity may be treated so that no personal identifiable information can be determined for the user, or a user's geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined.
  • geographic location information such as to a city, ZIP code, or state level
  • the user may have control over how information is collected about the user and/or used.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Library & Information Science (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)
US15/157,297 2016-05-17 2016-05-17 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user Active 2038-12-31 US10783178B2 (en)

Priority Applications (12)

Application Number Priority Date Filing Date Title
US15/157,297 US10783178B2 (en) 2016-05-17 2016-05-17 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
EP16826623.7A EP3458975A1 (en) 2016-05-17 2016-12-27 Generating natural language output based on natural language user interface input
CN201680085778.4A CN109564578B (zh) 2016-05-17 2016-12-27 基于自然语言用户界面输入生成自然语言输出
PCT/US2016/068737 WO2017200588A1 (en) 2016-05-17 2016-12-27 Generating natural language output based on natural language user interface input
KR1020187036265A KR102196976B1 (ko) 2016-05-17 2016-12-27 자연 언어 사용자 인터페이스 입력에 기초하여 자연 출력을 생성하는 방법 및 장치
JP2018560513A JP6978438B2 (ja) 2016-05-17 2016-12-27 自然言語ユーザインターフェース入力に基づいて自然言語出力を生成すること
KR1020207036980A KR102313475B1 (ko) 2016-05-17 2016-12-27 자연 언어 사용자 인터페이스 입력에 기초하여 자연 언어 출력을 생성하는 방법 및 장치
CN202311109978.7A CN117235335A (zh) 2016-05-17 2016-12-27 基于自然语言用户界面输入生成自然语言输出
US17/026,572 US11494427B2 (en) 2016-05-17 2020-09-21 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
JP2021017300A JP7222005B2 (ja) 2016-05-17 2021-02-05 自然言語ユーザインターフェース入力に基づいて自然言語出力を生成すること
US17/964,343 US11907276B2 (en) 2016-05-17 2022-10-12 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US18/413,846 US20240152537A1 (en) 2016-05-17 2024-01-16 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/157,297 US10783178B2 (en) 2016-05-17 2016-05-17 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/026,572 Continuation US11494427B2 (en) 2016-05-17 2020-09-21 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Publications (2)

Publication Number Publication Date
US20170337265A1 US20170337265A1 (en) 2017-11-23
US10783178B2 true US10783178B2 (en) 2020-09-22

Family

ID=57799898

Family Applications (4)

Application Number Title Priority Date Filing Date
US15/157,297 Active 2038-12-31 US10783178B2 (en) 2016-05-17 2016-05-17 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US17/026,572 Active 2036-06-12 US11494427B2 (en) 2016-05-17 2020-09-21 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US17/964,343 Active US11907276B2 (en) 2016-05-17 2022-10-12 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US18/413,846 Pending US20240152537A1 (en) 2016-05-17 2024-01-16 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Family Applications After (3)

Application Number Title Priority Date Filing Date
US17/026,572 Active 2036-06-12 US11494427B2 (en) 2016-05-17 2020-09-21 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US17/964,343 Active US11907276B2 (en) 2016-05-17 2022-10-12 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US18/413,846 Pending US20240152537A1 (en) 2016-05-17 2024-01-16 Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Country Status (6)

Country Link
US (4) US10783178B2 (zh)
EP (1) EP3458975A1 (zh)
JP (2) JP6978438B2 (zh)
KR (2) KR102313475B1 (zh)
CN (2) CN117235335A (zh)
WO (1) WO2017200588A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200365138A1 (en) * 2019-05-16 2020-11-19 Samsung Electronics Co., Ltd. Method and device for providing voice recognition service
US20220005481A1 (en) * 2018-11-28 2022-01-06 Samsung Electronics Co., Ltd. Voice recognition device and method
US11302331B2 (en) 2019-01-23 2022-04-12 Samsung Electronics Co., Ltd. Method and device for speech recognition
US11907276B2 (en) 2016-05-17 2024-02-20 Google Llc Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10515076B1 (en) * 2013-04-12 2019-12-24 Google Llc Generating query answers from a user's history
US11042505B2 (en) * 2018-04-16 2021-06-22 Microsoft Technology Licensing, Llc Identification, extraction and transformation of contextually relevant content
WO2020231453A1 (en) * 2019-05-13 2020-11-19 Google Llc Automatic evaluation of natural language text generated based on structured data
US11328018B2 (en) 2019-08-26 2022-05-10 Wizergos Software Solutions Private Limited System and method for state dependency based task execution and natural language response generation
US11593074B2 (en) * 2019-11-14 2023-02-28 BizFlow Corporation System, method, and apparatus for data-centric networked application development services
US20220253540A1 (en) * 2021-02-05 2022-08-11 Microsoft Technology Licensing, Llc Secure storage and processing of data for generating training data
US12045279B2 (en) 2021-11-30 2024-07-23 Microsoft Technology Licensing, Llc Method and system of content retrieval for visual data
US20230325391A1 (en) * 2022-04-08 2023-10-12 Microsoft Technology Licensing, Llc Method and system of retrieving assets from personalized asset libraries

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11272710A (ja) 1998-03-20 1999-10-08 Omron Corp 情報検索システム、情報検索方法および記録媒体
KR20090063092A (ko) 2007-12-12 2009-06-17 한국전자통신연구원 메타데이터 기반 색인 및 검색 장치와 그 방법
US20090198671A1 (en) * 2008-02-05 2009-08-06 Yahoo! Inc. System and method for generating subphrase queries
JP2009205552A (ja) 2008-02-28 2009-09-10 Internatl Business Mach Corp <Ibm> パーソナル・サービス支援のための方法、システム、および装置
US7685144B1 (en) * 2005-12-29 2010-03-23 Google Inc. Dynamically autocompleting a data entry
US20120016678A1 (en) * 2010-01-18 2012-01-19 Apple Inc. Intelligent Automated Assistant
US20130275164A1 (en) * 2010-01-18 2013-10-17 Apple Inc. Intelligent Automated Assistant
US20140163978A1 (en) * 2012-12-11 2014-06-12 Amazon Technologies, Inc. Speech recognition power management
US20140236986A1 (en) * 2013-02-21 2014-08-21 Apple Inc. Natural language document search
US20150066479A1 (en) 2012-04-20 2015-03-05 Maluuba Inc. Conversational agent
US20150100562A1 (en) * 2013-10-07 2015-04-09 Microsoft Corporation Contextual insights and exploration
US20150254247A1 (en) * 2014-03-07 2015-09-10 International Business Machines Corporation Natural language searching with cognitive modeling
US20150373183A1 (en) 2014-06-19 2015-12-24 Microsoft Corporation Use of a digital assistant in communications
US20160203002A1 (en) * 2015-01-09 2016-07-14 Microsoft Technology Licensing, Llc Headless task completion within digital personal assistants
US20160225370A1 (en) * 2015-01-30 2016-08-04 Microsoft Technology Licensing, Llc Updating language understanding classifier models for a digital personal assistant based on crowd-sourcing
US20160260433A1 (en) * 2015-03-06 2016-09-08 Apple Inc. Structured dictation using intelligent automated assistants
US20170068423A1 (en) * 2015-09-08 2017-03-09 Apple Inc. Intelligent automated assistant in a media environment
US20170091612A1 (en) * 2015-09-30 2017-03-30 Apple Inc. Proactive assistant with memory assistance

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8126881B1 (en) * 2007-12-12 2012-02-28 Vast.com, Inc. Predictive conversion systems and methods
US8145482B2 (en) 2008-05-25 2012-03-27 Ezra Daya Enhancing analysis of test key phrases from acoustic sources with key phrase training models
RU2542937C2 (ru) 2011-09-30 2015-02-27 Эпл Инк. Использование контекстной информации для облегчения обработки команд в виртуальном помощнике
US9547647B2 (en) * 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
US10031968B2 (en) * 2012-10-11 2018-07-24 Veveo, Inc. Method for adaptive conversation state management with filtering operators applied dynamically as part of a conversational interface
DE112014002747T5 (de) * 2013-06-09 2016-03-03 Apple Inc. Vorrichtung, Verfahren und grafische Benutzerschnittstelle zum Ermöglichen einer Konversationspersistenz über zwei oder mehr Instanzen eines digitalen Assistenten
US11048736B2 (en) * 2013-12-05 2021-06-29 Lenovo (Singapore) Pte. Ltd. Filtering search results using smart tags
US9892208B2 (en) 2014-04-02 2018-02-13 Microsoft Technology Licensing, Llc Entity and attribute resolution in conversational applications
US20160019316A1 (en) * 2014-07-21 2016-01-21 Splunk Inc. Wizard for creating a correlation search
US20160063874A1 (en) * 2014-08-28 2016-03-03 Microsoft Corporation Emotionally intelligent systems
US10719524B1 (en) * 2015-04-15 2020-07-21 Arimo, LLC Query template based architecture for processing natural language queries for data analysis
US10783178B2 (en) 2016-05-17 2020-09-22 Google Llc Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11272710A (ja) 1998-03-20 1999-10-08 Omron Corp 情報検索システム、情報検索方法および記録媒体
US7685144B1 (en) * 2005-12-29 2010-03-23 Google Inc. Dynamically autocompleting a data entry
KR20090063092A (ko) 2007-12-12 2009-06-17 한국전자통신연구원 메타데이터 기반 색인 및 검색 장치와 그 방법
US20090198671A1 (en) * 2008-02-05 2009-08-06 Yahoo! Inc. System and method for generating subphrase queries
JP2009205552A (ja) 2008-02-28 2009-09-10 Internatl Business Mach Corp <Ibm> パーソナル・サービス支援のための方法、システム、および装置
US20120016678A1 (en) * 2010-01-18 2012-01-19 Apple Inc. Intelligent Automated Assistant
US20130275164A1 (en) * 2010-01-18 2013-10-17 Apple Inc. Intelligent Automated Assistant
US20150066479A1 (en) 2012-04-20 2015-03-05 Maluuba Inc. Conversational agent
US20140163978A1 (en) * 2012-12-11 2014-06-12 Amazon Technologies, Inc. Speech recognition power management
US20140236986A1 (en) * 2013-02-21 2014-08-21 Apple Inc. Natural language document search
US20150100562A1 (en) * 2013-10-07 2015-04-09 Microsoft Corporation Contextual insights and exploration
US20150254247A1 (en) * 2014-03-07 2015-09-10 International Business Machines Corporation Natural language searching with cognitive modeling
US20150373183A1 (en) 2014-06-19 2015-12-24 Microsoft Corporation Use of a digital assistant in communications
US20160203002A1 (en) * 2015-01-09 2016-07-14 Microsoft Technology Licensing, Llc Headless task completion within digital personal assistants
US20160225370A1 (en) * 2015-01-30 2016-08-04 Microsoft Technology Licensing, Llc Updating language understanding classifier models for a digital personal assistant based on crowd-sourcing
US20160260433A1 (en) * 2015-03-06 2016-09-08 Apple Inc. Structured dictation using intelligent automated assistants
US20170068423A1 (en) * 2015-09-08 2017-03-09 Apple Inc. Intelligent automated assistant in a media environment
US20170091612A1 (en) * 2015-09-30 2017-03-30 Apple Inc. Proactive assistant with memory assistance

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
European Patent Office; Communication Pursuant to Article 94(3) EPC issue in Application No. 16826623.7; dated Mar. 4, 2020.
European Patent Office; International Preliminary Report on Patentability of International Application No. PCT/US2016/068737 dated Jun. 27, 2018.
European Patent Office; Written Opinion of the International Preliminary Examining Authority of International Application No. PCT/US2016/068737; 10 pages; dated Mar. 29, 2018.
International Search Report and Written Opinion of PCT Ser. No. PCT/US16/68737 dated Mar. 13, 2017.
Japanese Patent Office; Office Action, Application No. JP2018-560513, 10 pages, dated Dec. 2, 2019.
The Korean Intellectual Property Office; Office Action issued in Application No. 10-2018-7036265 dated Feb. 19, 2020.

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11907276B2 (en) 2016-05-17 2024-02-20 Google Llc Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US20220005481A1 (en) * 2018-11-28 2022-01-06 Samsung Electronics Co., Ltd. Voice recognition device and method
US11961522B2 (en) * 2018-11-28 2024-04-16 Samsung Electronics Co., Ltd. Voice recognition device and method
US11302331B2 (en) 2019-01-23 2022-04-12 Samsung Electronics Co., Ltd. Method and device for speech recognition
US20200365138A1 (en) * 2019-05-16 2020-11-19 Samsung Electronics Co., Ltd. Method and device for providing voice recognition service
US11605374B2 (en) * 2019-05-16 2023-03-14 Samsung Electronics Co., Ltd. Method and device for providing voice recognition service

Also Published As

Publication number Publication date
KR20200145860A (ko) 2020-12-30
US20230034812A1 (en) 2023-02-02
EP3458975A1 (en) 2019-03-27
CN109564578A (zh) 2019-04-02
JP6978438B2 (ja) 2021-12-08
US20210004394A1 (en) 2021-01-07
JP2019519037A (ja) 2019-07-04
US20170337265A1 (en) 2017-11-23
US11907276B2 (en) 2024-02-20
US11494427B2 (en) 2022-11-08
KR20190008331A (ko) 2019-01-23
WO2017200588A1 (en) 2017-11-23
KR102313475B1 (ko) 2021-10-15
CN117235335A (zh) 2023-12-15
JP2021093173A (ja) 2021-06-17
US20240152537A1 (en) 2024-05-09
CN109564578B (zh) 2023-09-15
KR102196976B1 (ko) 2020-12-30
JP7222005B2 (ja) 2023-02-14

Similar Documents

Publication Publication Date Title
US11907276B2 (en) Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user
US11960543B2 (en) Providing suggestions for interaction with an automated assistant in a multi-user message exchange thread
US20220262360A1 (en) Providing prompt in an automated dialog session based on selected content of prior automated dialog session
US11017037B2 (en) Obtaining responsive information from multiple corpora
US11971935B2 (en) Autonomously providing search results post-facto, including in assistant context
US11238242B2 (en) Generating output for presentation in response to user interface input, where the input and/or the output include chatspeak
US20150358447A1 (en) Populating user contact entries
US9805142B2 (en) Ranking suggestions based on user attributes
GB2556194A (en) Autonomously providing search results post-facto, including in conversational assistant context

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: GOOGLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GARRETT, MARYAM;QUAH, WAN;HORLING, BRYAN;AND OTHERS;SIGNING DATES FROM 20160517 TO 20160518;REEL/FRAME:039345/0761

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044567/0001

Effective date: 20170929

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4