US11178280B2 - Input during conversational session - Google Patents

Input during conversational session Download PDF

Info

Publication number
US11178280B2
US11178280B2 US15/628,080 US201715628080A US11178280B2 US 11178280 B2 US11178280 B2 US 11178280B2 US 201715628080 A US201715628080 A US 201715628080A US 11178280 B2 US11178280 B2 US 11178280B2
Authority
US
United States
Prior art keywords
user
input
detecting
conversational session
information handling
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US15/628,080
Other languages
English (en)
Other versions
US20180367669A1 (en
Inventor
John Weldon Nicholson
Daryl Cromer
Ming Qian
Song Wang
David Alexander Schwarz
Aaron Michael Stewart
Lincoln Penn Hancock
Jonathan Jen-Wei Yu
Jeffrey E. Skinner
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lenovo PC International Ltd
Lenovo Singapore Pte Ltd
Original Assignee
Lenovo Singapore Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lenovo Singapore Pte Ltd filed Critical Lenovo Singapore Pte Ltd
Priority to US15/628,080 priority Critical patent/US11178280B2/en
Assigned to LENOVO (SINGAPORE) PTE. LTD. reassignment LENOVO (SINGAPORE) PTE. LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: QIAN, MING, Schwarz, David Alexander, WANG, SONG, CROMER, DARYL, HANCOCK, LINCOLN PENN, NICHOLSON, JOHN WELDON, SKINNER, JEFFREY E., STEWART, AARON MICHAEL, YU, JONATHAN JEN-WEI
Priority to CN201810315982.1A priority patent/CN109101517B/zh
Priority to DE102018114658.2A priority patent/DE102018114658A1/de
Publication of US20180367669A1 publication Critical patent/US20180367669A1/en
Application granted granted Critical
Publication of US11178280B2 publication Critical patent/US11178280B2/en
Assigned to LENOVO PC INTERNATIONAL LIMITED reassignment LENOVO PC INTERNATIONAL LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LENOVO (SINGAPORE) PTE LTD
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/453—Help systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/487—Arrangements for providing information services, e.g. recorded voice services or time announcements
    • H04M3/493—Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
    • H04M3/4936—Speech interaction details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L15/222—Barge in, i.e. overridable guidance for interrupting prompts
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4812—Task transfer initiation or dispatching by interrupt, e.g. masked
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223—Execution procedure of a spoken command
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M2203/00—Aspects of automatic or semi-automatic exchanges
    • H04M2203/20—Aspects of automatic or semi-automatic exchanges related to features of supplementary services
    • H04M2203/2027—Live party detection

Definitions

  • Information handling devices for example smart phones, tablet devices, smart speakers, laptop and personal computers, and the like, may be capable of receiving command inputs and providing outputs responsive to the inputs.
  • a user interacts with a voice input module, for example embodied in a personal assistant through use of natural language.
  • a voice input module for example embodied in a personal assistant through use of natural language.
  • This style of interface allows a device to receive voice inputs from a user (e.g., queries, commands, etc.), process those inputs, and provide output responsive to the input.
  • one aspect provides a method, comprising: engaging, at an information handing device, in a conversational session with a user; receiving an input from a source other than the user during the conversational session; and performing, at the information handling device, an action related to the conversational session in response to the received input.
  • an information handling device comprising: a processor; a memory device that stores instructions executable by the processor to: engage, at an information handing device, in a conversational session with a user; receive an input from a source other than the user during the conversational session; and perform, at the information handling device, an action related to the conversational session in response to the received input.
  • a further aspect provides a product, comprising: a storage device that stores code, the code being executable by a processor and comprising: code that engages in a conversational session with a user; code that receives an input from a source other than the user during the conversational session; and code that performs an action related to the conversational session in response to the received input.
  • FIG. 1 illustrates an example of information handling device circuitry.
  • FIG. 2 illustrates another example of information handling device circuitry.
  • FIG. 3 illustrates an example method of recognizing and responding to an input received during a conversational session.
  • Digital assistants are able to provide outputs (e.g., audible outputs, visual outputs, etc.) that are responsive to a variety of different types of user inputs (e.g., voice inputs, etc.).
  • a conversational session may include a single exchange, referred to herein as a single-turn conversation, where the user provides an input to the device and the device performs the action.
  • the device can process and complete the function associated with the input with only a single request or input from the user.
  • the user may ask “What song is currently playing?”
  • the assistant can provide the response to the question without any additional information from the user.
  • the conversational session may include multiple exchanges, referred to herein as a multi-turn conversation, where the user provides an input to the device, the device provides an output requiring additional user input, the user provides the additional input, and so on until the exchange is complete.
  • the device requires additional user input in order to process and complete the function associated with the initial input received from the user.
  • the user may provide the command “Order pizza.”
  • the device then needs to ask questions in response to the command in order to completely fulfill the command, for example, the device may ask “where did you want to order the pizza from?”, “what size pizza did you want to order?”, “what kinds of toppings do you want on your pizza?”, and the like.
  • the device will not recognize inputs or interruptions either associated with the user or with the device. For example, if a user is providing input to the device and someone else walks in the room and asks a question of the user, the digital assistant will attempt to process both the input received by the user and the additional input provided by the additional person. This not only results in non-responsive output by the digital assistant, but also causes the user to have to correct the assistant and provide the input again, which can be inconvenient, non-intuitive, and confusing. First, the device is unable to determine that in interruption has occurred. In other words, the device is unable to determine that the received input was not actually directed at the device. Additionally, the device is unable to process an interruption in a human-like way, for example, recognizing the interruption and determining that the interrupting input is not directed at the assistant.
  • an embodiment provides a method for recognizing and responding to an input received from a source other than the user during a conversational session.
  • An embodiment may engage in a conversational session with a user.
  • An embodiment may then receive an input from a source other than the user during the conversational session.
  • an embodiment may detect an interruption of the conversational session by another person or other source.
  • detecting the interruption may include identifying that another person has entered the room.
  • Detecting the interruption may also include determining that input was received from another user in the room.
  • an embodiment may use natural language processing and understanding to determine that the input is unrelated to the conversational session.
  • an embodiment may perform some action related to the conversational session in response to the received input.
  • an embodiment may pause the conversational session, lower the volume of output, save the conversational session, and the like.
  • Such a method assists a user in conducting a more human-like conversation with the assistant by allowing the device to recognize and respond to interruptions in a manner that is similar to how people would recognize and respond to interruptions.
  • FIG. 1 includes a system on a chip design found for example in tablet or other mobile computing platforms.
  • Software and processor(s) are combined in a single chip 110 .
  • Processors comprise internal arithmetic units, registers, cache memory, busses, I/O ports, etc., as is well known in the art. Internal busses and the like depend on different vendors, but essentially all the peripheral devices ( 120 ) may attach to a single chip 110 .
  • the circuitry 100 combines the processor, memory control, and I/O controller hub all into a single chip 110 .
  • systems 100 of this type do not typically use SATA or PCI or LPC. Common interfaces, for example, include SDIO and I2C.
  • power management chip(s) 130 e.g., a battery management unit, BMU, which manage power as supplied, for example, via a rechargeable battery 140 , which may be recharged by a connection to a power source (not shown).
  • BMU battery management unit
  • a single chip, such as 110 is used to supply BIOS like functionality and DRAM memory.
  • System 100 typically includes one or more of a WWAN transceiver 150 and a WLAN transceiver 160 for connecting to various networks, such as telecommunications networks and wireless Internet devices, e.g., access points. Additionally, devices 120 are commonly included, e.g., an image sensor such as a camera, audio capture device such as a microphone, a thermal sensor, etc. System 100 often includes a touch screen 170 for data input and display/rendering. System 100 also typically includes various memory devices, for example flash memory 180 and SDRAM 190 .
  • FIG. 2 depicts a block diagram of another example of information handling device circuits, circuitry or components.
  • the example depicted in FIG. 2 may correspond to computing systems such as the THINKPAD series of personal computers sold by Lenovo (US) Inc. of Morrisville, N.C., or other devices.
  • embodiments may include other features or only some of the features of the example illustrated in FIG. 2 .
  • FIG. 2 includes a so-called chipset 210 (a group of integrated circuits, or chips, that work together, chipsets) with an architecture that may vary depending on manufacturer (for example, INTEL, AMD, ARM, etc.).
  • INTEL is a registered trademark of Intel Corporation in the United States and other countries.
  • AMD is a registered trademark of Advanced Micro Devices, Inc. in the United States and other countries.
  • ARM is an unregistered trademark of ARM Holdings plc in the United States and other countries.
  • the architecture of the chipset 210 includes a core and memory control group 220 and an I/O controller hub 250 that exchanges information (for example, data, signals, commands, etc.) via a direct management interface (DMI) 242 or a link controller 244 .
  • DMI direct management interface
  • the DMI 242 is a chip-to-chip interface (sometimes referred to as being a link between a “northbridge” and a “southbridge”).
  • the core and memory control group 220 include one or more processors 222 (for example, single or multi-core) and a memory controller hub 226 that exchange information via a front side bus (FSB) 224 ; noting that components of the group 220 may be integrated in a chip that supplants the conventional “northbridge” style architecture.
  • processors 222 comprise internal arithmetic units, registers, cache memory, busses, I/O ports, etc., as is well known in the art.
  • the memory controller hub 226 interfaces with memory 240 (for example, to provide support for a type of RAM that may be referred to as “system memory” or “memory”).
  • the memory controller hub 226 further includes a low voltage differential signaling (LVDS) interface 232 for a display device 292 (for example, a CRT, a flat panel, touch screen, etc.).
  • a block 238 includes some technologies that may be supported via the LVDS interface 232 (for example, serial digital video, HDMI/DVI, display port).
  • the memory controller hub 226 also includes a PCI-express interface (PCI-E) 234 that may support discrete graphics 236 .
  • PCI-E PCI-express interface
  • the I/O hub controller 250 includes a SATA interface 251 (for example, for HDDs, SDDs, etc., 280 ), a PCI-E interface 252 (for example, for wireless connections 282 ), a USB interface 253 (for example, for devices 284 such as a digitizer, keyboard, mice, cameras, phones, microphones, storage, other connected devices, etc.), a network interface 254 (for example, LAN), a GPIO interface 255 , a LPC interface 270 (for ASICs 271 , a TPM 272 , a super I/O 273 , a firmware hub 274 , BIOS support 275 as well as various types of memory 276 such as ROM 277 , Flash 278 , and NVRAM 279 ), a power management interface 261 , a clock generator interface 262 , an audio interface 263 (for example, for speakers 294 ), a TCO interface 264 , a system management bus interface 265 , and
  • the system upon power on, may be configured to execute boot code 290 for the BIOS 268 , as stored within the SPI Flash 266 , and thereafter processes data under the control of one or more operating systems and application software (for example, stored in system memory 240 ).
  • An operating system may be stored in any of a variety of locations and accessed, for example, according to instructions of the BIOS 268 .
  • a device may include fewer or more features than shown in the system of FIG. 2 .
  • Information handling device circuitry may be used in devices such as tablets, smart phones, smart speakers, personal computer devices generally, and/or electronic devices which may include digital assistants that a user may interact with and that may perform various functions responsive to receiving user input.
  • the circuitry outlined in FIG. 1 may be implemented in a tablet or smart phone embodiment
  • the circuitry outlined in FIG. 2 may be implemented in a personal computer embodiment.
  • an embodiment may engage in a conversational session with a user.
  • a conversational session may be defined as a session with a digital assistant or other interactive application in which a user provides input, the digital assistant processes or analyzes the input, and the digital assistant then provides an output responsive to the input.
  • a conversational session may include a single exchange of input and output, referred to herein as a single-turn conversational session, or multiple exchanges of input and output, referred to herein as a multi-turn interaction session.
  • Engaging in the conversational session may include any point during the conversational session, for example, starting the conversational session, processing input received from a user, providing output, and the like.
  • engaging in the conversational session may include receiving an indication to begin a conversational session.
  • the indication to begin a conversational session may be associated with user-provided input.
  • the user-provided input indication may be a wakeup input or action provided by a user (e.g., one or more wakeup words or predetermined commands, a depression of a button for a predetermined length of time, a selection of a digital assistant icon, etc.).
  • the wakeup action may be provided prior to or in conjunction with user input.
  • a user may provide the vocal input, “Ok Surlexana, order a pizza.”
  • “Ok Surlexana” is the wakeup word and upon identification of the wakeup word an embodiment may prime the system to listen for additional user input. Responsive to the identification of the wakeup action, an embodiment may initiate a conversational session.
  • the indication may not be associated with a wakeup action. For example, the system may simply “listen” to the user and determine when the user is providing input directed at the system. The conversational session may then be initiated when the system determines that the user input is directed to the system.
  • engaging in the conversational session may include receiving a query from a user.
  • the term query is used herein, however, it should be understood that a query may not necessarily include a question. Rather, the query may include a command or other input for processing by the digital assistant.
  • the query may be received immediately after the conversational session has been activated, for example, using the pizza example mentioned above, the query (“order a pizza”) is received after receipt of the activation command. Alternatively, the query may be received at another point in the conversational session, for example, in response to output provided by the digital assistant.
  • an embodiment may receive user input (e.g., voice input, touch input, etc.) including or associated with a user query or a user command, referred to herein as a query input, at a device (e.g., smart phone, smart speaker, tablet, laptop computer, etc.).
  • a device e.g., smart phone, smart speaker, tablet, laptop computer, etc.
  • the device may employ digital assistant software capable of receiving and processing user input and subsequently providing output (e.g., audible output, textual output, visual output, etc.) corresponding or responsive to the user input.
  • the user input may be any input that requests the digital assistant to provide a response.
  • the user may ask the digital assistant a general question about a topic, the user may ask the digital assistant to provide instructions to assemble an object, the user may ask the digital assistant's opinion on a topic, the user may make a statement which allows a response, and the like.
  • the input may be received at an input device (e.g., physical keyboard, on-screen keyboard, audio capture device, image capture device, video capture device, etc.) and may be provided by any known method of providing input to an electronic device (e.g., touch input, text input, voice input, etc.).
  • an input device e.g., physical keyboard, on-screen keyboard, audio capture device, image capture device, video capture device, etc.
  • voice input e.g., touch input, text input, voice input, etc.
  • voice input e.g., a microphone, a speech capture device, etc.
  • the user may provide text input to the digital assistant, for example, through a chat assistant or instant messaging application.
  • the input device may be an input device integral to the digital assistant device.
  • a smart phone may be disposed with a microphone capable of receiving voice input data.
  • the input device may be disposed on another device and may transmit received input data to the digital assistant device.
  • voice input may be received at a smart speaker that may subsequently transmit the voice data to another device (e.g., to a user's smartphone for processing, etc.).
  • Input data may be communicated from other sources to the digital assistant device via a wireless connection (e.g., using a BLUETOOTH connection, near field communication (NFC), wireless connection techniques, etc.), a wired connection (e.g., the device is coupled to another device or source, etc.), through a connected data storage system (e.g., via cloud storage, remote storage, local storage, network storage, etc.), and the like.
  • a wireless connection e.g., using a BLUETOOTH connection, near field communication (NFC), wireless connection techniques, etc.
  • NFC near field communication
  • wireless connection techniques e.g., the device is coupled to another device or source, etc.
  • a connected data storage system e.g., via cloud storage, remote storage, local storage, network storage, etc.
  • the input device may be configured to continuously receive input data by maintaining the input device in an active state.
  • the input device may, for example, continuously detect input data even when other sensors (e.g., cameras, light sensors, speakers, other microphones, etc.) associated with the digital assistant device are inactive.
  • the input device may remain in an active state for a predetermined amount of time (e.g., 30 minutes, 1 hour, 2 hours, etc.). Subsequent to not receiving any input data during this predetermined time window, an embodiment may switch the input device to a power off state.
  • the predetermined time window may be preconfigured by a manufacturer or, alternatively, may be configured and set by one or more users.
  • Engagement in the conversational session may also include an action by the digital assistant.
  • engagement in the conversational session may include the digital assistant processing the input (e.g., accessing a database for an answer, parsing the input to identify the request, etc.), providing output responsive to the input (e.g., asking a successive question, providing audible output answering a provided question, providing audible output indicating a problem with responding, etc.), performing a function associated with the input (e.g., dimming the lights in response to a corresponding request, ordering a pizza in response to a corresponding request, providing a responsive output to a corresponding request, etc.), and the like.
  • engagement in the conversational session includes any point in which a conversational session is ongoing between the user and digital assistant, even if no input or output is being provided at that exact moment.
  • an embodiment may determine whether an input from a source other than the user has been received during the conversational session. For example, an embodiment may determine whether the conversational session has been interrupted by at least one other user or source. Receiving input from a source other than the user may include receiving input from a variety of sources or detecting the input using a variety of techniques. One embodiment may receive an input associated with another person entering the same room as the user. This may signal to an embodiment that an interruption of the conversational session may be imminent. Determining if another person has entered the room may be performed using one or more sensors or devices integral to or accessible by the device (e.g., image capture device, thermopile sensors, near field communication device, microphone, audio capture device, etc.).
  • one embodiment may use a thermopile sensor to detect a heat signature or heat source.
  • the device may determine that there is more than one heat source, one heat source has recently appeared, or the like, to determine that a person has entered the room or another person is present within the room.
  • the device may use heat signature techniques to distinguish between people, pets, plants, and other heat generating sources.
  • an embodiment may use an image capture device (e.g., camera, video recorder, infrared sensors, etc.) to detect that another person has entered the area.
  • One embodiment may use radio signatures/signals or near field communication signals (e.g., Bluetooth®, wireless network signals, MAC addresses, etc.) to determine that another person has entered the range of the device.
  • radio signatures/signals or near field communication signals e.g., Bluetooth®, wireless network signals, MAC addresses, etc.
  • an embodiment may detect wireless signals from a user's mobile device. Upon detection of a second wireless signal, an embodiment may determine that another person has entered the same space.
  • One embodiment may associate the radio signatures or near field communication signals to a particular user. For example, an embodiment may compare the signal to known signals and determine that the device belongs to User A.
  • an embodiment may compare the second signal to known signals and determine that the device belongs to User B. Alternatively, if the signals are not found to match a known signal an embodiment may assume that the signal belongs to another user. Identification of the user associated with a signal may allow an embodiment to distinguish whether it is the same user with more than one device or different users.
  • An embodiment may use sensors or devices in conjunction with other techniques to receive an input or detect an interruption.
  • an embodiment may use an audio capture device to detect input provided by one or more users. An embodiment may then use a technique to distinguish users or determine if the input is being provided to the device.
  • an embodiment may use voice identification or other techniques to identify the user providing the input and determine that the user providing the input is not the same user who was previously providing input.
  • an embodiment may use voice identification or other voice differentiation techniques to determine that input has been provided by more than one user.
  • the system may identify characteristics of the input (e.g., pitch, frequency, volume, etc.) and determine that the characteristics associated with each input do not match each other. Thus, the embodiment may determine that more than one user has provided input.
  • One embodiment may use sound source localization techniques to determine the location or direction of the user input. An embodiment may then use this information to determine if a new user has entered the room or provided the input. As an example, an embodiment may determine that the user providing the conversational session input is located at one spot in the room and the new input has been provided at a new location in the room.
  • Other techniques are possible and contemplated, for example, diarization techniques, signal processing techniques, blind source separation techniques, and the like.
  • One embodiment may use natural language processing or understanding techniques to process and analyze the input to determine if the input is likely being provided to the assistant or another user. For example, an embodiment may analyze the input to determine that the new input provided is unrelated to the first input, for example, the new input includes a different topic, the new input is non sequitur in relation to the first input, and the like. To make this determination that the new input is unrelated to the first input, an embodiment may use different speech or natural language processing techniques, for example, domain classification, intent mapping, dialog act recognition, and the like. An embodiment may use natural language processing or understand to determine that the new input is not input likely to be provided to a digital assistant.
  • an embodiment may determine that the input “Mom, what's for dinner?” is not a request that the digital assistant could process and provide output in response to.
  • an embodiment may understand that the term “Mom” is a term that indicates that the input is directed at an individual and not the digital assistant.
  • An embodiment may receive an input related to an event that may precede an interruption or give an indication that an interruption may be coming. For example, an embodiment may determine that notifications or communications may indicate that the conversational session may be interrupted. Examples of notifications or communications may include sound-based notifications other than on the device (e.g., doorbell, telephone ring, etc.), incoming communications or notifications on the device or coupled devices (e.g., text message, calendar notification, instant message, alarm notification, etc.), communications or notifications sent from another device (e.g., smart home doorbell, smart home thermostat notification, alarm, etc.), or the like. In other words, events may be detected by the device that may be associated with a possible interruption or that generally lead to an interruption. Any combination of the above techniques, sensors, and/or devices may be used by an embodiment to detect an interruption to the conversational session.
  • an embodiment may continue the conversational session at 304 in a traditional manner. If, however, the conversational session has been interrupted at 302 , an embodiment may perform an action related to the conversational session in response to the received input at 303 . In response to receipt of the input or detection of the interruption an embodiment may use the contextual information related to the input or interruption to perform an action. In other words, depending on the type or manner of input or interruption, an embodiment may perform different actions.
  • Possible actions include, but are not limited to, pausing or halting the conversational session, wait for the additional person to begin talking, adjusting a characteristic of the output (e.g., lowering the volume of provided output, speeding up the output, slowing the output, etc.), wait for the user to provide additional input (e.g., say “continue”, say “go on”, provide a resume gesture, etc.), save the current conversational session to be resumed later, and the like.
  • a characteristic of the output e.g., lowering the volume of provided output, speeding up the output, slowing the output, etc.
  • additional input e.g., say “continue”, say “go on”, provide a resume gesture, etc.
  • an embodiment may provide techniques for resuming the conversational session.
  • a user may request that the conversational session be resumed by providing input indicating the conversational session should be resumed, for example, providing the input “Where were we?”, providing the input “please resume”, or the like.
  • One embodiment may prompt the user to resume a session. For example, when the user is either alone or activates the device again, the device may indicate that a session has been saved and ask if it should be resumed.
  • One embodiment may save all the conversational sessions and maintain a list or database of the unfinished conversations.
  • Each session in the list or database may be associated with information for retrieval at a later time.
  • the session may include an indication of the topic of the conversation, a user associated with the conversational session, a time associated with the conversational session, and the like. This may allow an embodiment to either search for the conversational session based upon a request by a user to continue the session. Alternatively, an embodiment may prompt the user to continue the session once an embodiment identifies the user as having a previously saved conversational session.
  • the action performed may be chosen based upon the context of either the input or interruption or the context of the conversational session. For example, if the interruption is a short interruption, providing by a particular person, or the like, an embodiment may perform one action, whereas if the interruption is long, provided by a different person, or the like, an embodiment may perform a different action. As another example, if the conversational session is related to providing a weather forecast, an embodiment may perform one action, whereas if the conversational session is related to a reading a bank statement, an embodiment may perform a different action.
  • the action performed may be previously programmed, set by a user, based upon the user providing the input, or learned by the device. For example, the device may perform one action in response to a particular type of interruption for one user, and a different action in response to the same type of interruption for another user. Learning an action in response to an interruption may include the device associating an action with a particular context, interruption, or user. For example, the device may determine that a user prefers that one action be performed when an interruption is short and may therefore learn and store this action and perform it when the interruption is short. The action performed may also be determined based upon the emotional state of the user. For example, if the user is annoyed, an embodiment may determine that no output should be provided and the conversation should be saved for later. However, if the user is happy or neutral, an embodiment may provide output and indicate that the conversational session will be saved for later and then save the conversational session.
  • the various embodiments described herein thus represent a technical improvement to conventional communications with a digital assistant.
  • the digital assistant can recognize and respond to input and/or interruptions received during a conversational session in a manner similar to how a person would recognize and respond to an input or interruption.
  • the device can recognize that some input may not be directed at the device and may perform some action with regard to the conversational session in response to this extra input.
  • Such techniques provide a more intuitive digital assistant that can recognize and respond to an interruption of the conversational session.
  • aspects may be embodied as a system, method or device program product. Accordingly, aspects may take the form of an entirely hardware embodiment or an embodiment including software that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects may take the form of a device program product embodied in one or more device readable medium(s) having device readable program code embodied therewith.
  • a storage device may be, for example, a system, apparatus, or device (e.g., an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device) or any suitable combination of the foregoing.
  • a storage device/medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
  • a storage device is not a signal and “non-transitory” includes all media except signal media.
  • Program code embodied on a storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, et cetera, or any suitable combination of the foregoing.
  • Program code for carrying out operations may be written in any combination of one or more programming languages.
  • the program code may execute entirely on a single device, partly on a single device, as a stand-alone software package, partly on single device and partly on another device, or entirely on the other device.
  • the devices may be connected through any type of connection or network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made through other devices (for example, through the Internet using an Internet Service Provider), through wireless connections, e.g., near-field communication, or through a hard wire connection, such as over a USB connection.
  • LAN local area network
  • WAN wide area network
  • Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
  • Example embodiments are described herein with reference to the figures, which illustrate example methods, devices and program products according to various example embodiments. It will be understood that the actions and functionality may be implemented at least in part by program instructions. These program instructions may be provided to a processor of a device, a special purpose information handling device, or other programmable data processing device to produce a machine, such that the instructions, which execute via a processor of the device implement the functions/acts specified.

Landscapes

  • Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Acoustics & Sound (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • User Interface Of Digital Computer (AREA)
US15/628,080 2017-06-20 2017-06-20 Input during conversational session Active 2038-01-29 US11178280B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US15/628,080 US11178280B2 (en) 2017-06-20 2017-06-20 Input during conversational session
CN201810315982.1A CN109101517B (zh) 2017-06-20 2018-04-10 信息处理方法、信息处理设备以及介质
DE102018114658.2A DE102018114658A1 (de) 2017-06-20 2018-06-19 Eingabe während einer Unterhaltungssitzung

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/628,080 US11178280B2 (en) 2017-06-20 2017-06-20 Input during conversational session

Publications (2)

Publication Number Publication Date
US20180367669A1 US20180367669A1 (en) 2018-12-20
US11178280B2 true US11178280B2 (en) 2021-11-16

Family

ID=64457309

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/628,080 Active 2038-01-29 US11178280B2 (en) 2017-06-20 2017-06-20 Input during conversational session

Country Status (3)

Country Link
US (1) US11178280B2 (de)
CN (1) CN109101517B (de)
DE (1) DE102018114658A1 (de)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11373635B2 (en) * 2018-01-10 2022-06-28 Sony Corporation Information processing apparatus that fades system utterance in response to interruption
US10991369B1 (en) * 2018-01-31 2021-04-27 Progress Software Corporation Cognitive flow
CN111831795B (zh) * 2019-04-11 2023-10-27 北京猎户星空科技有限公司 多轮对话处理方法、装置、电子设备及存储介质
US11308958B2 (en) * 2020-02-07 2022-04-19 Sonos, Inc. Localized wakeword verification
US12002471B2 (en) * 2020-11-17 2024-06-04 Rovi Guides, Inc. Interruption detection and handling by digital assistants
WO2023140917A1 (en) * 2022-01-19 2023-07-27 Google Llc Dynamic adaptation of parameter set used in hot word free adaptation of automated assistant

Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040243281A1 (en) * 2002-03-15 2004-12-02 Masahiro Fujita Robot behavior control system, behavior control method, and robot device
CN1655233A (zh) 2004-01-29 2005-08-17 哈曼贝克自动系统股份有限公司 实现语音对话的方法及其语音对话系统
US20060036688A1 (en) 2004-08-12 2006-02-16 International Business Machines Corporation Method and system for managing interrupts in an instant messaging application
CN102054481A (zh) 2009-10-30 2011-05-11 大陆汽车有限责任公司 用于激活和/或进行语音对话的装置、系统和方法
WO2011133249A1 (en) 2010-04-19 2011-10-27 Qualcomm Incorporated System and method of providing voice updates
US20120161971A1 (en) * 2010-12-22 2012-06-28 Verizon Patent And Licensing, Inc. Automated attendance tracking and event notification
CN102945672A (zh) 2012-09-29 2013-02-27 深圳市国华识别科技开发有限公司 一种多媒体设备语音控制系统及方法
US20150099587A1 (en) * 2013-10-08 2015-04-09 Google Inc. Native gameplay experience across platforms
US20150213800A1 (en) * 2014-01-28 2015-07-30 Simple Emotion, Inc. Methods for adaptive voice interaction
US20150302867A1 (en) * 2014-04-17 2015-10-22 Arthur Charles Tomlin Conversation detection
CN105264524A (zh) 2013-06-09 2016-01-20 苹果公司 用于实现跨数字助理的两个或更多个实例的会话持续性的设备、方法、和图形用户界面
EP3005650A1 (de) 2013-06-04 2016-04-13 Google, Inc. Aufrechterhaltung einer videokonferenzsitzungskontinuität während der übertragung von sitzungen auf eine alternative vorrichtung
US20170289766A1 (en) * 2016-03-29 2017-10-05 Microsoft Technology Licensing, Llc Digital Assistant Experience based on Presence Detection
US20170358302A1 (en) * 2016-06-08 2017-12-14 Apple Inc. Intelligent automated assistant for media exploration
US20180074785A1 (en) * 2015-03-31 2018-03-15 Sony Corporation Information processing device, control method, and program
US20180190264A1 (en) * 2016-12-30 2018-07-05 Google Llc Conversation-Aware Proactive Notifications for a Voice Interface Device
US20180233139A1 (en) * 2017-02-14 2018-08-16 Microsoft Technology Licensing, Llc Intelligent digital assistant system
US20180336904A1 (en) * 2017-05-16 2018-11-22 Apple Inc. Providing an auditory-based interface of a digital assistant
US10170116B1 (en) * 2016-09-21 2019-01-01 Amazon Technologies, Inc. Maintaining context for voice processes

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8473949B2 (en) * 2010-07-08 2013-06-25 Microsoft Corporation Methods for supporting users with task continuity and completion across devices and time
US9368114B2 (en) * 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
WO2014168730A2 (en) * 2013-03-15 2014-10-16 Apple Inc. Context-sensitive handling of interruptions
CN105957521B (zh) * 2016-02-29 2020-07-10 青岛克路德机器人有限公司 一种用于机器人的语音和图像复合交互执行方法及系统

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040243281A1 (en) * 2002-03-15 2004-12-02 Masahiro Fujita Robot behavior control system, behavior control method, and robot device
CN1655233A (zh) 2004-01-29 2005-08-17 哈曼贝克自动系统股份有限公司 实现语音对话的方法及其语音对话系统
US20060036688A1 (en) 2004-08-12 2006-02-16 International Business Machines Corporation Method and system for managing interrupts in an instant messaging application
CN102054481A (zh) 2009-10-30 2011-05-11 大陆汽车有限责任公司 用于激活和/或进行语音对话的装置、系统和方法
WO2011133249A1 (en) 2010-04-19 2011-10-27 Qualcomm Incorporated System and method of providing voice updates
US20120161971A1 (en) * 2010-12-22 2012-06-28 Verizon Patent And Licensing, Inc. Automated attendance tracking and event notification
CN102945672A (zh) 2012-09-29 2013-02-27 深圳市国华识别科技开发有限公司 一种多媒体设备语音控制系统及方法
EP3005650A1 (de) 2013-06-04 2016-04-13 Google, Inc. Aufrechterhaltung einer videokonferenzsitzungskontinuität während der übertragung von sitzungen auf eine alternative vorrichtung
CN105264524A (zh) 2013-06-09 2016-01-20 苹果公司 用于实现跨数字助理的两个或更多个实例的会话持续性的设备、方法、和图形用户界面
US20150099587A1 (en) * 2013-10-08 2015-04-09 Google Inc. Native gameplay experience across platforms
US20150213800A1 (en) * 2014-01-28 2015-07-30 Simple Emotion, Inc. Methods for adaptive voice interaction
US20150302867A1 (en) * 2014-04-17 2015-10-22 Arthur Charles Tomlin Conversation detection
US20180074785A1 (en) * 2015-03-31 2018-03-15 Sony Corporation Information processing device, control method, and program
US20170289766A1 (en) * 2016-03-29 2017-10-05 Microsoft Technology Licensing, Llc Digital Assistant Experience based on Presence Detection
US20170358302A1 (en) * 2016-06-08 2017-12-14 Apple Inc. Intelligent automated assistant for media exploration
US10170116B1 (en) * 2016-09-21 2019-01-01 Amazon Technologies, Inc. Maintaining context for voice processes
US20180190264A1 (en) * 2016-12-30 2018-07-05 Google Llc Conversation-Aware Proactive Notifications for a Voice Interface Device
US20180233139A1 (en) * 2017-02-14 2018-08-16 Microsoft Technology Licensing, Llc Intelligent digital assistant system
US20180336904A1 (en) * 2017-05-16 2018-11-22 Apple Inc. Providing an auditory-based interface of a digital assistant

Also Published As

Publication number Publication date
DE102018114658A1 (de) 2018-12-20
CN109101517A (zh) 2018-12-28
US20180367669A1 (en) 2018-12-20
CN109101517B (zh) 2022-11-04

Similar Documents

Publication Publication Date Title
US10204624B1 (en) False positive wake word
US11178280B2 (en) Input during conversational session
US9990921B2 (en) User focus activated voice recognition
US9940929B2 (en) Extending the period of voice recognition
GB2534274A (en) Gaze triggered voice recognition
US20180364798A1 (en) Interactive sessions
US11282528B2 (en) Digital assistant activation based on wake word association
US11558848B2 (en) Intelligent notification delivery
EP3324289B1 (de) Kontextueller konversationsmodus für digitalen assistenten
US20190050195A1 (en) Output provision based on gaze detection
US10257363B2 (en) Coordinating input on multiple local devices
US10936276B2 (en) Confidential information concealment
US20210005189A1 (en) Digital assistant device command performance based on category
US11423893B2 (en) Response to secondary inputs at a digital personal assistant
US20190019505A1 (en) Sustaining conversational session
US20190065608A1 (en) Query input received at more than one device
US10614794B2 (en) Adjust output characteristic
US11302322B2 (en) Ignoring command sources at a digital assistant
US20190034554A1 (en) Extend conversational session waiting time
US20180364809A1 (en) Perform function during interactive session
US10847163B2 (en) Provide output reponsive to proximate user input
US20180343233A1 (en) Contextual name association
US11048782B2 (en) User identification notification for non-personal device
US11748415B2 (en) Digital assistant output attribute modification
US10897788B2 (en) Wireless connection establishment between devices

Legal Events

Date Code Title Description
AS Assignment

Owner name: LENOVO (SINGAPORE) PTE. LTD., SINGAPORE

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NICHOLSON, JOHN WELDON;CROMER, DARYL;QIAN, MING;AND OTHERS;SIGNING DATES FROM 20170612 TO 20170613;REEL/FRAME:042759/0331

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: LENOVO PC INTERNATIONAL LIMITED, HONG KONG

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LENOVO (SINGAPORE) PTE LTD;REEL/FRAME:060638/0067

Effective date: 20130401