WO2019231537A1 - Opération d'assistant virtuel dans des environnements multi-dispositifs - Google Patents

Opération d'assistant virtuel dans des environnements multi-dispositifs Download PDF

Info

Publication number
WO2019231537A1
WO2019231537A1 PCT/US2019/024332 US2019024332W WO2019231537A1 WO 2019231537 A1 WO2019231537 A1 WO 2019231537A1 US 2019024332 W US2019024332 W US 2019024332W WO 2019231537 A1 WO2019231537 A1 WO 2019231537A1
Authority
WO
WIPO (PCT)
Prior art keywords
electronic device
speech input
user
task
accordance
Prior art date
Application number
PCT/US2019/024332
Other languages
English (en)
Inventor
Brandon J. Newendorp
Anumita Biswas
Gagan A. GUPTA
Benjamin S. Phipps
Kisun You
Original Assignee
Apple Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from DKPA201870355A external-priority patent/DK201870355A1/en
Application filed by Apple Inc. filed Critical Apple Inc.
Publication of WO2019231537A1 publication Critical patent/WO2019231537A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Definitions

  • application 236-1 includes event sorter 270.
  • event sorter 270 is a stand-alone module, or a part of another module stored in memory 202, such as contact/motion module 230.
  • natural language processing module 732 is implemented using one or more machine learning mechanisms (e.g., neural networks).
  • the one or more machine learning mechanisms are configured to receive a candidate text representation and contextual information associated with the candidate text representation. Based on the candidate text representation and the associated contextual information, the one or more machine learning mechanisms are configured to determine intent confidence scores over a set of candidate actionable intents.
  • Natural language processing module 732 can select one or more candidate actionable intents from the set of candidate actionable intents based on the determined intent confidence scores.
  • an ontology e.g., ontology 760 is also used to select the one or more candidate actionable intents from the set of candidate actionable intents.
  • first electronic device 800 displays an animation or image indicating the virtual assistant operating on device 800 is invoked and/or displays a message such as“What can I help you with?”
  • accessory device 820 transmits the representation of speech input 824 to first electronic device 800, but not to second electronic device 830.
  • the virtual assistant operating on second electronic device 830 may not be invoked, and thus second electronic device 830 does not display a graphical user interface indicating receiving a representation of a user’s speech input.
  • Capability data of second electronic device 830 can include, for example, device capabilities, application capabilities, and/or informational capabilities.
  • Device capabilities can include data associated with, for example, the types of sensors installed on the device (e.g., optical sensor, microphone, heart rate sensor, force tough sensor, etc.), the size of a display, audio processing capabilities, the number of speakers, or the like.
  • Application capabilities can include data associated with the applications installed on or accessible to the device (e.g., a workout application, an activity monitoring application, a heart rate measuring application, or the like).
  • Informational capabilities can include data that indicate the availability of certain information at the device. As an example, certain user-specific data may be only available at one device but not at other devices. For instance, the most updated user activity data may only be available at second electronic device 830 (e.g., the user’s wearable device) but not first electronic device 800. Thus, such data are obtained from second electronic device 830.
  • the audio data can include representations of speech outputs such as“outdoor run starting in 3, 2, 1...” and“you have completed 2 miles, goal achieved.”
  • the audio data corresponding to the performing of the task in accordance with the user request can be transmitted directly from the second electronic device 830 to the accessory device 820.
  • the audio data corresponding to the performing of the task in accordance with the user request can be transmitted from second electronic device 830 to accessory device 820 through first electronic device 800 (e.g., routed through first electronic device 800).
  • a determination of whether the representation of speech input 824 should be transmitted to first electronic device 800 or second electronic device 830 can be based on which electronic device is wirelessly coupled to accessory device 820. In some embodiments, the determination of whether the representation of a speech input received by accessory device 820 should be transmitted to first electronic device 800 or second electronic device 830 can be based on other criteria or conditions, as described below in FIGs. 9A-9C.
  • the determination of whether a representation of speech input 1024 is to be transmitted to first electronic device 800 or second electronic device 830 can be based on one or more coupling statuses between the devices.
  • accessory device 820 can determine whether it is wirelessly coupled (e.g., via Bluetooth pairing) to first electronic device 800 or second electronic device 830.
  • FIG. 10A accessory device 820 is paired with first electronic device 800, but not second electronic device 830 (e.g., FIG. 10A shows that second electronic device 830 is searching for Bluetooth headphone, but not paired, as indicated by graphical user interface 1032).
  • accessory device 820 determines that it is wirelessly coupled to first electronic device 800 but not second electronic device 830.
  • a particular electronic device if a particular electronic device has more battery power and/or signal strength than another electronic device, it can be more efficient to transmit the user request from the accessory device to the particular electronic device, rather than communicating with another electronic device that has less battery power and/or weaker signal strength.
  • the particular electronic device can then intelligently and effectively route the user request to another device if required. This enhances the overall operation efficiency of the devices and improves the human-machine interface.
  • FIG. 11 illustrates another example of providing virtual assistant services in response to a speech input received at accessory device 820. Similar to those described above, accessory device 820 receives a speech input 1124 (e.g.,“Hey Assistant, start a web- search.”). Accessory device 820 determines that speech input 1124 includes a trigger phrase (e.g.,“Hey Assistant”). Accordingly, accessory device 820 obtains a determination of whether the representation of speech input 1124 is to be transmitted to first electronic device 800 or second electronic device 830. In some examples, as illustrated in FIG.
  • a speech input 1124 e.g.,“Hey Assistant, start a web- search.
  • Accessory device 820 determines that speech input 1124 includes a trigger phrase (e.g.,“Hey Assistant”). Accordingly, accessory device 820 obtains a determination of whether the representation of speech input 1124 is to be transmitted to first electronic device 800 or second electronic device 830.
  • a trigger phrase e.g.,“Hey Assistant
  • accessory device 820 obtains a determination that the representation of speech input 1224 is to be transmitted to first electronic device 800 (e.g., because only first electronic device 800 is paired with accessory device 820).
  • first electronic device 800 detects electronic devices 830, 860, and 870, which may be all communicatively coupled to first electronic device 800.
  • First electronic device 800 obtains data associated with devices 830, 860, and 870 (e.g., capability data and user-specific data associated with the devices); and transmits representation of speech input 1224 and data associated with the detected electronic devices 830, 860, and 870 to the third electronic device 840 (e.g., a natural language processing server).
  • third electronic device 840 can determine which device is to perform the task in accordance with the user request. For example, as shown in FIG.

Abstract

L'invention concerne des systèmes et des processus permettant de fournir un service d'assistant virtuel. Selon un ou plusieurs exemples, un procédé consiste à recevoir, d'un dispositif accessoire couplé en communication au premier dispositif électronique, une représentation d'une entrée vocale représentant une demande d'utilisateur. Le procédé consiste également à détecter un deuxième dispositif électronique ainsi qu'à transmettre, à partir du premier dispositif électronique, une représentation de la demande d'utilisateur et des données associées au deuxième dispositif électronique détecté à un troisième dispositif électronique. Le procédé consiste également à : recevoir, du troisième dispositif électronique, une détermination du fait qu'une tâche doit être effectuée par le second dispositif électronique conformément à la demande de l'utilisateur ; et selon une détermination du fait qu'une tâche doit être effectuée par le deuxième dispositif électronique, demander au deuxième dispositif électronique d'exécuter la tâche conformément à la demande de l'utilisateur.
PCT/US2019/024332 2018-06-01 2019-03-27 Opération d'assistant virtuel dans des environnements multi-dispositifs WO2019231537A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201862679660P 2018-06-01 2018-06-01
US62/679,660 2018-06-01
DKPA201870355A DK201870355A1 (en) 2018-06-01 2018-06-11 VIRTUAL ASSISTANT OPERATION IN MULTI-DEVICE ENVIRONMENTS
DKPA201870355 2018-06-11
US16/119,576 2018-08-31
US16/119,576 US11495218B2 (en) 2018-06-01 2018-08-31 Virtual assistant operation in multi-device environments

Publications (1)

Publication Number Publication Date
WO2019231537A1 true WO2019231537A1 (fr) 2019-12-05

Family

ID=66397413

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/024332 WO2019231537A1 (fr) 2018-06-01 2019-03-27 Opération d'assistant virtuel dans des environnements multi-dispositifs

Country Status (1)

Country Link
WO (1) WO2019231537A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114063968A (zh) * 2021-11-11 2022-02-18 北京字节跳动网络技术有限公司 一种音频设备选择方法、装置及电子设备
WO2022221360A1 (fr) * 2021-04-15 2022-10-20 Apple Inc. Techniques de communication entre un dispositif concentrateur et de multiples terminaux
US20230036798A1 (en) * 2018-06-01 2023-02-02 Apple Inc. Systems and methods for providing search interface with contextual suggestions
GB2619894A (en) * 2021-04-15 2023-12-20 Apple Inc Techniques for communication between hub device and multiple endpoints
US11914537B2 (en) 2021-04-15 2024-02-27 Apple Inc. Techniques for load balancing with a hub device and multiple endpoints

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015183401A1 (fr) * 2014-05-30 2015-12-03 Apple Inc. Assistant intelligent pour domotique
US20170358305A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US20170358300A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Digital assistant providing automated status report
US20180062691A1 (en) * 2016-08-24 2018-03-01 Centurylink Intellectual Property Llc Wearable Gesture Control Device & Method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015183401A1 (fr) * 2014-05-30 2015-12-03 Apple Inc. Assistant intelligent pour domotique
US20170358305A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US20170358300A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Digital assistant providing automated status report
US20180062691A1 (en) * 2016-08-24 2018-03-01 Centurylink Intellectual Property Llc Wearable Gesture Control Device & Method

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230036798A1 (en) * 2018-06-01 2023-02-02 Apple Inc. Systems and methods for providing search interface with contextual suggestions
WO2022221360A1 (fr) * 2021-04-15 2022-10-20 Apple Inc. Techniques de communication entre un dispositif concentrateur et de multiples terminaux
GB2619894A (en) * 2021-04-15 2023-12-20 Apple Inc Techniques for communication between hub device and multiple endpoints
US11914537B2 (en) 2021-04-15 2024-02-27 Apple Inc. Techniques for load balancing with a hub device and multiple endpoints
CN114063968A (zh) * 2021-11-11 2022-02-18 北京字节跳动网络技术有限公司 一种音频设备选择方法、装置及电子设备

Similar Documents

Publication Publication Date Title
US20230036798A1 (en) Systems and methods for providing search interface with contextual suggestions
AU2021107576A4 (en) Attention aware virtual assistant dismissal
US11887585B2 (en) Global re-ranker
US11838734B2 (en) Multi-device audio adjustment coordination
US11231903B2 (en) Multi-modal interfaces
US10755703B2 (en) Offline personal assistant
US11783827B2 (en) Determining suggested subsequent user actions during digital assistant interaction
EP3699907A1 (fr) Maintien de la confidentialité d'informations personnelles
DK201970487A1 (en) Voice assistant discoverability through on-device targeting and personalization
WO2018208439A1 (fr) Traitement de serveur client d'une entrée de langue naturelle pour maintenir la confidentialité d'informations personnelles
AU2019280008A1 (en) Synchronization and task delegation of a digital assistant
US20220374727A1 (en) Intelligent device selection using historical interactions
WO2019231537A1 (fr) Opération d'assistant virtuel dans des environnements multi-dispositifs
US20230058929A1 (en) Digital assistant interaction in a communication session
EP3602540A1 (fr) Traitement de serveur client d'une entrée de langue naturelle pour maintenir la confidentialité d'informations personnelles
EP3959713B1 (fr) Identification par assistant vocal grâce au ciblage et à la personnalisation sur un dispositif
WO2020242711A1 (fr) Identification par assistant vocal grâce au ciblage et à la personnalisation sur un dispositif
EP3596625A1 (fr) Assistant personnel hors ligne
US11908473B2 (en) Task modification after task initiation
US20230359334A1 (en) Discovering digital assistant tasks
EP4287022A1 (fr) Exécution de tâche basée sur un contexte
WO2023219908A1 (fr) Modification de tâche après initiation de tâche
WO2023014893A1 (fr) Compréhension d'assistant multimodal à l'aide d'un contexte sur écran et dispositif
WO2019231552A1 (fr) Rejet d'assistant virtuel sensible à l'attention
WO2018212951A2 (fr) Interfaces multimodales

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19721882

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19721882

Country of ref document: EP

Kind code of ref document: A1