KR20190133642A - Providing a communications channel between instances of automated assistants - Google Patents

Abstract

According to embodiments of the present invention, the present invention relates to an interaction between an automatic assistance of a vehicle computer system and an individual client device through a wireless communication method. According to a part of the embodiments of the present invention, a client device(s) can be paired with a vehicle computer system, and the client device(s) and vehicle computer system can include individual instances of an automated assistance which can be associated with a different user account. According to other added or alternative embodiments of the present invention, certain actions to be performed by a local application of a client device can be enhanced through user interaction with a vehicle automatic assistance in spite of the correspondence to the local application which is not directly installed on the vehicle computer system. For example, despite the vehicle computer system which is not installed with a third party messaging application, the vehicle automatic assistance can be accessed to an action requested to be performed by the third party messaging application to add a requested action. The vehicle computer system transmits content through a wireless communication method for a third party application to be able to perform further requested action.

Description

PROVIDING A COMMUNICATIONS CHANNEL BETWEEN INSTANCES OF AUTOMATED ASSISTANTS}

This disclosure relates to providing a communication channel between instances of an automatic assistant.

Humans may also be referred to herein as "automated assistants" ("digital agents", "chat bots", "interactive personal assistants", "intelligent personal assistants", "assisted applications", "interactive agents", etc.). Can be used to participate in human-to-computer conversations. For example, humans (also known as "users" when interacting with an automatic assistant) can use spoken natural language input (i.e. speech) to provide commands and / or requests to the automatic assistant, and in some cases convert to text. And then processed and / or provide textual (eg typed) natural language input.

Automated assistants can be installed in a variety of devices such as, for example, mobile phones and personal computers. A user can have a particular application on a mobile phone or personal computer perform certain actions by interacting with each (auto) automated assistant. However, to indicate uniformity of functionality between different devices, a user may need to install instances of the same application synchronously on different devices. As a result, network bandwidth can be quickly immersed in sending redundant application updates for each instance of the application and generally in communicating data between each instance of the application. Also, since each instance of an application may require some disk space on that device, memory allocation across devices can be used inefficiently.

In some situations, interacting with a particular application via automatic assistant may not be convenient depending on how the user can participate in a particular environment and / or whether another device is operated by another nearby user. For example, replying to a message via the automatic assistant of a mobile phone can be difficult while the user is driving in a vehicle. Also, when a user drives with others in a vehicle, they may have a device with an automatic assistant, which may be called indirectly when the user contacts each automatic assistant. This results in an environment that is distracted while driving and wastes computer resources in devices that can be accidentally affected.

Embodiments disclosed herein relate to systems, methods, and apparatus for providing a communication channel through which individual instances or installations of automatic assistants can communicate to facilitate particular operations (actions). The communication channel may be provided between a primary auto assistant (eg, an automobile auto assistant or a client device auto assistant) and one or more subsidiary auto assistants (eg, another one of an automobile auto assistant and a client device auto assistant). For example, the main auto assistant may be the user's preferred assistant in certain situations, such as in a car. In particular, the automotive computer system may include an automatic assistant of the user's preference for furthering certain behaviors while the user is in the vehicle. Depending on the configuration of the car and / or client device, the car auto assistant may be responsible for the pending intents and / or actions of the subsidiary auto assistant as a 'principal' when the client device connects to the car computer system. I can take over. In this way, the automotive automatic assistant can perform the extension function of the secondary automatic assistant. This allows all client-side applications to tunnel or channel specific intents and / or behaviors for the automotive auto assistant. In addition, such tunneling may provide a system architecture that supports the ability of a user to participate in each client-side application without necessarily using or installing correlated "car-side" applications in the automotive computer system. In this way, more efficient use of automotive side memory can be achieved without requiring that correlated third party applications be necessarily installed on all automotive side devices. In addition, network traffic and wasteful power consumption can be mitigated simply by the user relying on an automatic assistant to communicate with at least one instance of the application, rather than using multiple instances of multiple applications for synchronous operation. In particular, a reduction in network and computational resources can be seen by application and messaging servers that send redundant copies of messages and acknowledgments to various other devices.

For example, in some implementations, the car's car assistant can be used to deliver a message received from a third party application of a client device, such as a mobile phone, and respond to the message in accordance with the interaction between the user and the car auto assistant. have. To achieve this functionality, the client device and the automotive computer system of the vehicle may perform a handshake to establish a secure communication channel between the client device and the automotive computer system. In particular, the client device may assign one or more network sockets (eg, Bluetooth sockets) as nodes for transmitting data to and / or from the automotive computer system. If a secure communication channel is established between the client device and the automotive computer system, the action intent may be shared between the client automatic assistant and the automotive automatic assistant.

An action intent may represent a request of an application for a specific action to be performed through an application or a separate application. For example, an action intent request may be generated by a third party application and associated with one or more actions to be performed, custom identifiers, content data associated with the action to be performed, and / or actions to be performed by the application. Any other data that may be present. The action intent may be initially received by the operating system (system), device application, automatic assistant and / or other engine or module available in the device where the action intent originated. The action intent can then be cataloged or otherwise queued at the originating device, and can access one or more instances of the automatic assistant. For example, the client auto assistant of the originating client device can access the pending action intent when the client device is paired with the car computer system and the car auto assistant can access the pending action intent.

For the example described above, the automobile auto assistant may communicate with the client auto assistant to identify the pending action intent request. The action intent request may be associated with a message received at a third party application on the client device while the user is in the car. The term "third party" may refer to the entity that created and / or provided the third party application, and is different from the creators and / or providers of client devices, client auto assistants and / or client operating systems. By accessing the action intent request from the client device, the car auto assistant can serve as an intermediary through which the user can proceed with a particular pending action intent request. In addition, the automotive auto assistant may cause the corresponding instance of the action intent request to be generated in the automotive computer system, as well as other action intent requests pending at the client device. That instance of the action intent request in the automotive computer system may be handled by the automotive automatic assistant even though the automotive computer system invalidates the third party application that sent the message.

The automotive automated assistant may determine in some implementations whether the action intent request is associated with a message based on the content of the action intent request, regardless of the third party application that provided the action intent request. In this way, automatic assistants can manage action intents in various applications that seem irrelevant to the application generating the action intent. For example, a first service provider of a first application and a second service provider of a second application may each provide a message in a different manner, but each message is irrespective of the "body" and "sender" fields. Will have In some implementations, overlap of message structures, which may be partly due to each application that satisfies the API, causes each message to be parsed similarly. The automatic assistant can then use the parsed data of the message to provide notifications, respond to the message, or perform other actions related to the message. In some implementations, similar overlap can be represented by other applications that can interact with the automatic assistant. Such other applications may include navigation applications, where the content of the notification may include route steps and / or media applications, and the content of the notification may include the address of specific media data such as images and / or video. can do.

When the car auto assistant determines that the instance of the action intent request is associated with a message, the car auto assistant can cause the content of the instance to be parsed. The parsed content may be used to identify individual portions of the message, such as body, author, subject, time sent and / or received, and / or any other data that may be associated with the message. Since the car auto assistant has determined that the action intent request corresponds to a message, the car auto assistant can use the parsed content to present the parsed content to the user. For example, a car auto assistant may be a text-to-speech for converting at least a portion of the content of an action intent request into audio that may be output from an auto assistant interface of a car computer system such as an audio system. The engine can be used. The audio may be, for example, "You received a text message from Emily that says 'I'll be late'." The authors "Emily" and "I'll be late" can be based on data parsed from that instance of an action intent request.

In response to informing the user of the message received at the client device, the car auto assistant may follow up a prompt that may present the user with an option to respond to the message. The prompt may include responsive audio, such as "Are you sure to reply to Emily?" In response, the user may provide a voice utterance such as "Yes, respond, 'Okay.'" Accordingly, the automotive auto assistant may receive a speech (speaked) utterance through the automobile's automatic assistant interface, and may employ a speech-to-text engine that is accessible to the automotive computer system for processing speech utterances. It is available. The text derived from the speech utterance may be embodied into content that may be part of a subsequent action intent request generated by the automotive automated assistant. Subsequent action intent requests may be further generated to identify an action, such as sending a message, that should be performed by the target application (eg, a third party application).

In order to synchronize the subsequent action intent with the action intent request generated by the client device, the car auto assistant generates the subsequent action intent using the same custom identifier as the action intent request generated by the client device (device). can do. In this way, the client device accesses the subsequent action intent request and / or determines whether the subsequent action intent request was generated in the automotive computer system and determines the correlation between the subsequent action intent request and the action intent request. Can be. In some implementations, the custom identifier can be used as at least one key for encrypting transmissions between automatic assistants.

If the subsequent action intent request originated from the automotive computer system, the client auto assistant may determine that the subsequent action intent request has been generated, and generate a corresponding instance of the subsequent action intent request at the client device. Since the subsequent action intent request is specified when sending a message through the third party application, the client auto assistant and / or the client device may invoke the third party application to send the message. That is, since the automotive computer system does not include an instance of a third party application, the generation of subsequent action intent requests does not directly call any instance of a third party application installed on the automotive computer system. Rather, in response to the client automatic assistant confirming the creation of the subsequent action intent request and / or the corresponding instance of the subsequent action intent request being generated at the client device, the third party application performs transmission of the response message.

In some implementations, the dependency of an action intent request on the automotive computer system and client device in response to the user acknowledging receipt of the message and / or indicative of the user wanting to respond to the message. Can be released. For example, in response to providing a voice spoken by the user (eg, answer, 'Okay'), the car auto assistant may cause that instance of the action intent request in the car computer system to be released. In addition, as the user provides the aforementioned utterance, the client automatic assistant may cause the action intent request to be released at the client device. If a separate corresponding instance of the action intent request associated with the originally received message is available at the remote server device, for example, the separate corresponding instance may also be dismissed.

The above description is provided as an overview of some implementations of the present disclosure. Further details of these and other embodiments are described in more detail below.

In some implementations, the method implemented by the one or more processors is described as including an operation, such as identifying a first action intent request generated at the client device by the automotive automatic assistant. The client device is paired with a car computer system via a wireless communication channel, the car computer system including a car auto assistant. The actions may further include in response to identifying the first action intent request, wherein the first action intent request is: a first requested action to be performed, content data to be accessed to perform the requested action, and Determining whether to identify an identifier associated with an application that makes the first action intent request at least partially available to the automotive computer system. The operations may also include making an automatic assistant interface accessible to the automotive computer system to provide an output to a user of the automotive computer system while promoting the first requested action to be performed. Is based on the content data of the first action intent request. The operations include determining, by the automobile auto assistant, a user provided an input in response to the output provided at the auto assistant interface; And in response to determining that the user provided the input, providing a second action intent request to the client device. The second action intent request may identify a second requested action to be performed at the client device, other content data and an identifier based on input provided by the user. Also, the operations may include causing the second requested action to be performed at the client device according to the second action intent request. The second requested action may be performed by an application using other content data, and the application may be identified by the client device using the identifier provided in the second action intent request.

In some implementations, the method can include providing a third action intent request to the client device in response to determining that the user provided the input, wherein the third action intent request is Identify an identifier and a third requested action, and in response to the client device receiving the third action intent request, the third action intent request is generated from the second action intent request at the client device. Allows fenders to be released. In some implementations, the method can include, in response to determining that the user provided input, cause a separate release of the corresponding redundancy of the instance of the second action intent request, wherein the second action is That instance of the tent request is provided to the car computer system. In some implementations, the dependency of the second action intent request is indicated by a first automatic assistant event handler accessible through the client device, and the separate dependency of that instance of the second action intent request is an automotive computer system. Represented by a second automatic assistant event handler accessible via.

In some implementations, the automotive computer system is provided by (i) a third party that provided the application on the client device, and (ii) there is no installed application that is at least partially correlated with the application on the client device. In some implementations, the automotive computer system is paired with a separate client device that includes another application provided by another third party with respect to an application at a client device, and the method further includes: by the automotive automated assistant, Identifying a separate action intent request generated at an individual client device, wherein the individual client device is paired with an automotive computer system. In some implementations, the separate action intent request includes another identifier generated by an individual client device, and the method is based on the difference between the identifier and the other identifier for the separate action intent request. Designating the first action intent request as a priority.

In other implementations, a method implemented by one or more processors is described as including an operation such as connecting using an automotive computer system with a first client device via a wireless communication scheme. A client automatic assistant, wherein the automotive computer system includes an automotive automatic assistant and automotive sensor application, wherein the automotive sensor application is configured to monitor at least one sensor of the automotive computer system. The method may also include connecting with a second client device via a wireless communication scheme using an automotive computer system, wherein the second client device includes another client automatic assistant. The method may further include determining, by the automotive computer system, that the first action intent request was generated at the first client device by the application, the application corresponding to a particular category of service, 1 is provided by the service provider. The method may also include determining, by the automotive computer system, that a second action intent request was generated at the second client device by another application, wherein the other application corresponds to a service of the particular category and Is provided by a second service provider. The method may further comprise determining, using the automotive computer system, that the third action intent request was generated in an automotive sensor application, wherein the third action intent request is a sensor from a sensor of the automobile computer system. It is generated based on the data. The method may also include generating, by the automotive automatic assistant, data indicative of the dependency of each action intent request of the first action intent request, the second action intent request, and the third action intent request. And the data is stored in an automotive computer system. The method according to the generated data causes one or more interfaces connected to the automotive computer system to determine that each of the first action intent request, the second action intent request, and the third action intent request is an action. The method may further include providing a notification about the tent request.

In some implementations, the method can include, in response to determining that the third action intent request was generated at an automotive sensor application, causing the instance of the third action intent request to be stored on the first client device. . In some implementations, the method includes receiving user input at the one or more interfaces connected to the vehicle computer system; And in response to receiving the user input, cause the redundancy of the notification associated with the third action intent request to be released from the vehicle computer system, and wherein the instance of the third action intent request is generated by the first client device. It may include the step of being released from. In some implementations, the particular category service is a messaging service and the first service provider is different from the second service provider. In some implementations, the method can include, for each action intent request of the first action intent request and the second action intent request, processing the content of each message specified by each action intent request. The processing of the content of each message may be performed regardless of whether each message has been provided by the service provider of the first service provider and the second service provider. In some implementations, generating data indicative of the dependency of each action intent request is, by the automotive automatic assistant, the first action intent request, the second action intent request, and the third action being Generating a corresponding instance of each action intent request of the tent request.

In another embodiment, a method implemented by one or more processors is described as including an operation such as being connected to a client device via a wireless module and a wireless computer system including a vehicle auto assistant, which is essential to the vehicle, and the client The device includes a client automatic assistant. The method may also include accessing, by a client device, an action intent request received by an automotive auto assistant from an application of the automotive computer system via the automotive computer system, the application communicating with a sensor of the vehicle. And the action intent request is based at least on a signal from the sensor. The method may further comprise generating a corresponding instance of the action intent request by the client automatic assistant and based on accessing the action intent request, wherein the client device comprises one or more other applications at the client device. It includes one or more other action intent requests provided by. The method may further comprise detaching the client device from the automotive computer system based at least on the client device becoming less proximally to the vehicle or automotive computer system. The method may further include performing, by the client auto assistant, the action according to the instance of the action intent request after being disconnected from the automotive computer system, wherein performing the action comprises the action intent request. Providing the content identified by the interface to the client device.

In some implementations, the client device indicates that there are no installed applications (i) provided by the third party that provided the application in the automotive computer system and (ii) at least partially correlated with the application in the automotive computer system. In some implementations, determining and performing a specific context is responsive to determining the specific context. In some implementations, determining the specific context includes determining a location of the client device, and the method includes whether the content identified by the action intent request is related to the location of the client device. The method further includes determining a, wherein the action is performed in response to determining whether the content identified by the action intent request is associated with a location of the client device. In some implementations, the method can include providing a request for an action intent request to be released in the automotive computer system based on a client automatic assistant performing an action in accordance with the instance of the action intent request. . In some implementations, the request is provided to a remote server device configured to communicate with the automotive computer system and cause the action intent request to be released in the automotive computer system. In some implementations, an action intent request is generated in response to a user providing voice speech to an automatic assistant interface of an automotive computer system, the method responsive to the user providing the voice speech to the automatic assistant interface. Receiving data corresponding to the voice utterance from the vehicle computer system; And providing response data to the vehicle computer system based on the data received from the vehicle computer system, wherein the action intent request is further based on the response data received by the vehicle computer system.

Other implementations include non-transitory computer readable storage media, graphics processing unit (s) (GPU (s), that store instructions executable by one or more processors (eg, central processing unit (s) (CPUs)). And / or tensor processing unit (s) (TPU (s)) for performing the methods described above and / or as elsewhere herein. Still other embodiments include one or more robots that include one or more processors operable to execute stored instructions to perform a system of one or more computers and / or methods as described above and / or elsewhere herein. can do.

It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein may be considered as part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this specification are considered to be part of the subject matter disclosed herein.

1 shows a perspective view of a vehicle in which a number of different users allow the vehicle auto assistant to interact with the application on their respective devices.
2 illustrates a system that provides an automatic assistant that can interact with other automatic assistants to add actions requested by third party applications.
3A, 3B, and 3C also illustrate perspective views of scenarios in which a client auto assistant of a portable computing device can act as the main or core auto assistant for multiple (plural) instances of the auto assistant.
4A and 4B illustrate a method for processing an action intent request generated when a client device communicates with an automotive computer system.
5 illustrates a method of providing output from an interface according to a number of different action intent requests provided from different instances of an automatic assistant.
6 is a block diagram of an exemplary computer system.

1 shows a perspective view 100 of a vehicle 114 that allows a plurality of different users to interact with an application in each device. When boarding the vehicle 114, the first user 110 and the second user 112 can interact with different devices during their journey in the vehicle 114. For example, first user 110 may bring cellular device 124 (ie, client device), and second user 112 may bring tablet device 116 (ie, client device). The vehicle 114 may include an automotive computer system 102 that can provide a user with access to the automotive automatic assistant 104. Each of the cellular device 124 and the tablet device 116 may include a client automatic assistant 126 and a client automatic assistant 118, respectively. In addition, the automotive automatic assistant 104 can act as the primary (primary) automatic assistant when each user brings each device to the vehicle 114. In this way, each user need not exclusively pair his or her respective device to the automotive computer system 102 only. Further, according to some implementations of the automotive automated assistant 104, the automotive computer system 102 does not necessarily have the same application installed on each client device. In this way, the automotive computer system 102 can make more efficient use of memory, and with fewer installed applications, the automotive computer system 102 can take up as much network bandwidth as it retrieves updates and communicates with the server. Will not consume. This advantage can be realized in part through the interaction between the respective automatic assistants for the wireless communication scheme available in the vehicle 114. Similar benefits can also be achieved with servers that use phone numbers to authenticate and forward each message.

For example, each user may enter the vehicle 114 with each client device, and each client device may include and / or generate an action intent request. For example, tablet device 116 may include one or more tablet applications 120 that may generate action intent request 122, and cellular device 124 generates action intent request 130. It may include one or more cellular applications 128 capable of doing so. Tablet application 120 and / or cellular application 128 may be an application and / or a third party application provided or developed by an entity other than the one that provided or developed one or more instances of the automatic assistant.

When the first user 110 and the second user 112 initially enter their vehicle 114 with their respective client devices, each client device may communicate with the vehicle 114 in a wireless manner. It may be determined to include an automotive computer system 102. In addition, each client device may determine that the automotive computer system 102 includes an automotive automated assistant 104. In particular, client auto assistant 118 and client auto assistant 126 can determine whether auto computer system 102 includes car auto assistant 104. In response to making the determination, the client automatic assistant 118 may cause the automotive computer system 102 to create a corresponding instance of the action intent request 122, and the client automatic assistant 126 may request an action intent request. Corresponding instances of 122, 130 may be created in the automotive computer system 102. As a result, the car auto assistant 104 may interact with one or more users and the car auto instance 104 via any interaction with that instance of the action intent request (stored locally as the action intent request 108). Can promote action. In addition, the automotive automotive assistant 104 may include an automotive computer 106 that may not be the same as any one application of the tablet application 120 and / or the cellular application 128 or may be an instance otherwise. Despite the system 102, the action identified by the action intent request 108 can be further performed. For example, automotive application 106 may include a maintenance application that provides information about vehicle 114 to automotive automotive assistant 104. The information may be shared with one or more of the tablet device 116 and the cellular device 124, even if either device does not contain an instance or equivalent of the maintenance application. This allows each user to relay each notification and / or any particular action depending on the car auto assistant 104 without necessarily paying attention to his or her device during the excursion.

2 illustrates a system 200 for providing an automated (automated) assistant that can interact with another auto assistant to add an action requested by a third party application. The automatic assistant 204 can be a first client device 216 (eg, cellular phone 242), a second client device 218 (eg, tablet device 244) and / or server device 230. Act as part of an assistant application provided in one or more computing devices, such as remote computing device 202. The user is via an assistant interface (eg, inside a vehicle 240), which may be a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between a user and an application. Interact with the automatic assistant (s) 204. For example, a user may enter language, text, and / or graphical input into the assistant interface so that the automatic assistant 204 may perform a function (eg, provide data, control peripherals, access an agent, etc.). By providing the automatic assistant 204 can be initialized. The first client device 216 is a display panel that includes a touch interface for receiving touch input and / or gestures for a user to access the cellular application 250 of the first client device 216 via a touch interface. It may include a display device that can be. The second client device 218 is also a display panel that includes a touch interface that receives touch input and / or gestures so that a user can control the cellular application 254 of the second client device 218 via a touch interface. It may include a display device that can be. In some implementations, the second client device 218 does not provide a graphical user interface output, and lacks a display device, thereby providing an audible user interface output. In addition, the second client device 218 can provide a user interface, such as a microphone, for receiving voice natural language input from the user.

The first client device 216 and the second client device 218 can communicate with the remote computing device 202 over a network 214, such as the Internet. The first client device 216 and the second client device 218 respectively transfer the computational task to the remote computing device 202 to conserve computational resources at the first client device 216 and the second client device 218. Can be offloaded. For example, the remote computing device 202 can host the automatic assistant 204, and the first client device 216 and / or the second client device 218 can remotely receive input received at one or more assistant interfaces. Transmit to computing device 202. However, in some implementations, the automatic assistant 204 can be hosted at the first client device 216 and / or the second client device 218. In various implementations, less than or all of all aspects of the automatic assistant 204 can be implemented on the first client device 216 and / or the second client device 218. In some of these implementations, aspects of the automatic assistant 204 implement other aspects of the local assistant application and the automatic assistant 204 of the first client device 216 or the second client device 218. It is implemented via an interface with a remote computing device 202. The remote computing device 202 can optionally provide a plurality of users and their associated assistant applications via a plurality of threads. In some implementations, all or part of the automatic assistant 204 is implemented through the client automatic assistant 226 of the first client device 216 or the client automatic assistant 226 of the second client device 218, and client automatic. The assistant may be an application separate from the operating system of the first client device 216 and / or the second client device 218 (eg, installed on the “top” of the operating system), or optionally It may be implemented directly by the operating system of the first client device 216 or the second client device 218 (eg, considered an application of the operating system but considered to be integrated with the operating system).

In some implementations, the remote computing device 202 can include a speech-to-text conversion engine 216 that can process audio data received at the assistant interface to identify text embodied in the audio data. The process of converting audio data into text may include a speech recognition algorithm that may use neural networks, word-to-vector algorithms and / or statistical models to identify groups of audio data corresponding to words or phrases. The text converted from the audio data can be parsed by the text parser engine 208 and sent to the automatic assistant 204 as text data that can be used to generate and / or identify command phrases from users and / or third party applications. Become available.

In some implementations, the remote computing device 202 can include a text-to-speech engine 228 for converting text data into audio data. For example, in response to the car auto assistant 234 identifying the pending action intent request 238 associated with the received message, the car auto assistant 234 may cause the content corresponding to the message to be text-to-speech engine 228. ) Can be sent. In response to receiving the content, the text-to-speech engine 228 may convert the content into audio data, which may be sent back to the automotive automatic assistant 234. The automobile auto assistant 234 may then cause the audio data to be output as audio output from the audio interface (eg, speaker (s)). Text can be converted to audio using any other device or module that can convert neural network (s), statistical models, and / or text data into audio (eg, natural language output corresponding to audio). In some implementations, each speech-to-text conversion engine 216, text-to-speech engine 228 and / or text parser engine 208 may be used in the automotive computer system 232, the first client device 216 and / or the like. Or on one or more client-side devices, such as second client device 218. In this way, each automatic assistant does not necessarily have to send data to the remote server for processing, but rather can process the data locally to generate and / or interpret specific data for the user.

In some implementations, one or more of the operations, applications, and / or engines of server device 230 may be performed at automotive computer system 232, first client device 216, and / or second client device 218. May be provided and / or implemented. For example, in some implementations, automotive computer system 232 can include text-to-speech engine 228 and / or voice-to-text conversion engine 216. In addition, in some implementations, any content received at car auto assistant 234, client auto assistant 246, and / or client auto assistant 248 may be between the device hosting each assistant and the server device 230. May be sent to the server device 230 when there is a connection. Alternatively or additionally, when no connection is available between each device or system and server device 230, any content received is processed exclusively at each device so that the content can be processed in parallel. Or shared with other local devices. For example, when content is received at the second client device 218 and corresponds to a message sent to the car computer system 232, the car computer system 232 provides audio that may be provided to the user while in the car. You can use the local text-to-speech engine to generate it.

In some implementations, each device or system that includes one or more applications can also include one or more corresponding action intent generation engines. The action intent generation engine may be controlled by an application to cause specific actions to be performed by the application, other applications, and / or other modules accessible to the application. For example, the application may be a messaging application, and the messaging application may receive an incoming message. In response to receiving the incoming message, the action intent generation engine of the messaging application may generate an action intent to provide notification to the user via an automatic assistant. The action intent may identify the action to be performed (eg, provide a notification), the content of the received message, a custom identifier associated with the message, and / or any other data that may be associated with the message.

In some implementations, each action intent generation engine and / or application can be associated with a declaration and / or manifest file that indicates the type of action that can be performed by the application. For example, the messaging application may include a manifest file indicating that the message application can receive at least the message, notify the user, and / or respond to the message. The operating system and / or the automatic assistant can access each manifest file on one or more devices to determine whether a particular application can handle a particular action. In this way, for example, the action intent created in the automotive computer system 232 may be based on whether or not the particular manifest file of one of the devices indicates that there is an application on the device that can handle the action intent. It may be provided to the first client device 216 or the second client device 218. If more than one application can handle a particular action intent, the originating device and / or a separate device may select a device to handle the particular action intent (eg, via a graphical user interface). You can prompt the user.

In some implementations, each action intent generation engine can also generate and / or filter action intents according to a customer identifier associated with each action intent of the action intent. For example, the action intent generation engine may also generate a custom identifier when the action intent generation engine generates an action intent in the direction of the application. The custom identifier is an identifier for the content to be used to perform the action, an identifier representing the originating application, an identifier for the recipient, an identifier for the originating user, an identifier for the context of the action intent, and / or for generating an identifier. It can be generated based on any other information that can be used as a basis.

Each automatic assistant and / or operating system may be configured to parse content data, an intended action to be performed, and / or an action intent to identify a custom identifier. In some implementations, the automatic assistant can handle each action intent that appears agnostic with respect to the originating application. For example, although the first client device 216 and the second client device 218 may include different messaging applications, the automotive computer system 232 may be capable of messaging applications through the automatic assistant interface of the automotive computer system 232. Can retrieve action intent requests and read the message. For example, the message application at the first client device 216 may generate an action intent request 252, and when the user enters the vehicle 240, the pending action intent request may be generated by the car auto assistant ( 234 may be accessed. The car auto assistant 234 may then create a corresponding instance of the pending action intent request. Since the content data provided by the action intent request may include data such as "Hey, have you left yet?", The car auto assistant 234 may request that the content data be an incoming message. Can be determined to correspond to. Thus, because many other messaging applications generate action intents in a similar manner, the automotive automatic assistant 234 is in a universal manner and / or in such messaging applications (eg, first service provider, second service provider, etc.). It is possible to present and respond to such action intents relatively agnosticly for the entity that provided.

In some implementations, computational resources can be conserved by using a speech-to-text conversion engine to process audio data corresponding to multiple speech utterances rather than performing multiple turns between the speech recognition engine and the automatic assistant. . For example, the core (major) automatic assistant may receive a user command such as "send a message to Robert." In response, the core (major) automatic assistant may have a preconfigured response such as "Ok, which messaging application do you want to use in your first application, second application, or third application?" The user can provide a response such as "My First Application" and can direct a message like "How was today?" In some implementations, the automatic assistant confirms which messaging application to use and asks if the user sends or changes a message (for example, "Do I need to confirm, send a message, or change the message?"). At this point, the user may request to select another messaging application. For example, the user may reply "Send to the second messaging application instead of the first messaging application." In response, the automatic assistant can cause the message to be sent by the second messaging application. This reduces the number of turns between the user and the automatic assistant because the message processing does not have to be tailored for each individual messaging application.

In some implementations, each auto assistant (eg, auto assistant 234) can access and / or manage an event handler that includes data corresponding to a pending action intent request. When an action that performs a pending action intent request is performed, each event handler of each auto assistant can be updated. In some implementations, the event handler can include a pending action intent request corresponding to the same event. For example, you can associate several different actions with a received message. Thus, when a user receives a text message at his or her first client device 216, the messaging application may generate a number of action intent requests to process the received text message. When the first client device 216 pairs with the automotive computer system 232 via a wireless communication scheme such as Bluetooth, Wi-Fi, LTE, or the like, the automotive automatic assistant 234 may associate corresponding instances of multiple action intent requests. Can be generated. That instance of the multiple action intent request may be managed by an event handler of the automobile auto assistant 234, which associates the instance of the multiple action intent requests with the text message received at the first client device 216. It can contain a certain amount of data. For example, the action intent request identified by the event handler of automobile auto assistant 234 may correspond to an action that informs the user of a text message and prompts the user whether or not they want a response. The automotive automated assistant 234 may execute two actions related to the received text (text) message without having direct local access to a messaging application installed on the first client device 216.

3A, 3B, and 3C also illustrate perspective views 300, 302, and 304 of a scenario in which a client auto assistant of a portable computing device can operate as a primary (core, primary) auto assistant for multiple instances of an auto assistant. ). For example, while the user is at home 318, a malfunction may be detected in the car 316 associated with the user. For example, it may be an issue that rodents chewed a cable, such as a wire, that is connected to the brake sensor of the car 316 at night. Although automotive computer system 306 of automobile 316 may operate in a standby mode, automobile auto assistant 308 may generate one or more action intent requests 312 associated with the issue in response to the issue. . Specifically, the automobile auto assistant 308 may request an action intent request 314 to display a car issue notification (eg, a "brake sensor") to the user and maintain the maintenance ( Example: "Modify Schedule?") May generate another action intent request 320 to prompt for scheduling. Each action intent request can be generated with the same custom identifier (eg '4KAJC92W') that can be associated with the original issue. In addition, each action intent request may identify actions to be performed (eg, "notifications" and "schedules").

3B shows a perspective view 302 of a user 322 entering a car 316 with a cellular device 328 that also includes an instance of the automatic assistant as the client automatic assistant 328. In some implementations, automotive computer system 306 may include automotive application 310 that is not directly installed on cellular device 328. In addition, in some implementations, the cellular device 328 can include a cellular application 332 that is not directly installed in the automotive computer system 306. Nevertheless, the action intent request may be sent between the automotive computer system 306 and the cellular device 328 via a wireless scheme, and may be parsed by each automatic assistant.

For example, action intent requests 314 and 320 may be generated by the maintenance application of the car application 310 and accessed by the car auto assistant 308. When a user 322 enters a vehicle with their cellular device 328, the automotive computer system 306 and the cellular device 328 establish a handshake to ensure that a secure communication channel 326 is provided between them. Can be done. Once the communication channel 326 is established, the client auto assistant 330 can identify the pending action intent request 312 available to the automotive computer system 306. In addition, automotive computer system 306 may identify any action intent request 334 available at cellular device 328. When the client auto assistant 330 determines that the action intent requests 314 and 320 do not have any corresponding instances at the cellular device 328, the client auto assistant 330 is responsible for the action intent requests 314 and 320. The instance may be made available to the cellular device 328. The instances of action intent requests 314 and 320 at the cellular device 328 may include the same custom identifier from the automotive computer system 306, thus causing a particular event (eg, an issue where a brake sensor was detected). To be compiled over time.

3C shows a perspective view 304 of user 322 in his home 346 after instances of action intent requests 314 and 320 have been created. In particular, user 322 may enter his home 346, and in response, the client auto assistant may attempt to complete the action indicated by action intent requests 314 and 320. For example, the client automatic assistant can cause the display interface of the cellular device 328 to provide selectable elements 338 and 340 that can correspond to action intent requests 314 and 320. Alternatively, user 322 may voice 336 the interface (eg, a microphone) of cellular device 328 to cause the client auto assistant to present any notification associated with any pending action intent request. Can be provided. For example, the user 322 can provide a voice utterance 336 such as "show assistant, pending notification." In response, the client automatic assistant causes the nearby display panel to present a notification to the user 322 related to the action intent request.

When the user 322 selects one of the selectable elements 338 and 340, the device's dependency on action intent requests 314 and 320 respectively includes an instance of the action intent requests 314 and 320. Can be dismissed from. For example, instances of action intent requests 314 and 320 may be released at automotive computer system 306 and cellular device 328. In some implementations, in response to the user 322 selecting the selectable elements 338 and 340, if data can be sent to the remote device 344 via the network 342, the action intent request ( 314 and 320 may be released in an automotive computer system. The remote device 344 may then be accessed by the automotive computer system 306, which may update the event handler of the automotive computer system 306 according to the data at the remote computer 344.

4A and 4B also illustrate methods 400 and 410 for handling action intent requests generated when a client device communicates with an automotive computer system. The methods 400 and 410 may be performed by one or more of a computing device, an application, and / or any other device or module capable of communicating over a wireless connection. The method 400 may include an operation 402 of identifying, by the automotive automated assistant, a first action intent request generated at the client device. The client device can be paired with the automotive computer system via a wireless communication scheme. The first action intent request may be generated by the application at the client device, for example in response to an action performed at the application (eg, receiving a message). In some implementations, the automotive computer system can void that instance of the application of the client device. In particular, the automotive computer system may invalidate a separate application provided by a third party that provides the application on the client device and / or at least partially correlates with the application on the client device. For example, the application may be a text messaging application, and the automotive computer system may invalidate the text messaging application.

The method 400 may further include an operation 404 of determining whether the first action intent request identifies a first requested action, content data, and a custom identifier to be performed. The first requested action may be a request to inform the user that an incoming message has been received at a messaging application installed on a client device. The content data may identify some or all of the text provided in the received text message (eg, “in progress?”). In addition, the custom identifier may be a string of data that may provide a correspondence between the received text message and the first action intent request, such that any additional action taken with the first action intent request was received. May be subsequently associated with the message.

The method 400 may also include an operation 406 that causes an automatic assistant interface accessible to the automotive computer system to provide output to a user of the automotive computer system. For example, the output may be provided in response to an automobile auto assistant that determines that the first action intent request corresponds to an action that provides a notification. The notification may be provided at an automatic assistant interface, which may be a display panel connected to the automotive computer system. Alternatively, the automatic assistant interface may be a speaker connected to the automotive computer system, and the notification may be an audible audio output such as "Meeting has sent you the message 'Are you going?'".

The method 400 may further include an operation 408 of determining, by the automobile auto assistant, whether the user provided an input in response to an output provided at the auto assistant interface. For example, a user may provide a spoken speech that responds to the car's automatic assistant interface (such as a microphone) such as "Respond. Yes, go." The automotive assistant device may receive audio data corresponding to the responding voice utterance, and may cause the audio data to be locally processed or transmit the audio data to a separate device for processing. By processing the audio data, the car auto assistant can determine whether the user has approved the notification of the text message and any other action the user wants to perform. For example, in response to receiving the responding voice utterance, the automotive automated assistant may cause the second action intent request to be generated.

Method 400 may proceed to method 410 as indicated by the continuous element “A” enclosed in circles in FIGS. 4A and 4B. The method 410 may include an act 412 of providing a second action intent request to the client device in response to determining that the user provided the input. The second action intent request may identify a second requested action, other content data, and / or a custom identifier associated with the first action intent request. The second requested action may be a request to respond to a text message received at the messaging application. The other content data may include text based on the responding voice utterance (eg, "content-data = 'text: yes, go"), the custom identifier being the same custom identifier as the first action intent request. Can be.

The method 410 may further include an operation 414 for causing the second requested action to be performed at the client device by an application using other content data. For example, if the action includes sending a replying message, the client device may use the content data to generate a replying message and send a response message to the sender who initially sent the text message to the client device. . In order to identify the text message and / or application to which the second action intent request corresponds, the instance of the automatic assistant at the client device can compare the custom identifier with the custom identifier provided with the first action intent request. Optionally or additionally, the automatic assistant can identify the messaging application based on whether the messaging application can perform the second requested action, as indicated by the manifest file of the messaging application. The automatic assistant of the client device allows the messaging application to perform the second requested action and thus send a text message corresponding to the responding voice utterance.

5 shows a method 500 for providing output from an interface in accordance with a number of different action intent requests provided from different instances of an automatic assistant. Method 500 may be performed by one or more of a computing device, an application, and / or any other device or module capable of interacting with an automatic assistant. The method 500 may include an operation 502 for connecting, by an automotive computer system, with a first client device that includes a client automatic assistant via a wireless communication scheme. The first client device may be, for example, a portable computing device such as a mobile phone or tablet computer, and the automotive computer system may be a computing device essential for a vehicle capable of carrying a large number of passengers. In addition, the wireless scheme may be one or more wireless communication protocols, such as Bluetooth, Wi-Fi, LTE, and / or any other manner in which multiple computing devices may interact.

The method 500 may further comprise an operation 504 of connecting (connecting) with a second client device including another client automatic assistant by the automotive computer system and via a wireless communication scheme. The other client automatic assistant may be provided in the same or different entity as the client automatic assistant, and the second client device may be the same or different type of client device as the first client device. For example, the first client device may be owned by the driver of the car and the second client device may be owned by the passenger of the car. Further, when the driver and the car passenger enter the car, the operations 502 and 504 may be performed. In this way, the car auto assistant can interact with the client auto assistant and other client auto assistants when the driver and passenger are in the car.

The method 500 further includes an operation 506 of determining that the first action intent request was generated by the application at the first client device and the second action intent request was generated by another application at the second client device. can do. The first action intent request may correspond to a request for a notification to be provided to a driver, wherein the notification may provide information about a message received at the first client device. The second action intent request may correspond to another request for another notification to be presented to the passenger. The other notification can provide different information about the other message received at the second client device.

The method 500 may also include an operation 508 for determining that the third action intent request was generated by a separate application in the automotive computer system. The separate application can be, for example, a maintenance (maintenance) application that monitors one or more sensors of the vehicle. In addition, the first client device and the second client device may invalidate any corresponding instance of the maintenance application. In some implementations, the third action intent request can correspond to a request for a notification to be provided to the driver regarding a vehicle condition, such as delayed oil change and / or brake fluid needs to be replaced.

The method 500 also operates 510 for generating, in the automotive computer system, data indicative of the dependency of each action intent request of the first action intent request, the second action intent request, and the third action intent request. It may include. The data may be implemented or accessible as an event handler engine capable of sorting data corresponding to various different action intent requests from various different applications and / or devices. For example, the data can identify the priority of each action intent request. The priority for an action intent request may be, for example, whether the action intent request was generated on a device that includes a primary (core, primary) automatic assistant (i.e., an automatic assistant with which at least one other automatic assistant is communicating). Can be based on For example, when a driver and a passenger board a car, the primary auto assistant may be a car auto assistant because it is installed in the car computer system of the car. The next priority may be given to the same action intent request provided by the device associated with the driver of the vehicle, such as the first client device. Thereafter, another priority may be assigned to the action intent request associated with the passenger of the vehicle.

The method 500 causes the one or more interfaces connected to the automotive computer system to cause each action intent of the first action intent request, the second action intent request, and the third action intent request based on the generated data. It may further comprise an operation 512 to provide a notification for the request. In some implementations, the automotive computer system can include an interface, such as a display panel, on which graphical elements are presented, and one or more graphical elements can correspond to any of the action intent requests. The display panel may also be a touch interface that allows the user to select graphical elements to confirm notifications and to generate other action intent requests in the automotive computer system.

6 is a block diagram of an example computer system 610. Computer system 610 typically includes at least one processor 614 in communication with a number of peripheral devices through bus subsystem 612. Such peripherals may include, for example, storage subsystems including memory 625 and file storage subsystem 626, user interface output device 620, user interface input device 622, and network interface subsystem 616. 624). Input and output devices allow user interaction with computer system 610. Network interface subsystem 616 provides an interface to an external network and is coupled to a corresponding interface device of another computer system.

The user interface input devices 622 may be a pointing device such as a keyboard, mouse, trackball, touch pad or graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a speech recognition system, a microphone and / or other type. It may include an input device. In general, the use of the term “input device” is intended to include all possible types of devices and methods for entering information on the computer system 610 or communication network.

User interface output device 620 may include a non-visual display such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanism for generating a visible image. The display subsystem may also provide a non-visual display through the audio output device. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from computer system 610 to a user or other machine or computer system.

Storage subsystem 624 stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic and / or automotive automatic assistants, automotive computer systems, client automatic assistants, client devices, remote devices, servers to perform the selected aspects of the method 400, method 500. Logic to implement one or more of an apparatus, a vehicle, and / or any other module or apparatus discussed herein.

These software modules generally execute on the processor 614 alone or in combination with other processors. Memory 625 used in storage subsystem 624 includes main random access memory (RAM) 630 for storage of instructions and data during program execution and read-only memory (ROM) 632 in which fixed instructions are stored. It may include a plurality of memory including a. File storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Can be. Modules that implement the functionality of a particular implementation may be stored on the file storage subsystem 626 of the storage subsystem 624 or on another machine accessible by the processor (s) 614.

Bus subsystem 612 provides a mechanism for various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is schematically depicted as a single bus, other implementations of the bus subsystem may use multiple buses.

Computer system 610 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 610 shown in FIG. 6 is intended only as a specific example for describing some implementations. Many other configurations of computer system 610 are possible having more or fewer components than the computer system shown in FIG. 6.

If the systems described herein collect personal information about you (or often referred to herein as "participants"), or if personal information is available to you, you may not be able to obtain a program or feature that contains information about you (e.g. your social network). , Social behavior or activities, occupations, preferences of the user, or the current geographical location of the user), or may be provided with the opportunity to control whether It is possible to control whether to receive content from the content server and / or a receiving method. Certain data may also be treated in one or more ways before it is stored or used so that personally identifiable information can be removed. For example, the user's identity may be processed so that no personally identifiable information can be determined for the user or the user's geographic location may be generalized to where the geographic location information is obtained (such as city, zip code or state level). Can be processed, and thus a particular geographic location of the user cannot be determined. Thus, the user can control how to collect and / or use information about the user.

While some implementations have been described and illustrated herein, various other means and / or structures may be used to perform the functions and / or obtain the results and / or one or more advantages described herein, and such variations and / or Modifications are considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, and actual parameters, dimensions, materials, and / or configurations will depend upon the particular application or use for which the teaching is used. Those skilled in the art will recognize, or be able to ascertain, many equivalents to the specific embodiments described herein using routine experimentation only. Accordingly, it is to be understood that the foregoing embodiments are presented by way of example only, and that within the scope of the appended claims and equivalents thereto, the embodiments may be practiced otherwise than as specifically described and claimed. Embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, all combinations of two or more such features, systems, articles, materials, kits, and / or methods are included within the scope of the present disclosure unless such features, systems, articles, materials, kits, and / or methods are inconsistent with each other. .

Claims (1)

A method implemented by one or more processors,
Identifying, by the automotive auto assistant, the first action intent request generated at the client device, wherein the client device and the individual client device are paired with the automotive computer system via a wireless communication channel, the automotive computer system being Includes a car auto assistant;
In response to identifying the first action intent request, the first action intent request is executed: a first requested action to be performed, content data to be accessed to perform the first requested action, and the automotive computer Determining if the system identifies an identifier associated with an application at the client device that makes the first action intent request at least partially available; wherein the respective client device is a third party providing the application at the client device; Another application provided by a third party with respect to the third party, wherein the automotive computer system is free of installed applications provided by the third party;
Identifying, by the automotive automated assistant, an individual action intent request generated at the respective client device;
In the promotion of the first requested action to be performed, making the automatic assistant interface accessible to the automotive computer system to provide an output to a user of the automotive computer system, wherein the output is to determine the first action intent request. Based on content data;
Determining, by the car auto assistant, whether a user provided an input in response to the output provided at the auto assistant interface;
In response to determining that the user has provided the input, providing a second action intent request to the client device, wherein the second action intent request is a second requested action to be performed at the client device; Identifying the identifier and other content data based on the input provided by a user; And
Causing the second requested action to be performed at the client device in accordance with the second action intent request;
The second requested action is performed by the application using the other content data, wherein the application is identified by the client device using the identifier provided in the second action intent request The method implemented by the above processor.

Images