JP6934092B2 - Initialize conversations with automation agents via selectable graphical elements - Google Patents

Description

Automation assistants (also known as "personal assistant modules", "mobile assistants", or "chatbots") are used by users through various computing devices such as smartphones, tablet computers, wearable devices, automotive systems, and stand-alone personal assistant devices. Can be interacted with. The automation assistant receives input from the user (eg, type and / or spoken natural language input) and responds with responsive content (eg, visual and / or audible natural language output).

Automation assistants can provide a wide range of functionality through interaction with various local and / or third-party agents. In order for a user to utilize an automation assistant to perform a particular function, the user often first explicitly launches the automation assistant (for example, through certain verbal phrases and / or certain hardware inputs). Then you have to give a specific activation phrase related to a specific function. The launch phrase launches an agent that can perform a particular function through an automation assistant. However, the user may be unaware of the various features of the automation assistant and / or the launch phrase for invoking such features through the automation assistant. In addition, the user may not have an automation assistant loaded to answer the query (for example, get flight details from the tablet) on one of its devices, but the other has an automation assistant. You may not know that the user can get an answer from the device. As a result, in some situations, users may utilize other resource-inefficient applications to perform certain functions. In addition, even if the user launches the automation assistant, in some situations the user aggregates extensive resources with the automation assistant to know how to perform a particular function through the automation assistant. You may still need to be involved in a typical dialogue (for example, a dialog turn). For example, a large number of dialog turns may be required even for the user to know that an automation assistant can enable the performance of a particular function.

Techniques for invoking an automation assistant to communicate with an agent module associated with an application are described herein. Some implementations allow the user to be presented with selectable elements within the non-automated assistant graphical user interface of an application separate from the automated assistant application. In response to the user selection of the selectable element, the automation assistant may launch an agent module that corresponds to the selectable element and is associated with what is presented via the non-automation assistant interface. In some of those implementations, selectable elements are selectable by single tap, single click, or other "single selection" method. In these and other methods, the user can select selectable elements to transition from the non-conversational interface to the conversational automation assistant interface, and the agent (related to the content in the non-conversational interface) is the automation assistant interface. Launched within. In some implementations, launching the agent within the automation assistant interface in such a manner may reduce the amount of input required by the user to perform the functionality of the agent module. Specifically, instead of first launching the automation assistant and then giving a particular agent module or a specific launch phrase related to the function of the agent module, the agent module is launched directly based on the graphical user interface. obtain. Agent modules associated with the graphical user interface can be launched directly through the automation assistant. The number of inputs required by the user is reduced, at least by not requiring additional invocation phrases for a particular agent module. In this way, the automation assistant does not need to output general queries and handle additional launch terms for a particular agent module. This reduction in input saves computational resources and benefits a variety of users, for example, users with dexterity problems. In addition, the discovery of various automation assistant features can be facilitated, thereby facilitating the use of potentially more resource-efficient automation assistant interfaces for further dialogue.

In some implementations, the method implemented by one or more processors involves steps such as receiving a selection of selectable elements in a graphical user interface rendered by a non-automated assistant application on a computing device. It is explained as. The selectable element may indicate that the agent associated with the graphical user interface can be launched through an automated assistant application that is separate from the non-automated assistant application. The step may further include invoking the agent through the automation assistant application in response to the selection of selectable elements. The agent can be one of several available agents that can be launched through the automation assistant application. The steps further include receiving a response from the agent in response to invoking the agent, and providing output based on the response received from the agent by the automation assistant application via the automation assistant interface. obtain.

In other implementations, the method implemented by one or more processors is described as involving steps such as displaying selectable elements on a computing device operating a non-automated assistant application. The selectable element can be configured to initialize the agent module associated with the non-automation assistant application to the automation assistant. The steps may further include receiving a selection of selectable elements and determining whether the automation assistant is accessible to the computing device in response to receiving the selection of selectable elements. The step also performs a link corresponding to the selectable element when the automation assistant determines that it is inaccessible to the computing device, and puts a spoken command phrase on the default web page to communicate with the agent module. It can also include opening and presenting.

In yet another implementation, one step involves determining that a user is browsing the application interface of a non-automated assistant application when a non-temporary computer-readable medium is run by one or more processors. Alternatively, it is described as storing instructions to be executed by a plurality of processors. The application interface may include a first selectable element for initializing communication with the agent module via the automation assistant application. The agent module can be configured to perform actions related to the non-automated assistant application. The step may further include receiving a selection of the first selectable element. The first selectable element may include a link that identifies the agent module and parameters for performing the action. Steps can also include having the user present a conversational interface. The conversational interface can be configured by an automation assistant to act as an intermediary between the user and the agent module. Further, the steps may include providing a second selectable element in the conversational interface. The second selectable element may be based on the parameters identified within the link to proceed with the action.

In addition, some implementations include one or more processors of one or more computing devices, one or more processors capable of operating to execute instructions stored in the associated memory. The instruction is configured to implement any of the methods described above. Some implementations also include one or more non-temporary computer-readable storage media that store computer instructions that can be executed by one or more processors to perform any of the methods described above.

FIG. 5 shows a diagram in which a user activates a conversational user interface on a mobile device. FIG. 5 shows a diagram in which a user activates a conversational user interface on a mobile device. FIG. 5 shows a diagram in which a user activates a conversational user interface on a mobile device. FIG. 5 illustrates a conversational user interface that can be used to interact with agent modules associated with a website. FIG. 5 illustrates a system for providing a conversational interface on a client device to familiarize a user with utterable commands available to control various applications and / or websites. It is a figure which shows that the conversational user interface is presented in the user interface of a computing device. It is a figure which shows that the conversational user interface is presented in the user interface of a computing device. It is a figure which shows that the conversational user interface is presented in the user interface of a computing device. It is a figure which shows the method for having an agent module perform a function through an automation assistant. FIG. 5 illustrates a method for limiting an action according to the presence or absence of a native application for performing the action performed in response to the selection of selectable elements. FIG. 5 illustrates a method for interacting with an agent module through an automation assistant, depending on whether the agent module is accessible. FIG. 6 is a block diagram of an exemplary computer system or computing device.

The implementation described relates to a system, method, and device for interacting with an agent module associated with an application using an automation assistant. As an example, suppose a user visits the Pizza Company's "Order Pizza" web page through a web browser application on a client device. When selected, select through a web browser application that causes the automation assistant application on the client device to launch the "Pizza Company" agent and present the user with the automation assistant interface output generated by the "Pizza Company" agent. Possible elements can be presented. In other words, in response to element selection, the automation assistant application may launch the agent and allow the user to engage in a dialog with the agent through the automation assistant interface. In some implementations, selectable elements can be included as the content of a web page (eg, incorporated within a web page by The Pizza Company). In addition, in some implementations, the agent responds to the selection of selectable elements with values for intents and / or intent parameters based on user interaction through the web browser application (for example, "slot value"). Can be activated. For example, if a user interacts with the Order Pizza web page and selects the Large 1 Topping pizza, the agent is launched with the Order Pizza intent and the slot values Large and 1 Topping. obtain.

Figures 1A-1C show a diagram in which the conversational user interface 114 is activated from application 106 on a mobile device. Specifically, FIG. 1A shows Diagram 100 of application 106 displayed in the user interface 108 of the mobile device. Application 106 can be, for example, a hotel booking application that allows a user to book a hotel through a user interface. The mobile device may include an automation assistant to assist various functions of the mobile device, or may communicate with an automation assistant of a separate device. For example, an automation assistant can respond to voice commands and translate voice commands into text that can be used by agent modules associated with the application, such as application 106. The application on the mobile device 104 may be associated with an agent module specifically designed to assist the user in performing the functions associated with the application 106. In some implementations, the automation assistant can initialize a voice or text conversation with the user and act as an intermediary between the user and the agent module associated with application 106. However, the user may not be aware of the available utterable commands for the automation assistant, or the agent module associated with application 106, which leaves the user with an inefficient means of interacting with the agent module. To introduce the user to the utterable commands available to communicate with the agent module through the automation assistant, the automation assistant is a conversational user interface that is initialized through the selection of the first selectable element 112. 114 can be provided.

The first selectable element 112 may include a phrase indicating to the user that the user can use the user's automation assistant to communicate with the agent module associated with application 106. For example, the first selectable element 112 may include the phrase "use an automation assistant" so that the user can perform functions or agent modules related to application 106 through the user's automation assistant 102. Notify to. Initially, if the user 102 does not know any utterable command, the user 102 may select the first selectable element 112 or speak the phrase "use an automation assistant". In response to user 102 selecting the first selectable element 112 or speaking the phrase "use the automation assistant", the automation assistant initializes and launches the agent module for application 106. Can be done. The first selectable element 112 may be associated with a link or command that specifically identifies the command or intent to be executed by the agent module and / or the agent module. In some implementations, the link can be a universal resource locator (URL) such as "http://assistant.url/hotel-agent-module/hotel-booking", or any command that identifies the agent module. The link may also contain any information provided to the application by the user prior to selecting the first selectable element 112. For example, as shown in Figure 1A, the user may have already selected a date (“3/14”) and number of guests (“1”) for hotel reservations. Therefore, the link can identify the agent module and include the date and number of guests. In this way, the automation assistant may be notified about the progress of the hotel reservation and may activate the agent with a specified date and number of guests. For example, an agent can be activated with a "Hotel Reservation" intent and a value of "3/14" for the "Date" slot parameter and a value of "1" for the "Number of Guests" slot parameter. An example of such a link could be "http://assistant.url/agent-module/hotel-booking-date_0314_guests_1". In some implementations, the link or command is an opaque parameter (for example,) to hide details about how the agent receives input from the automation assistant in order to eliminate the damage caused by malicious URL authors. "... / date_889293") can be included.

In some implementations, in response to the user selecting the first selectable element 112, the automation assistant uses the link and the agent module identifier in the link so that the agent module is accessible to the automation assistant. It can be determined if there is. If the agent module is available to the automation assistant, the automation assistant may launch the agent module and optionally present the user with commands available for further interaction with the agent module. For example, in Figure 1B, the agent module associated with application 106 is launched in conversational interface 114, and the output (“Agent” is in the conversational interface based on the response generated by the agent module in response to the launch. What is: ...? ") Is presented. For example, the agent module may be launched with a "hotel reservation" intent and in response to provide the output shown in Figure 1B.

In some implementations, the automation assistant can access the index of historical commands given to application 106. The automation assistant can use the historical command index to make suggestions for the user to interact with the agent module. Once the corresponding agent module has been identified or the history command has been identified, the automation assistant or other application on the mobile device 104 will be able to use the user within the conversational user interface 114, as given in diagram 116 in Figure 1B. Can present a list of commands to. For example, proposal element 120 may include the phrase "my previous booking location". If proposal element 120 is selected, the automation assistant may instruct the agent module to book a hotel location in the same location where the user previously booked the hotel. The automation assistant may retrieve the user data corresponding to the dialogue between the user and the application 106 to determine the location of the previous hotel reservation. Alternatively, proposal element 122 may include the phrase "place in my calendar". If Proposal Element 122 is selected, the Automation Assistant instructs the Agent Module to book a hotel location according to an event stored in the user's calendar on the date shown in the link ("3/14"). Can be done. The automation assistant also speaks the input to the automation assistant when selected, with a text input element 124 that allows the user to type input within the conversational user interface 114. Can be provided with an audio input element 132 that enables. In this way, the user can optionally choose between selecting the suggested input for the agent module or giving text or voice input to the agent module via an automation assistant. Input from the user can then be given to the agent module by the automation assistant, allowing the agent module to generate further response content to continue the dialog.

The list of commands may contain commands for proceeding with the actions initiated in the application using the interaction between the automation assistant and the agent module. The list of commands can be spoken commands that are understood by the agent module, or text commands that are spoken by the user and can be converted to text using an automation assistant or other application on a mobile device or remote device. For example, the link associated with the first selectable element 112 may identify the modality of how the automation assistant receives the input and / or provides the output. Modality can be text, audio, or any other medium for receiving input and providing output. Modality can be identified within the link provided to the automation assistant. For example, a link can identify an agent module, intent or action, and / or modality (eg, "http://assistant.url/agent-module/hotelbooking-text_modality").

The conversational user interface 114 may be updated as shown in Diagram 118 of FIG. 1C in response to the user selecting any of the proposed elements 120, 122, or 124 in Diagram 116 of FIG. 1B. The updated conversational user interface 130 in Figure 1C may contain additional responses from the agent module. For example, if the automation assistant communicates the location of the hotel reservation to the agent module, the agent module may send further responses to the automation assistant corresponding to the payment for the hotel reservation. The automation assistant may present the user with an optional response element for responding to the agent module. The automation assistant can use the response content from the agent module to search for user data accessible to the automation assistant and generate a selectable response based on the search. For example, the agent module provided the automation assistant with a query asking about payment, so that the automation assistant can retrieve user data for payment information. If the automation assistant determines that the user has a payment card stored on the mobile device, the automation assistant may present a response element 126 containing the phrase "pay with my stored card". In some implementations, the automation assistant can predict that the user will also want to perform some other function once the interaction with the agent module is complete. In such cases, the automation assistant can respond to one agent module (eg, a hotel booking agent module) and launch another agent module to complete another function (eg, booking a flight). Agent response element 128 may be presented to the user. For example, an automation assistant may instruct the agent module to charge a user for a hotel reservation using a memorized card when spoken or selected, "Pay with memorized card ... Can provide a dual agent response element 128 containing the phrase. At the same time, the automation assistant may provide the phrase "... help me book a flight" to the dual agent response element 128. In response to the user selecting dual agent response element 128, the automation assistant may also launch the agent module corresponding to the flight booking application or website. In this way, the number of inputs required by the user to perform some actions can be reduced. This can be beneficial for users with dexterity problems or other illnesses that can prevent users from efficiently giving input to their mobile devices.

FIG. 2 shows a conversational user interface 214 that can be used to interact with the agent module associated with website 206. While browsing website 206 on mobile device 204, user 202 learns that some features related to website 206 can be controlled through spoken commands that can be interpreted by the automation assistant and / or agent module. Sometimes not. To better familiarize user 202 with utterable commands, user 202 may be instructed to initialize an automation assistant that may provide a conversational user interface 214 according to the implementation described herein. For example, user 202 may be browsing food ordering website 206, as shown in Diagram 200 of FIG. While browsing website 206, user 202 may identify the first selectable element 212 presented in user interface 208 of mobile device 204. The first selectable element 212 is a phrase indicating to user 202 that an automation assistant can be used to interact with the agent module associated with website 206 to perform some functions associated with website 206. May include. For example, the first selectable element 212 may include the phrase "use an automation assistant" as shown in FIG. The first selectable element 212 may be associated with a link such as a URL that may be provided to the automation assistant to open a conversational user interface on the mobile device 204. For example, a link may identify an agent module suitable for receiving a speakable command from user 202 to perform a function associated with website 206 (eg, ordering food). Such links have the structure, for example, http://assistant.url/food%ordering%agent%module, where "%" indicates the spacing character and the type of agent module (eg, "food order"). Is identified after the host name. In some implementations, if the mobile device 204 contains a third-party application (for example, a food ordering application) that supports website 206, the link is automated to continue the conversation between user 202 and the food ordering application. Can be transferred to a third party application by an assistant. Otherwise, the link may be received by the automation assistant and a conversational user interface 214 may be provided on the mobile device 204 or a separate computing device 224.

As yet another example, website 206 can be a food delivery order website, where users interact with website 206 (eg, via drop-down menus, radio buttons, freeform text) to discuss food orders. Items and / or ingredients (eg toppings for pizza) may be selected and, optionally, delivery orders may be terminated and payments may be made. If the user partially fills out a food order through interaction with Website 206, the value for one or more of the selected ingredients is sent to the Automation Assistant, which is the agent associated with Website 206. It is possible to start a module with such a value (for example, such a value is included as the slot value sent to the agent module at the start of the agent module). In some implementations, the server hosting website 206 may generate such a value to pass to the automation assistant. For example, a link about a selectable graphical element can be dynamically generated by the server in response to a user interaction with website 206, thus the link contains an indication of such a value (eg, "" http://assistant.url/agent-module/order-pizz_toppings=pepperoni-mushroom-peppers "). For example, links related to selectable graphical elements may be dynamically updated by the server in response to a user's interaction with website 206. As another example, the server may send commands in response to a selection of selectable graphical elements, which include such values and optionally also indicate an agent module. For example, the command given by the server to the automation assistant can be adjusted by the server to include the toppings selected for the pizza through interaction with the user's website 206 (for example, the command "ACTION = com.assistant." toppings_pepperoni-bacon-onion.StartConversation "). In some implementations, the automation assistant itself may process the contents of the website 206's interface to determine such values directly. For example, one or more screenshots of the interface are processed to determine the text of the field title and / or the selected value for the field, and such title and / or value is utilized to make a launch request. With can determine the appropriate value to pass to the agent module. In some implementations where values derived from the user's interaction with the non-automated assistant interface are used in launching the associated agent module, duplicate re-entry of such values through the automated assistant interface is reduced ( For example, it can be resolved). This can save various resources. The agent module can be launched with such a value, thereby eliminating the need for a dialog turn through an automated assistant interface to define such a value.

In some implementations, the automation assistant may not be available on the mobile device 204 browsing website 206, but the mobile device 204 is a separate computing device 224 that includes the automation assistant over the network. Can be connected to. In this implementation, when the user 202 selects the first selectable element 212, the mobile device 204 provides the computing device 224 with a link (or other content) to activate the automation assistant on the computing device 224. Can be. Automation assistants can use links to identify agent modules and identify data related to the status of current operations running on website 206.

The conversational user interface 214 may include a plurality of different selectable elements, including words and phrases corresponding to utterable commands for interacting with the agent module via the automation assistant. The phrase is the preconfigured commands given to the automation assistant by the agent module or website as a result of the automation assistant processing processing the contents of the website, and / or the history with the user's website recorded by the automation assistant. Get based on dialogue. For example, selectable elements in the conversational user interface 214 may include a phrase such as "order food delivery," which is an order (or other command) detailed in the link provided to the automation assistant. ) Based on the status. This phrase can be spoken by user 202 and converted to text by an automation assistant. The text may then be provided to the agent module associated with website 206. The agent module receives the text and can complete the food delivery according to the text.

In some implementations, the link may contain parameters to guide the automation assistant during the interaction with the agent module. For example, user 202 may at least partially fill out food ordering website 206 before selecting the first selectable element 212. The portion of website 206 completed by user 202 may include order data such as location for delivery, food quantity, and / or beverage order. This data can be embedded in the link corresponding to the first selectable element 212. For example, the link corresponding to the first selectable element 212 could be "http://assistant.url/agent-module/breakfast-order/drink-coffee-location-market-street". The link can be parsed by the automation assistant to identify parameters for subsequent actions or intents to be performed by the agent module. For example, an intent "breakfast-order" identified within a link may contain multiple parameters that need to be identified before the intent can be completed. The automation assistant uses the parameters "coffee" and "market street" to notify the user of the current status of the order and request additional information to complete the order (for example, "pay for food"). obtain.

In some implementations, each of the selectable elements provided in the conversational user interface 214 may be preconfigured according to the content of website 206. In other words, the automation assistant and / or parser engine of the mobile device 204 may process the contents of website 206 to generate selectable elements and / or utterable commands for interacting with website 206. In other implementations, the agent module of website 206 may be associated with preconfigured commands and parameters that may be stored on or accessible to mobile device 204 or computing device 224. These preconfigured commands and parameters are produced by the automation assistant and / or parser engine on mobile device 204 to generate utterable commands and / or selectable elements for interacting with the agent module through the automation assistant. Can be processed. In this way, user 202 may know that he can interact with website 206 or the agent module by voice in order to simplify the functionality of website 206 by not relying on completely typing out the command. This can be beneficial for users who are tired or have dexterity problems.

In some implementations, website 206 may be associated with several different agent modules, and the automation assistant identifies the most suitable agent module to initialize according to user 202's current and / or previous activity. Can be done. For example, user 202 may be browsing food ordering website 206, which may be associated with a number of different agent modules, each capable of specializing in ordering a particular type of food. For example, the first agent module may specialize in ordering breakfast foods and the second agent module may specialize in ordering dinner foods. The automation assistant determines that User 202 is more likely to be interested in ordering breakfast food and says, "Talk to an agent about ordering breakfast food." It may provide selectable elements, including utterable commands. The Automation Assistant provides the time when User 202 is browsing Website 206, the history of ordering breakfast food from Website, media accessible to Mobile Device 204 such as messages and / or calendar entries, and / or Based on any other data suitable for use in predicting the behavior of user 202, it may be determined that user 202 is more likely to order breakfast food. The selectable elements provided by the website may correspond to links that specifically identify multiple different agent modules. For example, selectable elements can correspond to commands that enumerate agent modules for automation assistants to select and initialize (for example, "ACTION = com.assistant.BreakfastAgentModule.StartConversation, com.assistant.LunchAgentModule." StartConversation, OR com.assistant.DinnerAgentModule.StartConversation "). The command may be provided on website 206 for reception by the automation assistant. Alternatively, the selectable element may correspond to a link that identifies the agent module (for example, "http://assistant.url/agentmodules/breakfast-agent_lunch-agent_dinner-agent"). The link or command can then be received by the automation assistant, so that the automation assistant can select the most suitable agent module to initialize based on the analysis of the user data.

In some implementations, when the automation assistant is initialized from mobile device 204 to compute device 224, should the automation assistant parse user data and another agent be initialized from compute device 224? You can judge whether or not. For example, the automation assistant may be aware of the agent module associated with the movie website that user 202 frequently visits on computing device 224. User 202 may select the first selectable element 212 to initialize the automation assistant to interact with the agent module associated with food ordering website 206. At the same time, the automation assistant may also provide user 202 with the option of communicating with the agent module associated with the movie website, as shown in FIG. For example, an automation assistant may simultaneously provide selectable elements 218 and selectable elements 222 that may be associated with different actions for different agent modules or websites, respectively. In this way, the user sequentially speaks two commands (for example, "order food delivery and start the last movie watched") to a single automation assistant that accomplishes two separate actions. , Can communicate by voice with two different agent modules.

FIG. 3 shows a system 300 for providing a conversational interface 316 on a client device 302 to familiarize a user with utterable commands available to control various applications 304 and / or websites. The application 304 running on the client device 302 may be associated with one or more agent modules 310 that help the user perform functions related to the application 304. The agent module 310 may be stored in a remote device such as a client device 302 or a server device. In some implementations, the server device may store one or more automation assistants accessible to the client device 302. The automation assistant 320 may receive the voice data recorded by the microphone of the client device 302, control some features of the client device 302, and interpret the voice data for the purpose of interacting with the agent module 310.

In some implementations, the client device 302 and / or the server device may include a selectable element engine 326 that may generate data to provide selectable elements in the client device 302. Selectable element engine 326 is selectable to help the user initialize communication with agent module 310 through automation assistant 320 to perform functions related to application 304 and / or web browser 314. Can generate elements. For example, the selectable element engine 326 may be notified when a user is interacting with application 304 or browsing a website in a web browser 314. In response, the selectable element engine 326 may generate selectable elements that can initialize the conversational interface 316 on the client device 302 when selected on the interface of the client device 302. The selectable element may include words generated by the selectable element engine 326 or provided by the automation assistant 320. Selectable Elements Engine 326 and / or Automation Assistant 320 recognizes Agent Module 310 associated with Application 304 and provides a phrase about Selectable Elements that indicates that Automation Assistant can be used to interact with Agent Module 310. Can be done. For example, the phrase of the selectable element can be "perform an application function using an automation assistant" that can be spoken by the user to open the conversational interface 316. Alternatively, the user may select selectable elements to open the conversational interface 316.

The conversational interface 316 may include a plurality of different selectable elements, including words and phrases that may be based on user data 312. For example, the agent module 310 may correspond to the gaming application 304 and the agent module 310 may accept commands typed by the user. The agent module 310 can be designed by the manufacturer to accept the typed preconfigured command 312, while the automation assistant 320 is to translate the user's utterances into commands that can be understood by the agent module 310. Can be used. For example, initially when a user opens a gaming application 304, the selectable element engine 326 may be notified that the gaming application 304 has been opened and provide the selectable element with the phrase "use an automation assistant". .. The user can then select selectable elements or speak words for the purpose of initializing the automation assistant 320 to communicate with the agent module 310 corresponding to the gaming application 304. When application 304 is, for example, a chess game application, the conversational interface 316 may include a plurality of different selectable elements with phrases corresponding to chess movements. The phrase may be based on user data 312 such as previous commands entered by the user, data communicated from the agent module 310, and / or the parsed application content provided by the text parsing engine 324. To select the movement corresponding to the selectable element, the user can select the selectable element or speak a phrase located in the selectable element. For example, the selectable elements of conversational interface 316 may include the phrase "move a pawn." The selectable element can correspond to a link that identifies the action to be performed (eg, moving a pawn in a chess application), and the conversational interface 316 has additional words available to complete the action (eg, moving the pawn). , "Move to A5") can be updated. The link may then be provided to the agent module 310 associated with application 304.

In some implementations, the user can speak and the audio data captured by the client device 302 can be provided to the automation assistant 320 for processing. For example, the automation assistant 320 may include a voice-text engine 322 that can receive audio data captured by client device 302 and convert the audio data to text. The text may correspond to the user's utterance while the audio data was being captured by the microphone of client device 302. Automation Assistant 320 also has a text parsing engine 324 that can parse text and identify specific words or phrases that correspond to input parameters for websites provided in Agent Module 310 and / or Web Browser 314. Can include. The Automation Assistant 320 may then generate a link or command from the parsed text and send the link or command to the agent module or website for processing. For example, when the user sees the phrase "move a pawn" in the conversational interface 316 for the gaming application 304, the user may speak the phrase "move a pawn" on the client device 302. The client device 302 can then capture the audio audio data of the audio and share the audio data with the automation assistant 320. Automation Assistant 320 then generates a link containing the parsed text that corresponds to the voice and sends the link to Agent Module 310 or Gaming Application 304. For example, the link could be a URL such as "http://assistant.url/chess-agent-module/move-pawn", where the URL is processed by the agent module 310 and used by application 304 to advance the chess game. Can be translated into a command for. The link may also be used by the selectable element engine 326 to generate a new selectable element that replaces at least some of the previous selectable elements in the conversational interface 316.

In some implementations, if the user is browsing the website in a web browser 314 and the website is not associated with the agent module 310, the user is still presented with a conversational interface 316 to interact with the website. Can be done. For example, a user may be browsing a hotel website that is not associated with Agent Module 310. Nevertheless, the selectable element engine 326 may cause the client device 302 to display selectable elements that include the phrase "use an automation assistant." In this way, the user can be notified that the hotel website is available to receive spoken commands without the agent module 310 available. In response to the user selecting a selectable element or speaking the phrase "use an automation assistant", the web browser 314 provides a conversational interface 316 to present the user with additional selectable elements. Can be open. Additional selectable elements include data from application 304, such as website content and / or messages, calendar data, browser history, order history, driving instructions, and / or any other data based on user activity. It can be generated by the selectable element engine 326 based on the contents of the user data 312 which may be included. The selectable element may correspond to a web link that identifies at least part of the content of the spoken command and may be processed by the server hosting the website. A web link may then be provided to the website or web browser 314 to facilitate the hotel booking process. If the user chooses to speak the phrase "book a hotel in Illinois" presented in the clickable element, the audio data corresponding to the phrase "book a hotel in Illinois" will be sent to Automation Assistant 320. Can be provided. The automation assistant 320 can then convert the speech to text in the speech-text engine 322 and then parse the text in the text parsing engine 324. The parsed text can then be converted to a web link in the selectable element engine 326, and then the web link can be provided to the website or web browser 314 to proceed with the booking process. The selectable element engine 326 may then repopulate the selectable element into the conversational interface 316 in order to proceed with the booking process according to the query received from the agent module 310. For example, the selectable element may correspond to commands such as "select reservation date", "select room size", and / or any other command related to booking a hotel.

4A-4C show that the conversational user interface is presented in the user interface of the computing device. Specifically, Diagram 400 of FIG. 4A shows a first user interface 406 for a website presented in a web browser of a computing device. A website can be, for example, a radio website that advertises various radio stations that a user can listen to. The website may include a first selectable element 408 presented in the first user interface 406. The first selectable element 408 may provide the user with instructions that an automation assistant may be used to interact with the agent module associated with the website. For example, the first selectable element 408 may include the phrase "use the automation assistant" which may be spoken by the user to open the conversational user interface 410 in the computing device to the automation assistant. The user may also select the first selectable element 408 by touching the first user interface 406 or by giving the computing device some other selection command. In response to the first user interface 406 being selected or the phrase being spoken by the user, the automation assistant may receive a link identifying the agent module being associated with the website. The automation assistant can also perform queries to identify whether the application associated with the website exists on the computing device or is accessible to the computing device. The automation assistant may then provide results in the conversational user interface 410.

The conversational user interface 410 of Diagram 402 of FIG. 4B may include a first agent module part 412 and a second agent module part 416. The first agent module part 412 may correspond to the part of the conversational user interface 410 related to the agent module associated with the website. The second agent module part 416 may correspond to the part of the conversational user interface 410 related to the native application associated with the website. For example, because a radio website is an audio listening website, the automation assistant presents the user with utterable commands to identify the audio-related application on the computing device and interact with the agent associated with the audio-related application. obtain. As shown in Diagram 400, one audio-related application can be a podcast application. However, in some implementations, the associated application can be a third-party application that is unique to those who control the automation assistant application.

The automation assistant may simultaneously present selectable elements related to both the agent module associated with a website (eg, a radio website) and the agent module associated with a related application (eg, a podcast application). Selectable elements may include utterable command phrases based on user data that correspond to the interaction between the user and both the website and the application. For example, a first set of selectable elements 414 may correspond to an action previously performed by a user using a website. The spoken command phrase "start music" may correspond to a start button previously selected by the user to start a radio website that plays music. The previous selection of the start button recorded by the automation assistant can be converted into selectable elements that are selected in the conversational user interface 410 and / or can be spoken to the computing device. In addition, a second set of selectable elements 418 may correspond to actions taken by the user in related applications such as podcast applications. For example, actions prior to the user starting a podcast can be recorded by an automation assistant and used as the basis for providing selectable elements that include the spoken command phrase "start a podcast."

In some implementations, the conversational user interface 410 may notify the user that it can speak to the agent module associated with the associated application. For example, the conversational user interface 410 may provide a selectable element that includes the spoken command phrase "speak to a podcast agent." In response to selecting the selectable element labeled "Talk to Podcast Agent," the Automation Assistant now makes the conversational user interface 410 a second user interface 426 that contains suggestions related to the podcast application. Can be updated. For example, the first updated interface part 418 may contain a number of different selectable elements 420 that correspond to previous or predicted actions recorded by the agent module associated with the automation assistant or podcast application. In addition, a second updated interface portion 422 may be provided by the automation assistant based on user data related to historical actions previously performed by the user while operating the podcast application. For example, a historical action related to a podcast could be a food delivery order. Users may have a routine to go home, turn on podcasts, and order food, so automation assistants may recognize this routine and provide a conversational interface to complete the routine more efficiently. .. Each selectable element 424 provided in the second updated interface part 422 is to the computing device to perform an action that is selected by the user or identified within the utterable command phrase of the selectable element 424. Can be spoken orally. In addition, the automation assistant may provide a selectable element 424 that can identify one or more agent modules corresponding to the associated application and, when selected, initialize the conversation with the agent module through the automation assistant. .. For example, related applications (eg, food delivery applications) can be associated with Asian and Ethiopian culinary agents. Each of the different agent modules associated with the food delivery application can specialize in assisting the category of actions of the food delivery application, and the automation assistant allows the user to interface with the agent module through the automation assistant. Can notify the user.

FIG. 5 shows a method 500 for the agent module to perform a function through an automation assistant. Method 500 may be performed by a client device, a server device, a module or application running on the device, and / or any other device suitable for interacting with the application. Method 500 may include block 502 that receives in the computing device a selection of links that identify the agent module associated with the application. The application can be any application on a computing device that includes content that can be manipulated through user input. For example, an application can be a home monitoring application that allows a user to use the application to control various devices in the user's home. The link can be a selectable element that contains a phrase indicating that the automation assistant can be used to interact with the agent module associated with the application. For example, a link may contain the phrase "use an automated assistant" and the link may be activated by selecting the link or speaking the phrase "using an automated assistant". At block 504, a determination is made as to whether the automation assistant is accessible to the computing device. The automation assistant can be a separate application or module that acts as an interface between the agent module and the user to guide the user in performing the functions associated with the application. In block 508, a conversational interface can be opened using the automation assistant if the automation assistant is available on the computing device. For example, a conversational interface can be opened with an activated agent module. For example, a conversational interface can be opened and can provide output based on what is received from the agent module in response to the activation of the agent module. If the automation assistant is not available for the computing device, in block 506, a conversational interface may be opened in the browser's default web page. At block 510, the user interface input is received on the conversational interface. The user interface input can be a type and / or spoken input given by the user, and / or a selection of selectable elements presented in the conversational interface. For example, a user may be given typed or spoken input to participate in a dialog with an agent module (via a conversational interface) and / or a text phrase proposed to proceed with a dialog with an agent module. Alternatively, the presented selectable elements may be selected, including other proposed content. At block 512, additional content is provided to the agent module based on the user interface input received at block 510. The agent module can then generate additional response content based on the additional content provided to the agent module. Such additional response content (or a transformation thereof) may be provided to present to the user in facilitating a conversation with the user, and optionally, the method returns to block 510 to further respond in response to further response content. Can accept user interface input. Further response content generated by the agent module depends on the user interface input received in block 510.

FIG. 6 shows method 600 for limiting the actions performed in response to the selection of selectable elements, depending on whether there is a client application to perform the actions. Method 600 may be performed by a client device, a server device, a module or application running on the device, and / or any other device suitable for interacting with the application. Method 600 may include block 602 that opens a conversational user interface containing selectable elements and utterable command terms. The conversational user interface may include multiple selectable elements that correspond to the actions that can be performed by the agent module associated with the application or website. At block 604, a selection of selectable elements can be received. The selectable elements may be selected by the touch interface of the computing device, peripheral devices connected to the computing device, and / or any other mechanism for selecting the selectable element in the interface of the computing device. At block 606, it is determined whether the command phrase corresponds to a restricted action. A restricted action can be an action that may use private information about the user during the action. For example, a non-automated assistant application or website can be a food ordering website, and selectable elements can correspond to the behavior of ordering food using payment information relevant to the user. Alternatively, the non-automated assistant application or website can be a social media website and the selectable elements can correspond to the behavior of publicly posting the user's image. In block 606, if the command phrase does not correspond to a restricted action, then in block 608 the command corresponding to the selectable element may be executed. For example, a command that does not correspond to restricted actions can be to perform a function within a gaming application or to access a news article provided by a news website. Such commands can be performed by speaking the command phrase or by selecting selectable elements. If the command phrase corresponds to a restricted action, block 610 determines if there is a client application to execute the command. If a client application exists, block 614 causes the client application to receive commands. For example, if the command phrase corresponds to an order for food from a website, the client application can be a food ordering application associated with the website. In this way, restricted actions can be left in the client application. The command may be provided to the client application in the form of a link, such as a URL, that identifies the agent module for executing the command. For example, when the client application is a food ordering application, the link could be, for example, "http://assistant.url/foodwebsite/food%ordering%agent". In this way, the client application orders food when the user has previously visited a food website, as indicated by the identifier "food% ordering% agent" (% can be understood as a space). Get notified that you are interested in. The food ordering agent can then be initialized and the conversational user interface can be updated to include different selectable elements that correspond to the command terms for continuing the food order. Alternatively, in block 612, an instruction may be provided in the computing device that the command is restricted if there is no client application for executing the initial utterable command phrase. The user can then manually continue the order through the website (rather than the conversational user interface) or cancel the order.

Figure 7 shows method 700 for interacting with an agent module through an automation assistant, depending on whether the agent module is accessible. Method 700 may be performed by a client device, a server device, a module or application running on the device, and / or any other device suitable for interacting with the application. Method 700 may include block 702, which provides selectable elements in the conversational user interface for performing functions related to the non-automated assistant application. Selectable elements are associated with links or commands that can identify the agent module, actions or intents to be performed by the agent module, and / or parameters for use by the agent module when performing the action or intent. obtain. Alternatively, the selectable element may correspond to a command call to initialize the automation assistant. At block 704, the selection of selectable elements may be received by the automation assistant. The automation assistant can then identify the agent module that corresponds to the selectable element. At block 706, a determination is made as to whether the agent module is accessible to the automation assistant. The agent module may be accessible to the automation assistant when the agent module is loaded on the same device as the automation assistant, or when the automation assistant can communicate with network devices that include the agent module. If the automation assistant has access to the agent module, at block 710, the agent module may be initialized to facilitate dialogue with the non-automation assistant application. In other words, the automation assistant can act as an interface for the user to communicate more efficiently with the agent module. However, if the agent module is not accessible to the automation assistant, in block 712, the link corresponding to the selectable element provides a spoken command phrase to perform functions related to the non-automation assistant application through the automation assistant. Can be opened on the default web page to communicate. In this way, the user is still provided with assistance in facilitating the user's interaction with the non-automation assistant application, even if the automation assistant does not have an accessible agent module.

FIG. 8 is a block diagram 800 of an exemplary computer system 810. Computer system 810 typically includes at least one processor 814 that communicates with several peripheral devices via bus subsystem 812. These peripheral devices may include, for example, a storage subsystem 824 including a memory subsystem 825 and a file storage subsystem 826, a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. Input and output devices allow user interaction with computer system 810. The network interface subsystem 816 provides an interface to an external network and is coupled to a corresponding interface device in another computer system.

The user interface input device 822 is a pointing device such as a keyboard, mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen built into the display, a voice recognition system, an audio input device such as a microphone, and / Or it may include other types of input devices. In general, the use of the term "input device" shall include all possible types of devices and methods for inputting information within computer system 810 or over communication networks.

The user interface output device 820 may include an invisible display such as a display subsystem, printer, fax machine, or audio output device. A display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or any other mechanism for creating a visible image. The display subsystem may also provide an invisible display, such as through an audio output device. In general, the term "output device" shall include all possible types of devices and methods for outputting information from computer system 810 to the user or another machine or computer system.

The storage subsystem 824 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, storage subsystem 824 implements selected embodiments of methods 500, 600, 700 and / or of the server devices, client devices, databases, engines, and / or modules described herein. It can contain logic to implement one or more.

These software modules are typically run on processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 includes a main random access memory (RAM) 830 for storing instructions and data during program execution, and a read-only memory (ROM) 832 for storing fixed instructions. May include some memory including. The file storage subsystem 826 may provide persistent storage for programs and data files and may include hard disk drives, floppy disk drives and related removable media, CD-ROM drives, optical drives, or removable media cartridges. .. Modules that implement the functionality of some implementations may be stored in the storage subsystem 824 by the file storage subsystem 826 or in other machines accessible by the processor 814.

Bus subsystem 812 provides a mechanism for the various components and subsystems of computer system 810 to communicate with each other as desired. Although the bus subsystem 812 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

The computer system 810 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 810 shown in FIG. 8 shall be merely a specific example to illustrate some implementations. Many other configurations of the compute device 810 are possible, with more or fewer components than the compute device shown in FIG.

In situations where the system described herein collects or utilizes personal information about a user, the program or feature may be the user information (eg, the user's social network, social behavior or activity, occupation, user). Controls whether to collect (information about the user's current geographic location), or whether and / or how to receive content from the server that may be more relevant to the user. Opportunities for control may be provided to the user. In addition, some data may be treated in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's geographic location can be generalized when geolocation information is obtained so that personally identifiable information cannot be determined for the user or a user's specific geographic location cannot be determined (for example). User identification information can be handled, such as at the city, zip code, or state level. Therefore, the user may have control over how information is collected and / or used for the user.

Although some implementations are described and illustrated herein, to perform a function and / or to obtain one or more of the results and / or the advantages described herein. Various other means and / or structures may be utilized and each such variant and / or modification is considered to be within the scope of implementation described herein herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, and actual parameters, dimensions, materials, and / or configurations are specific applications. It depends on the example, or the application in which this teaching is used. Using only routine experiments, one of ordinary skill in the art will understand or be able to confirm many equivalents of the particular implementation described herein. Therefore, the above implementations are presented merely as an example, and are specifically described and implemented in a manner different from that claimed, within the appended claims and their equivalents. Understand what you get. Implementations of the present disclosure are directed to their respective individual features, the systems, articles, materials, kits, and / or methods described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods conflicts with each other such features, systems, articles, materials, kits, and / or methods. If not, it is included within the scope of this disclosure.

204 nodes
100 diagram
102 users
104 mobile devices
106 application
108 User interface
112 Selectable elements
114 Conversational interface
116 Diagram
118 Diagram
120 Suggested elements
122 Proposed elements
124 Proposed elements
126 Response element
128 Response element
130 Updated conversational user interface
132 Voice input element
200 diagram
202 users
204 mobile device
206 website
208 user interface
212 Selectable elements
214 Conversational user interface
218 Selectable elements
222 Selectable elements
224 Computing device
300 system
302 client device
304 application
310 Agent Module
312 User data
314 web browser
316 Conversational interface
320 Automation Assistant
322 Voice-Text Engine
324 Text parsing engine
326 Selectable Element Engine
400 diagram
406 First user interface
408 First selectable element
410 Conversational user interface
412 First agent module part
414 Selectable elements
416 Second agent module part
418 Selectable elements
420 selectable elements
422 Second updated interface part
424 Selectable elements
426 Second user interface
810 computer system
812 Bus subsystem
814 processor
816 Network Interface Subsystem
820 user interface output device
822 User Interface Input Device
824 Storage Subsystem
826 File storage subsystem
830 Main Random Access Memory (RAM)
832 Read-only memory (ROM)

Claims (20)

A method implemented by one or more processors
A step of making the application interface of a non-automated assistant application accessible to the user, wherein the application interface includes a first selectable element for initializing communication with an agent module through the automated assistant application. A step and a step in which the agent module is configured to perform actions related to the non-automated assistant application.
A step of receiving a selection of the first selectable element, wherein the first selectable element corresponds to a link that identifies the agent module and a parameter to be used when performing the action. , Steps and
In response to receiving the selection of the first selectable element, the step of causing the user to present a conversational interface via the automation assistant application, wherein the conversational interface is with the user. A step that is configured to act as an intermediary with the agent module.
A step of providing a second selectable element in the conversational interface via the automation assistant application, wherein the second selectable element is identified within the link to facilitate the performance of the action. Based on the above parameters
How to include. A step of receiving a selection of the second selectable element in the conversational interface, wherein the second selectable element characterizes a value for fulfilling the parameter.
With the step of providing the agent module with the value to assign to the parameter identified within the link in response to receiving the selection of the second selectable element.
The method of claim 1, further comprising. Claims that the conversational interface comprises one or more other selectable elements, each of which characterizes one or more other proposed values, each assignable to the parameter identified within the link. The method described in item 2. The method of claim 1, wherein the second selectable element identifies a utterable command phrase associated with a previous action performed through the non-automated assistant application. The step of receiving the audio data corresponding to the utterable command phrase,
With the step of converting the audio data into text data for transmission to the agent module in response to receiving the audio data.
4. The method of claim 4, further comprising. A method implemented by one or more processors
A step that takes accept the selection of the selectable element in a graphical user interface rendered by non-automated assistant application of a computing device, before Symbol selectable element, said graphical user interface to the associated agent module is the non-automated Steps and steps that indicate that they can be launched through an automated assistant application that is separate from the assistant application.
In response to the selection of the selectable element,
Comprising the steps of start the agent module through the automated assistant application, when the automated assistant application is accessible via at least temporarily the computing device, the agent module, the automated assistant application One of the multiple available agent modules that can be launched through the step and
In response to activating the agent module, comprising the steps of taking accept the response content from the agent module,
By the automated assistant application through the automated assistant interface, the method comprising the steps of providing a based rather output to the response content received from the agent modules. In response to receiving the selection of the selectable element
Comprising: receiving a modality parameter indicating the modalities of a dialog with the previous SL automated assistant application, before SL output, Ru is provided through the modalities of a dialog indicated by said modality parameter, scan Te' flop
The method of claim 6 , further comprising. The method of claim 7, wherein the modality is either voice or text. The selectable element identifies a link specific to the agent module and
The method of claim 6, wherein the agent module is activated based on the content of the link. When pre-Symbol automated assistant application is inaccessible through the at least temporarily the computing device, the link is no start-up of the agent module, a display of additional content relating to the agent module in the graphical user interface causing method of claim 9. In response to receiving the selection of the selectable element
A step of receiving an agent content including an intent parameter for the agent module, wherein the step of invoking the agent module includes a step of invoking the agent module together with the intent parameter indicated by the agent content. The method of claim 6, further comprising. Before Symbol intent parameters are dynamically generated through interaction with the non-automated assistant application, The method of claim 11. The agent module, from the side which provides an automated assistant application is a third-party agent module provided by the specific third party, The method of claim 6. The automation assistant application runs on a computing device separate from the non-automation assistant application.
The output is provided by the separate computing device, The method of claim 6. The output based on the response received from the selected agent module is the prompt
Following the prompting , the step of receiving natural language input,
A step of providing the agent module with additional content based on the natural language input,
A step of receiving further response content from the agent module in response to providing the additional content to the agent module.
Through said automated assistant interface, further comprising the steps of: providing said received from the agent module outputs additional according to a further response content by the automated assistant application, The method of claim 6. A method implemented by one or more processors
A step of displaying selectable elements on a computing device operating a non-automated assistant application, wherein the selectable elements are configured to cause the automated assistant to initialize an agent module associated with the non-automated assistant application. To be done, steps and
The step of receiving the selection of selectable elements and
In response to receiving the selection of the selectable element
The step of determining whether the automation assistant is accessible to the computing device, and
With the step of initializing the automation assistant to communicate with the agent module when it is determined that the automation assistant is accessible to the computing device.
How to include. The step of initializing the automation assistant to communicate with the agent module is
A step of displaying another selectable element in the computing device, wherein the other selectable element corresponds to a link that identifies the agent module.
16. The method of claim 16. The other selectable element identifies the utterable command phrase associated with a previous action performed through the non-automated assistant application.
17. The method of claim 17, wherein the automation assistant is configured to respond to the spoken command phrase when spoken by a user and received on the computing device. When the automated assistant is judged to be accessible to the computing device,
Sending an activation request to the agent module,
In response to the activation request, receiving a response content from the previous SL agent module,
Through an interface associated with the prior SL automated assistant The method of claim 18, further comprising the step of providing an output based on said response content. The step of determining whether the automation assistant is accessible to the computing device is
16. A 16. the method of.

Images