KR20220019734A - Domain configuration platform for virtual assistant - Google Patents

Abstract

A virtual assistant platform comprises a server-hosted platform for selecting a domain to be included in a virtual assistant. In addition, when executed by a grammar for making a request to a plurality of selectable domains and domain providers and a processor in an embedded system independent of the platform server, the virtual assistant platform also comprises a means for exporting a code that allows an interpreter function to respond to the queries for the local domain.

Description

Domain Configuration Platform for Virtual Assistants

More and more people are using SoundHound Hound, Google Assistant, Amazon Alexa, 百度度秘, Apple Siri, Line/Naver Clova. Clova), Microsoft Cortana, KT Giga Genie, SK Telecom NUGU, and the open source Mycroft. First-generation virtual assistants were only available on computer, smartphone, or home speaker devices branded by companies that offered virtual assistant capabilities. They were not configurable and only supported custom capabilities developed internally by the company.

Current virtual assistants may be text counterparts. They can understand written natural language text and can compose responses written in natural language. Some embodiments are voice response. They understand spoken natural language and may respond in forms such as text, synthesized speech, images, graphic animations, audio and video clips, and the like.

Companies that create and sell applications (apps) or devices, such as cars, robots, shopping kiosks, and appliances that include virtual assistants (“VAs”), create VAs “from scratch” on their own. You don't necessarily want to invest in something. They want to seamlessly integrate pre-existing natural language functionality as part of their own custom virtual assistant. They found it frustrating not being able to specifically configure their virtual assistant to suit their own requirements.

A domain is a unit of natural language functionality. They are characterized by the set of sentences they can interpret, and by their ability to respond to and act upon the interpretation of the recognized sentences. Exemplary domains include a weather domain that answers questions about the weather and a calendar domain that allows users to ask questions as well as make and manage appointments, ie, request actions.

For example, some VA development platforms (“Platforms”), such as SoundHound Houndify (“Platform”), assist developers of virtual assistants (“VA Developers”) by providing a menu of domains available on the development platform. . In such platforms, VA developers select domains for inclusion in custom virtual assistants of their own design, including an array of domains already known to the platform. Some embodiments make it possible to use such a platform to configure a virtual assistant. Some embodiments include such a platform. In some embodiments, the platform provides an interface through which domain providers will provide their domains for integration into the virtual assistant. The information provided by the domain provider is sufficient to support the functionality of the domain in the virtual assistant selecting the domain. In some embodiments, some domains are free. In some embodiments, a given domain has an associated fee for its use. In various embodiments, payment is made to the domain provider by the platform provider, or by another party, such as a VA developer, user, or VA developer, through the platform provider. In some embodiments, the platform provider promotes certain domains over others.

Some embodiments enable VA developers to enter test queries to the virtual assistant and test how the virtual assistant will behave in response to these queries. Some embodiments make it possible to show and compare the behavior of the virtual assistant under different conditions in different domains selected from a menu of domains.

Some embodiments enable VA developers to enter large sets of test queries. Some embodiments display a view of the analysis of the virtual assistant's behavior on a set of test queries and in response to a selection of domains. Some embodiments enable cost analysis associated with answering all queries in a set. Some embodiments enable the analysis of a portion of the test query in a set that the virtual assistant can interpret.

Some embodiments provide a display view for analyzing the rate and cost of successful interpretation of queries in a set under the conditions of different selections of domains. Some embodiments provide a tabular view of domain results. Some embodiments provide a pricing model for each domain. Some embodiments provide typical queries recognized by each domain. Some embodiments provide a set of test queries and a total cost for responding to queries from the set of test queries. Some embodiments provide a cost stack view of the cost for each domain for answering queries in the test set. Some embodiments provide a histogram view of queries answered per domain. Some embodiments indicate that some domains are free and others have a cost.

Some embodiments provide a graphical user interface with a pointer icon. In some such embodiments, when the pointer is positioned over a certain portion of the display view, the corresponding information is displayed in a pop-up message.

Some embodiments make it possible to sort or filter a menu of domains according to useful criteria. Some embodiments make it possible to display different domains in a menu in different ways, such as background color, text color, font, and typestyle.

Some embodiments are parameterized for a domain, such as a linear pricing model, a piecewise linear pricing model, or a formula-based pricing model. It enables a pricing model. Some such embodiments provide tools to assist VA developers in making cost-effective selection of domains.

Some embodiments are computer readable media. Some embodiments are cloud servers. Some embodiments are mobile devices. Some embodiments are systems of devices that communicate with a cloud server. Some embodiments are self-contained devices that implement a natural language virtual assistant configured by the platform.

1 illustrates the flow of information from a domain provider, through a platform provider, through a VA developer, and to a user, in one embodiment.
2 shows a set of test queries.
3 illustrates a cloud-based system of domains, platforms, and virtual assistants, in one embodiment.
4 illustrates, in one embodiment, a single device system in a domain, and a virtual assistant configured from a platform.
5 depicts a user interacting with a virtual assistant that, in one embodiment, interprets queries and provides responses according to domains.
6 illustrates a VA developer using test queries to guide the construction of a natural language virtual assistant through a platform promoting a domain, in one embodiment.
7 illustrates a complete system with a querying user, a configuring VA developer, and a domain provider providing information to the user, in one embodiment.
8 is their pricing model, indicating how many of a set of test queries will be answered by each domain, and a test set, which makes it possible to select which domains to enable, in one embodiment; shows display of a menu of domains showing the cost per domain for answering queries in
9 shows a cost stack chart for selection of enabled domains for a set of test queries, in one embodiment.
10 shows a running cost chart for selection of an enabled domain for a set of test queries, in one embodiment.
11 shows a histogram of queries from a set of test queries answered by each of a selection of enabled domains, in one embodiment.
FIG. 12 shows a histogram of query responses per domain, highlighted under the hovering pointer and showing domain-specific information, in one embodiment.
13 illustrates, in one embodiment, a display of a menu of domains with promotional domains prominently shown.
14 shows a display for entering a test query and viewing a filtered menu of domains that can respond to the query and their responses.
Fig. 15(a) shows a linear pricing model.
Fig. 15(b) shows a piecewise linear pricing model.
15( c ) shows a formula-based pricing model.
16( a ) illustrates a rotating non-transitory computer-readable medium.
16( b ) illustrates a packaged solid state non-transitory computer readable medium.
Fig. 17(a) shows a rack of servers.
Fig. 17(b) shows a block diagram of the server.
18( a ) shows a packaged system-on-chip.
Fig. 18(b) shows a block diagram of a system on a chip.

The following describes various embodiments of the invention illustrating various features. In general, embodiments may use the described features in any combination.

Many companies make or sell devices or apps that can be enhanced by integrating virtual assistants. Many virtual assistants are voice-based. Examples of speech-enabled devices are automobiles, robots, shopping kiosks, appliances, and smartphones. All of them can be equipped with an integrated virtual assistant. A user communicates with the virtual assistant by issuing a natural language query, typically either a request for information or a request for an action. A query may be spoken or written. The response from the virtual assistant may include an action as well as a voice, text, graphic, audio, or video recording.

The development of intelligent multifunctional virtual assistants is complex and requires investment and expertise in all domains that virtual assistants can handle, in addition to the general architecture. When developing a virtual assistant, it makes sense for the developer to purchase or license domain software from a virtual assistant development platform provider. This allows the platform provider to hire enough experts to build a good platform and share the cost among the various VA developers.

In such an ecosystem, the logic or software for implementing the virtual assistant is stored on a computer-readable medium in a network-connected server operated by a platform provider or in a device that does not need to be connected to a network, the device logic or software comprising: It was built using the platform. In a typical network-connected embodiment, a client app or remote device accesses the server, via the Internet, using an application programming interface (“API”). Some single device virtual assistants operate autonomously, some operate only through network access, and some operate over intermittent network access by combining local and remote virtual assistant capabilities. Whether connected to a network or not, the virtual assistant receives natural language queries from users, interprets queries, and provides responses to users. Zero or more than one domain may recognize each query. In case of ambiguity, the virtual assistant chooses one of the competing domains and determines its response.

Virtual assistant developers are people (in most cases, experienced engineers) who work for a company that integrates virtual assistants into the company's specific devices and apps. VA developers use the platform to configure virtual assistants to their specific requirements. Virtual assistants typically support multiple domains. Some domains provide information such as weather, news, trivia, restaurant searches, simple math, and Wikipedia research. Some domains provide services such as controlling a thermostat or lighting, streaming music, sending SMS text messages, and managing calendar appointments. Some domains can be as simple as just telling the time. Some domains can be complex, such as providing travel agency services, and some are highly valuable to users, such as the ability to locate children. Many domains access web APIs to access specific or dynamic information or cause a requested action. Some domains provide information and services from third-party providers in exchange for payment. For example, some weather domains access the national weather service API to obtain weather report information.

The platform provides, among other features, a way for VA developers to select a domain for inclusion in their virtual assistant. A domain is typically provided by a domain provider entity. Some platforms support thousands of third-party domain providers. Some domain providers are small companies or even individual programmers.

In some embodiments, the set of queries recognized by the domain is defined by a semantic grammar code. Such grammar code is specifically generated for each domain by a domain developer (in most cases, an experienced engineer) who creates rules for interpreting a query as a specific request for information or a specific request for an action. . In such an embodiment, the natural language interpretation is based on the domain grammar code. The domain grammar includes syntax rules used by the natural language processing system to parse the query. In domain grammars, syntactic rules are augmented by semantic augments. An extension is the ability to construct the interpretation of a constituent from the interpretation of zero or more sub-constituents, where the sub-constituents are determined by parsing the syntax. In some embodiments, domain grammars are used to recognize and interpret queries for domains. In some embodiments, at least a portion of the interpreter includes a neural network that is trained using machine learning. In either case, the interpretation of the query assigns values to semantic slots, based on information from the query. Some embodiments encode intent slots that express the user's overall intent for a query, such as scheduling a meeting or booking a flight.

In an exemplary embodiment, the semantic grammar for requesting a weather notification is: (1) its syntax component is named <location> with a corresponding semantic slot named Location, and its value A component that must qualify as a location in all these instances; and (2) another component whose syntax component is named <time> with a corresponding semantic slot named Time, whose value must qualify as a time. will be. When the user asks "what's the weather in Timbuktu tomorrow", this embodiment may use a weather domain grammar with <location> and <time> components, and set the Intent slot to the value "weather_info" ", the Location slot with the value "Timbuktu", and the Time slot with the value "Tomorrow", or possibly the corresponding calendar date.

Similarly, if the user says "which is the closest French restaurant that has received at least 4 stars", the system can use the restaurant domain grammar with syntax rules for the <cuisine_type> and <star rating> components, and set the CuisineType and StarRating The corresponding slots for , will be filled with values passing "French Cuisine" and "At least 4 stars" respectively. As another example, if the user says "I love you mom", the system can recognize the query using the SMS texting domain grammar, set the intent slot to "Send_Text", and the Recipient side) slot, a reference to the contact corresponding to "Mom" in the user's contact list, and the Message slot with "I love you". The interpretation of the query will be constructed using these slots, and the implementation of the interpretation of the query specified by the intent will be sending the textual content of the message to the mobile phone of the receiving party. If the user says "how many claws does the cat have", the system may recognize a knowledge domain query, set the intent slot to (for example) "WikipediaLookup" and the Species slot You could also fill the Attribute slot with "Claws" and the Question slot with "How many". Other approaches to natural language are possible.

Some queries can be interpreted in multiple domains. It is common to have such overlapping grammars when different domain providers compete to provide the same domain functionality. For example, a platform might offer four competing domains for making hotel reservations, all of which could interpret the query "show me hotels in Paris". To help VA developers handle this situation, the platform can configure the virtual assistant to include certain domains but not others, or to prefer certain domains over others to the performance of the virtual assistant. there is. Such selection may be made in a "hard" manner or in a "soft" manner.

A hard choice is made at configuration time. Some domains are enabled in the virtual assistant and all remaining domains are disabled. This configuration step allows the VA developer to control which domains are involved in the interpretation of queries at runtime. At production time, query resolution only considers enabled domains.

Soft selection is made at runtime. The choice between competing domain resolutions of a query (where all domains are enabled) is typically made by selecting the highest scoring interpretation based on a scoring scheme. Scoring techniques may be used to give priority to domains. That is, when enabled domains A and B compete to interpret a query, domain A may always be given priority over B. Scoring can also resolve cases of ambiguity between unrelated domains, although such semantic conflicts are less frequent. For example, the query "how high is Denver" could be interpreted by the geographic fact domain knowing the elevation of the city, however, it would also be interpreted by the weather domain where "high" indicates the temperature. can The second interpretation might not be possible without context, however, it is quite likely in the context of weather and temperature during a conversation, such as "How's the weather in Chicago? [Answer: 56 degrees] How high is Denver?" Parsing and interpretation scores may be context sensitive, and scoring may take into account domain priorities as well as all of them.

For effective configuration selection, the platform may provide an empirical tool intended to assist VA developers in making informed decisions. In some embodiments, the cost effectiveness of a domain can be measured in terms of the following with respect to a test set: (1) coverage, ie, the proportion of queries recognized by the domain; (2) accuracy, ie, the proportion of recognized queries that are correctly interpreted by the domain; (3) cost consideration.

Regarding cost, according to some embodiments, the platform knows the pricing model associated with each domain and is able to provide pricing information to the VA developer. Many platforms offer some domains for free, such as those that tell the time or solve simple math problems. However, many domain providers offer high-value domains and want to be rewarded for using their data and services. For example, the domain for stock trading may charge a fee for each stock transaction requested by the user. In some virtual assistants, the user pays the VA developer for using the domain, and the VA developer pays the domain provider directly or through the platform. In some cases, platforms may raise the cost of domains to make up for the value they add.

Indeed, platform or VA developers may discount or subsidize certain domains to attract more customers. In a pricing model, the price per query is often subject to a volume discount. All such considerations can be part of a pricing model for a domain.

Coverage and accuracy are measured based on a set of test queries received by the platform from the VA developer. When an enabled domain recognizes a query, a "hit" is registered. If a test query hits at least one enabled domain, it is "covered". Coverage is measured by the number of test queries covered (absolutely), or by the corresponding portion of the test queries (in a relative sense). Accuracy is measured either (absolutely) by the number of correctly handled test queries, or (in a relative sense) by the corresponding portion of the covered test queries. When the test set includes query interpretation data, or when it includes query data, accuracy can be measured.

In some embodiments, test accuracy requires interpreting the test query as if it would be run against a user query in production, but does not require that the test query be executed (implemented). Accuracy is based on equivalence (or matching compatibility) of interpretations (which are internal representations of the meaning of the query). In some embodiments, test accuracy requires both interpreting and fulfilling the test query. The correctness is then based on the equivalence (or matching compatibility) of the responses.

In some embodiments, the VA developer may enter a test query, as if received from a user, and the platform may indicate to the VA developer one or more of the following: (1) which domains may recognize the query ; (2) interpretation of queries; (3) the information necessary to perform the interpretation and answer the inquiry, and (4) the cost of providing the response.

In some embodiments, the VA developer inputs a large set of test queries, and the platform provides statistical information about the domains that the set of queries will hit and the cumulative cost for the test set. In some embodiments, a query may hit multiple domains. In another embodiment, the system guarantees that only one hit per query. One type of statistical information is a histogram of hits per domain across a set of queries. Another type of statistical information is a table with one or more of the following: domain names; whether each domain is enabled for a virtual assistant; a pricing model for accessing each domain (eg, simple cost per query); the number of queries in the test set that hit each domain; the resulting cost attributable to having the domain enabled in the virtual assistant; the portion of the query in the test set to hit with at least one of the enabled domains; and total cost for handling all queries in the test set, using the enabled domains.

It is possible for some queries to request information from more than one domain. For example, if the query "what's the weather in the home city of the team that won the last Super Bowl game", the virtual assistant uses the sports information domain to determine which team won the last Super Bowl game and their You will need to find your home city, and then you will use the city information to look up the weather information using the weather domain. Therefore, the number of domains hit by the virtual assistant responding to a set of test queries may be greater than the number of queries, and the cost of the queries may be higher than if each test query hits only one domain. .

Some queries are not recognized by any domain. In some embodiments, such failed queries are answered along with default information, such as the results of a web search. Consequently, the number of hits may also be less than the number of queries.

In general, the more domains a virtual assistant has, the more queries it can answer in a satisfactory manner. The more users satisfied with the virtual assistant, the more users will use it. The more users use the virtual assistant, the more incentives the domain provider should create and improve the domain. This creates a virtuous cycle of ever-improving domains and virtual assistants that benefit users and generate profits for all participating companies.

The following is a description of certain representative embodiments illustrated in the drawings.

party

1 shows a diagram of a virtual assistant ecosystem and its operation. Different domain providers provide their information and services to the platform 12 . Specifically, the weather information domain provider provides a weather domain 11a; The map information domain provider provides the map domain 11b; The text messaging domain provider provides the text messaging domain 11c to the platform 12 . In addition to the domain grammar, the information provided to the platform may include a pricing model and test data.

VA developers use platform 12 to create virtual assistants such as appliances 13a , mobile phones 13b , and cars 13c , all of which provide VA capabilities to multiple users 14 . . Some users use more than one kind of virtual assistant. Information and services generally flow from left to right to reach users from domains, through platforms, and through VAs.

Generally, payments flow in the opposite direction. In various embodiments, the user pays directly to the VA developer, platform provider, or domain provider. In various embodiments, the VA developer pays directly to the platform provider or domain provider. In some embodiments, the platform provider pays the domain provider. In some embodiments, the platform provider offers virtual assistant developers a discount or at a loss access to several domains to succeed in business or to increase overall usage. In some embodiments, the domain provider pays or provides a discount to the platform provider in exchange for the platform provider to promote their domain. The techniques disclosed herein do not limit the possible business relationships using the framework.

In some embodiments, the company is both a domain provider and a virtual assistant developer. For example, a car's virtual assistant needs its own local domain to respond to status queries about the car, such as the car's fuel level, or to request an action, such as turning on headlights. This is, perhaps, in addition to including an external domain, such as a domain with weather information. As another example, video player set-top boxes sold by certain video content providers, as part of their custom virtual assistant, require access to a domain for retrieving their video content. This domain may be local or external. In some embodiments, some domains are exclusive to some virtual assistants and other virtual assistants are unavailable.

2 shows an example of a set of typical queries for a general purpose virtual assistant. Although queries about the weather are most frequent, other common queries, however, include searches for local business, map directions and traffic conditions, queries about news or sports, queries about trivial facts, and various other types of queries. is a question Since each query can appear multiple times in the test set, and the order is not important, an alternative expression, known as a bag, is to associate each unique query with a count. For example, the query "how's the weather" has a count of 3 in the set. Taking this one step further, each query can be weighted with a real value (rather than just a frequency count) that represents its importance to the experimenter compared to other queries. Such weights can be normalized to have a sum of 1.

Cloud and devices

3 shows a user 31 communicating with a first app 32 and a second app 33 . The app communicates with the platform 35 via a network 34 . App developer 36 previously used platform 35 to configure a first virtual assistant for first app 32 and a second virtual assistant for second app 33 . In some embodiments, different developers configure each app.

On platform 35, multiple domains are available for selection in configuration. The number of available domains can be hundreds, thousands, or even more. Domains are provided by domain providers. They are made available on the platform through a domain registration or ingestion process. During registration, information supplied for a domain includes a unique ID, short name, description, examples of recognized queries, and domains in executable or interpretable form, such as source code, object code, neural networks, or semantic grammars. You may. In this disclosure, the term "grammar" or "semantic grammar" generally refers to a description of code, such as a source, object, neural network, or any data structure instance, that can be executed to parse or recognize a query.

The virtual assistant configuration created by the developer 36 includes the selection of a first domain 37 and a second domain 38 . When the user 31 makes a query to any one of the apps, it is a platform that interprets the query and retrieves appropriate information from either the first domain 37 or the second domain 38 using a web API. Send the query to (35).

4 shows a user 41 communicating with an app 42 incorporated into a device 44 that operates independently of the platform 45 . The drawings are not drawn to scale. App developer 46 previously used platform 45 to configure a virtual assistant for the app. The platform provided the software needed to integrate the App Integrator configured virtual assistant into the App 42 .

When the app 42 receives queries from the user 41 , it interprets them using the integrated interpreter and, accordingly, requests information from the local information domain 47 or service action from the local service domain 48 . to request For example, in an automotive device, information domain 47 would have a grammar, providing information for queries such as "how much battery power is left" and "what radio station is this". In an automobile, service domain 48 will have a grammar and provide services such as "open sunroof" and "turn on radio". These exemplary information sources and services are automotive specific and do not require access to a network.

Some embodiments have configurations that allow for a mix of domains that require and do not require network access. Such an embodiment enables a domain to respond successfully when network access is available and to respond unsuccessfully when network access is not available.

Information flow and organization

5 shows a diagram of data flow through a platform at runtime, according to an embodiment. User 51 issues a natural language query for the platform to process. The interpreter 52 interprets the query and provides a response to the user 51 . Depending on the query, the response may be a simple indication that the interpreter 52 cannot interpret the query. In some embodiments, interpreter 52 may only provide responses to queries that it successfully interprets.

Domain providers provide data from their own servers such as music sources or other sources such as weather sensors or services such as message transmission or vehicle control. Generally, such data or services are known as domain 53b. To know when a user intends to call a domain, they may have a corresponding grammar code 53a. The grammar code contains slots that can be filled with words from user expressions and phrasings that call domains. For example, a grammar code with the expression "how's the weather" calls a weather domain, and the domain code includes slots for the location and time from which the weather forecast was retrieved. The code for the slot and representation of the weather domain enables the interpreter to know that the phrase "what's the weather in Timbuktu tomorrow" should make a request to the domain provider about the weather tomorrow in Timbuktu. Code associated with a domain, in many embodiments, is registered, collected, and stored by the platform.

If interpreter 52 interprets the query according to grammar code 53a associated with domain 53b and determines that the query requests information from domain 53b, interpreter 52 contacts one of the plurality of domain providers. It requests information from the appropriate domain 53b provided by the Interpreter 52 uses that information to form its response to user 51 .

6 shows a diagram of data flow through a platform upon configuration, according to an embodiment. Among many app integrators, a developer 66 provides one or more test query inputs 64 to the platform. Developer 66 also provides a selection of one or more domains 65 . The interpreter 62 interprets the test query according to the grammar code 63a of the selected domain 63b to determine whether any of the selected domains 63b can interpret the query. The display displays the results from the interpreter 62 to the developer 66 (67). Many display views are suitable and useful for various embodiments. Some display views will be illustrated and described below.

The embodiment of FIG. 6 further includes promoting one or more domains 68 based on the results of the interpreter 62 . For example, if the test query can be successfully interpreted according to the promotional domain, the platform shows to the developer 66 the result that the interpreter will show the user when the promotional domain is enabled. Some embodiments show their names, even when promotional domains are not enabled. Some embodiments show responses to be generated with information from the promotional domain as an alternative to responses that will be generated with information from the highest rated alternative domain.

For example, a weather domain provider with more accurate and detailed information may charge a higher price and provide itself as a promotional domain. A default weather domain may generally respond to weather queries. The platform according to the embodiment will show the response according to the information from the default weather domain, and it will show the response to be generated with the information from the promotional weather domain. This may be done with an offer to select a publicity domain and even a means for selecting a publicity domain for inclusion in the virtual assistant, eg a button icon.

7 shows a diagram of data flow through the platform. Developer 76 provides one or more test query inputs 74 to the platform. Developer 76 also provides a selection of one or more domains 75 . The interpreter 72 interprets the test query according to the grammar code 73a of the selected domain 73b to determine whether any of the selected domains can interpret the query. The display displays the results of the interpreter 72 to the developer 76 (77). The embodiment of FIG. 7 further includes promoting one or more domains 78 based on the results of the interpreter 72 . After the configuration time, during runtime, the user 71 issues a query that the interpreter 72 will interpret and provide a response to the user 71 .

In the embodiment of FIG. 7 , in response to receiving information from domain 73 , the platform calculates a fee 79 . Fees are calculated according to a pricing model. The pricing model is a function based on domain usage. Some embodiments have a simple pricing model, such as a single fee for each information access. Such charges are usually a small fraction of one US dollar. Some embodiments support more complex pricing models, which are functions based on the type of information accessed or the number of previous information accesses within a window of time. Equally, when other domains provide information, some domains charge for providing the service.

7 , the platform provides a payment to each domain provider providing information to the virtual assistant to construct a response to the user. This creates an incentive for domain providers to want developers to choose their domains for inclusion in the virtual assistant, and thus incentives for platform providers to promote domains of domain providers with which the platform providers have close business relationships. .

domain table

8 shows a display view of a domain table 80 according to an embodiment. Table 80 has one row per domain. The "Enabled" column indicates the status of domain enablement. In some embodiments, cells in this column may be used to enable or disable the corresponding domain. Various input means for selecting or deselecting domains are possible, such as a mouse for clicking, a keyboard with arrow keys or shortcut keys, a touch screen, or other suitable means for making selection of an object in the user interface.

Many different graphic layouts are possible. In general, relevant pieces of information about domains, such as their names, whether they are enabled, the number or percentage of queries they can interpret (out of the test set), and their pricing models (such as the simplest Grids are useful for visually organizing a large number of domains, as rows, and relatively few columns, to display a cost per query (in some cases). Visual objects that show related information, such as rectangles representing domains aligned in a grid, also work. It is also possible, for example, for objects to be arbitrarily dragged and dropped on the desktop of the personal computer.

Table 80 has a display view for showing the VA developer which domain has been selected from the menu of domains for inclusion in the virtual assistant. Display view 80 has five columns. The heading row is titled by its use in each column. The columns are: the name of each domain listed from the menu of available domains; whether the domain is enabled for the virtual assistant; a count of the number of queries in which the domain is a domain for the interpreter to select from among all enabled domains capable of interpreting each query in the test set; pricing model for answering queries; and the total cost for the domain to provide the information needed for the virtual assistant to provide a response when the domain is selected to respond to each query in the test set.

After the heading row, there is a row for each domain in the menu. Table 80 includes seven domains in the menu. Some embodiments support an even greater number of domains, a variable number of domains, and domains that dynamically change as new and existing domain suppliers create or remove their supply. The first column represents the menu's text string with the name of each domain in the menu. The domains represented are weather, food, sports, maps, facts, timers, and math.

The second column has a cell for each domain for which a tick mark is made visible when a domain is selected for inclusion in the virtual assistant. 8 shows a pointer that can be controlled by a mouse or a finger on a touchscreen. A click or double click of a mouse button or a tap or double tap or long press of a finger changes the state of whether the domain at the pointer position is selected or not. A state is indicated by the presence or absence of a check mark in the cell.

A third column represents a cell for each domain, which cell, given the selected domain, contains a count of the number of queries answered using information from the domain. In some embodiments, different columns are used for counts of queries recognized and counts of queries answered correctly when the correct query responses are known. In some embodiments, percentages are displayed instead of, or in addition to, absolute numbers using additional columns. Moreover, some queries are ambiguous about the set of domains being selected, ie they can be interpreted by more than one domain. This is most common for short queries. For example, the query "turkey" may be interpreted by a weather domain, a food domain, a sports domain, a map domain, and a facts domain. Some embodiments handle ambiguous queries by providing labeled responses to all ambiguous interpretations of the query.

Another embodiment more robustly removes ambiguity by using domain priorities and selecting the highest priority domain that matches the query. In another variant, a query interpretation score is used, and the domain with the highest scoring interpretation score from its corresponding grammar wins and forms the basis for the response. The two ideas can be combined. In all cases, ambiguity is removed by more robustly choosing one or fewer interpretations that contribute to the count of a single domain. If the state of the domain's selection changes, this may change the number of queries to which other domains will respond. Some embodiments dynamically recalculate the query count count whenever the selection of domains is modified. Alternatively, or in addition to, the query count column of FIG. 8 , some embodiments provide a total number of test queries for which each domain can provide information and an increment of test queries that can be answered correctly using the enabled domains. Represents both incremental numbers.

Column 4 shows the pricing model for accounting for domain queries. Table 80 represents a simple kind of pricing model based on a fixed price per query. The prices shown range from 0 (for free domains) to 5￠ ($ .05 US dollars) per query. Some embodiments support more complex pricing models, such as piecewise linear or formula-based models as a function of the number of hits within a time window. A recurring time window (eg, a calendar month) as well as a sliding time window (eg, the past 30 days) may be used. Embodiments that support more complex pricing models may not indicate the pricing model itself in the Pricing Model column; Instead, a click or tap on a pricing model cell for a domain may give access to a different display view, such as a pop-up window, that represents the domain's pricing model and allows editing it.

Column 5 indicates, for each domain, the amount of money to be paid to the corresponding domain provider, based on responses to all queries in the query's test set. Billing is done for all domains whose information is used in the response. For the simple pricing model shown in table 80, the domain cost for each domain is the query count multiplied by the cost per query. In embodiments that allow for complex queries, a single query may hit multiple domains. For example, "How old is the President of the United States" might hit both domains providing political information and domains providing personal information. In embodiments that allow for ambiguity in responses, a single query may hit multiple domains for different reasons. Multiple responses are given depending on competing interpretations. For the same reason that the query count may change each time a domain is selected or deselected, the domain cost may also change accordingly.

The lowest complete row in table 80 represents the number of queries that do not match any domain. Such queries can be counted, however, other columns are not applicable.

Below that, the row lists two tally. The first indicates the proportion of queries that receive responses in the current selection of domains. In table 80, the food domain and the map domain are not enabled. Each of them was able to respond to 1955 and 764 queries, respectively. With them, and 633 queries that cannot be interpreted by any domain, 73% of test queries successfully receive responses.

The last aggregation cell displays the total domain cost for the enabled domains. This number changes dynamically as VA developers enable or disable domains. In general, the more domains a VA developer enables, the more successful queries will be and the higher the total cost will be. Because higher quality success rates generally provide a satisfactory user experience, such dynamically informed display views allow VA developers to make an informed trade-off between user satisfaction and budgeting for domain usage. allow to do

In all of the above methods, a test query or group of test queries may be given multiplicity. Returning to FIG. 2 , which shows a set of test queries, note that, for example, the query “how is the weather” occurs 3 times. In a production environment, VA developers may have access to field statistics for a much larger set of data, and monthly counts for "how's the weather" can be in the millions. Having a large number of queries on the same domain makes a non-linear volume discount a reasonable business arrangement. This is especially true if the platform caches results for static or infrequently changing data. Alternatively, in the absence of full-blown statistics from the field, a small test set may be based on: (1) statistics from a small sample of queries; or (2) a curated set of queries, possibly inspired in part by data from the field. In either case, the test set counts can be multiplied by a variable factor that the VA developer can change to observe the effect of the non-linear buy-in discount.

Some embodiments, particularly those with a large number of domains, provide scroll bars that allow VA developers to view different domains within a limited display space. Some embodiments allow a VA developer to sort a list of domains according to one or any number of column criteria. One way to do so is by receiving a click or tap on a column, causing the list to be sorted. Moreover, clicking on a column for which the list has already been sorted causes the list to be sorted in the reverse order. Some embodiments determine whether each domain is enabled or not; range of pricing models; Provides a box for VA developers to enter filter criteria, such as to filter the list of domains by range of domain cost:

Every virtual assistant is a different project for VA developers. In some embodiments, each project can be individually opened or viewed and will specifically display a table of domains for a particular virtual assistant. Some embodiments of the platform allow VA developers to create an account and log in to configure their virtual assistant project. The platform administrator can control which domains are visible to different VA developers and which controls and tools VA developers can access to work on the project as a function of the terms of the account.

running cost chart

9 shows a display view of a running cost chart 90 according to an embodiment. The chart has the title "Cumulative amount consumed per domain". The chart has a vertical axis for cost and a horizontal axis for domain. Some embodiments plot domains along a vertical axis and costs along a horizontal axis. The former is generally better for virtual assistants where a small number of domains bear the majority of the cost, and the latter is generally better for virtual assistants with a large number of domains.

There are many useful ways to present visual data. For example, Edward Tufte's books, such as Envisioning Information and The Visual Display of Quantitative Information , provide information on the subject. Drawing black and white lines is not ideal, but may be sufficient to illustrate. In the embodiment shown in the black and white line imaging of FIG. 9 , seven domains are depicted on the horizontal axis. To improve readability without overlapping words, the domain names are slanted and right-aligned with the axis. The names of the enabled domains are shown in bold font and the names of the disabled domains are shown between parentheses and written in a smaller font. A domain that is free is indicated by "(f)" after the domain name. Domains that do not require access to external dynamic information are usually free because platform providers can cache the information and retrieve it at low cost.

In the column for each domain, a bar with a height proportional to the domain's cost to respond to a set of test queries represents the domain's contribution to the total cost of responding to all queries in the set. The bottom of the bar for each domain is placed at the height of the top of the bar in the previous domain. Domains with no cost have zero-height bars appearing as horizontal lines. The vertical axis is at the value of the cumulative cost and the height of the top of each bar, labeled with a horizontal dashed line extending from the vertical axis to the bar at the height of the label. The platform allows VA developers to identify available domains by various criteria, such as by name in alphabetical order; by the number of queries they match; by their cost per query; or by their total cost, making it easy to change the order of domains, by automatically sorting them. Also, the platform's interface allows the user to fix the order from a particular sort, but the selection will change, otherwise the order will change. It is also possible to manually rearrange domains. Some embodiments also make it possible to filter the list of domains by various criteria, such as by indicating enabled domains, or only selected disabled domains.

The display view of FIG. 9 also includes a pointer that, when clicked or tapped or otherwise activated as appropriate, toggles the state of whether the domain is enabled or not. Thus, VA developers can quickly and easily enable and disable domains to see their effect on the total cost of answering queries within the test set. FIG. 10 shows a display view of a running cost chart 100 for the same virtual assistant as that of FIG. 9 after the disabled food domain is enabled by clicking with a pointer over ((Food)). .

In Fig. 10, domains are sorted by the number of queries that hit the domain. Although the food domain (since it has a higher cost pricing model) is a larger contributor to the total aggregation cost to respond to the test set than the weather domain, since the test set has fewer food queries than the weather , it is shown second in the list.

Some embodiments display the cost axis on a linear scale and some embodiments represent it logarithmically. Some embodiments show an axis on the right side of the chart that represents the cumulative number of queries (or fraction or percentage of the test set) answered by the enabled domains. Such an axis will necessarily have an irregular scale if the cost scale is regular (linear or logarithmic). Alternatively, a regular (linear or logarithmic) display of the queries being answered may have a corresponding parallel axis to cost with irregular spacing.

Query Hit Histogram

11 shows a display view with a histogram 110 of query hits per domain, according to an embodiment. The histogram has the title "Histogram of queries hitting domain" with a vertical axis of hits and a horizontal axis representing domains. A domain hit refers to a query that is answered with information from the domain. The histogram 110 of FIG. 11 does not label the domain axis with the names of domains because there are too many domains that can fit into names. Alternatively, domains can be plotted on a vertical axis and hits can be plotted on a horizontal axis. If the histogram chart is long enough or allows the VA developer to scroll through its entries, it is possible to list domain names and indicate the number of hits.

The histogram 110 has smooth lines, but a cascading representation across the domain will also work. Although the histogram 110 does not label the horizontal axis with the names of domains, it does show a “$” symbol to indicate domains that have pricing models that exceed a threshold. In some embodiments, the threshold is zero (indicating free and prepaid domains). Some embodiments enable VA developers to configure thresholds to see affordable versus expensive domains.

In some cases, even domains with a small number of hits can provide disproportionate benefits to the user experience. For example, the domain available in a car virtual assistant that can locate a lost phone will seldom be used, however, in the rare case it is needed, users will appreciate it very much. That would be a strong selling point for the car. Even if accessing that domain information is very expensive, per usage, for a VA developer, it will be well worth the cost, in order to sell more cars.

Some embodiments allow VA developers to tag certain domains, eg, with a star symbol, as being of particularly high value to the user experience. Some embodiments show histograms as in FIG. 11 , where high value domains are highlighted with prominent colors or icons.

Because many queries can be interpreted by more than one domain, enabling or disabling a single domain can change the number of queries answered with information from all other domains. It can have the effect of rearranging the domains within the histogram. For example, two competing weather domains may be enabled, the first may provide all weather information including barometric pressure and the second the default source of information for all weather queries, but providing barometric pressure information at the time. If not, with both domains enabled, the first will have very few hits (and will be shown near the right side of the histogram 110 ) but with the second domain disabled The first one will have a very large number of hits.

FIG. 12 shows a display view of the histogram 110 as in FIG. 11 but with a pointer. When hovering over a column of domains in the histogram, after 1 second without movement, the platform returns the name of the domain, the number of hits it has for the set of test queries, and the number of hits for the domain to be enabled when responding to the queries in the test set. Pops up an information box indicating the total cost.

domain promotion

13 shows a display view with a domain table 130 according to an embodiment. Similar to the domain table 80 of FIG. 8 , the domain table 130 of FIG. 13 lists domains. The difference is that the domain table 130 represents two map domains, one named “map” and the other named “mapPRO”. The latter is a public relations domain. At least because it can provide information about 805 queries instead of 764 queries on the test set, it has the potential to provide a better user experience than the map domain. The platform displays the text of the mapPRO domain and its lines in bold font, with a highlight around its enable checkmark box. This makes the publicity domain prominently visible to VA developers for consideration.

Although the embodiment of Figure 13 uses bold fonts and highlight boxes, various methods for highlighting promotional domains are possible. For example, one or more promotional domains may: be individually listed; can be sorted to be at the top of the list; may be represented using different colors or transliterations; may be represented with an icon such as a star; may be presented with an image such as a company logo; Or it can be animated. Some of these means of highlighting domains can be used in combination.

14 illustrates a display view 140 for entering specific test queries and viewing results in accordance with embodiments. The display view 140 includes a text input box 141 . The virtual assistant developer may enter text using a physical keyboard or virtual keyboard or other suitable text input box. The entered text appears at the position of the cursor position indicator 142 . For user interface devices with a microphone, the VA developer may tap or click the microphone button 143 to speak a test query. A guidance item appears on the display when it is first called to notify the VA developer that "speak your query" is available.

Display view 140 also includes an upload button 144 . When activated, it invokes a dialog box for the user to browse a file and select it to be uploaded to the platform. A file can contain a single query or a delimited list of any number of queries, including the entire test set. Some embodiments enable VA developers to drag and drop files from their graphical operating system or browser display into a query box and have the files automatically uploaded to the platform. Various implementations of upload buttons and drag-and-drop uploads are well known and readily available in browser client-side script templates in languages such as Javascript.

Display view 140 further includes results box 145 . For a given single input query, the resulting box displays a list of domains that can provide the information needed to answer the query, the name of the domain, the pricing model (labeled "Cost"), and the domain With this enabled, the virtual assistant will display the query along with a column of responses to provide to the user. In some embodiments, multiple domain providers may provide source information and grammar to form very different responses to queries. Results box 145 shows the virtual assistant response using grammar and information from each of the four travel domain providers. The "Trip Booker" domain has a relatively low cost (only $1 per query, however, given that it refers to its unique name "Trip Booker" and recommends hotels of exactly one brand). Has a very self-contained response.Probably, the Trip Booker domain provider and the hotel brand have a paid business relationship.The "Travel Mate" domain has a mid-range cost ($3 per query and available It provides a fairly useful response showing a large number of results and a reasonable list of the top five most likely interesting results: The "TravelHound" domain has a high cost (8￠ per query, but It provides arguably the most useful result, with a number of hotels, a list of several, and a guide that allows the user to get a much more satisfactory result by sorting or filtering the list using an intuitive voice interface." Chee-po -tels (chee for tels)" domain has a low cost (only 1￠ per query, but its grammar, recognizing that the word Paris most likely refers to a large city in France, Assume a quirky little town in the rural American state of Idaho, with no hotels.

Regardless of the domain's sorting order (eg, pricing model, number of query hits, star selection by VA developer, query response type, etc.), the platform may present results in two sections. A first section, indicated by a sub-heading 146 named "Editor's Picks - Sponsored", indicates one or more domains that, if selected by the VA developer, the platform provider will like. In general, "Editor's Picks" domains are domains sponsored by or picked up by platform providers because they encourage more use or satisfactory results by virtual assistants. The rest of the domains that can provide information for completing responses to test queries are indicated in the section indicated by subheading 147 entitled “Other”.

pricing model

The example used above represents an unusually simple pricing model of a linear fee of one or several US cents per request for information. In some embodiments, the cost per request for information that is inexpensive to acquire and transmit may be much lower than 1 US cents per request. 15( a ) shows an exemplary linear pricing model with a price of $ .0005 per request (equivalent to $ .05 per request).

In the industry, it is common to use a pricing model in which a domain is non-linear with respect to the number of queries with the highest interpretation scores. One example of a non-linear type of pricing model is a graded pricing model. Fig. 15(b) shows such a pricing model. The cost per request decreases when the number of requests crosses a threshold. Specifically, for the first 10,000 queries within a month, the cost is $.0010 per query. For request quantities of 10,001 to 50,000 pieces, the cost is $ .0005 per request. For request quantities 50,001 to 250,000, the cost is $.0002 per request. For request quantity 250,001 and any others, the cost is $.0001 per request. The query count is reset once a month.

In some embodiments, the domain provider sends a monthly invoice to the platform provider for served information requests. In some embodiments, the platform provider pre-purchases the credit of the information request. Charges for credits are set in tiered pricing segments according to the quantity purchased. In various embodiments, there may be various combinations of subscribing users, app integrators purchasing credits from platform providers or domain providers, any receiving party paying by invoice, or any other suitable method of paying for services. there is.

15( c ) shows a pricing model based on a non-linear equation. Instead of a graded interval of a large number of information requests, the pricing per request is based on a formula. Although many equations are possible, the equation of FIG. 16 is as follows.

The platform provider or domain provider or both maintain a count of information requests served through a sliding window over the last 30 days. The price per request is a minimum of $.0001 per request, plus the number of requests plus the logarithmically varying cost. This effectively provides a bulk purchase discount to frequent users.

A variety of other non-linear pricing models are possible, limited only by the creativity of the person negotiating the business contract.

CRM

16A illustrates an exemplary non-transitory computer readable medium 161 that is a rotating magnetic disk. Data centers typically use magnetic disks to store code and data, including instructions for server processors. The non-transitory computer-readable medium 161 stores code comprising instructions that, when executed by one or more computers, will cause the computer to perform the steps of the methods described herein. Rotating optical disks and other mechanically moving storage media are possible.

16( b ) illustrates an exemplary non-transitory computer-readable medium 162 that is a flash random access memory (RAM) chip. Data centers typically use flash memory to store code and data for server processors. Mobile devices typically use flash memory to store code and data for a processor within a system-on-chip device. The non-transitory computer readable medium 162 stores code comprising instructions that, when executed by one or more computers, will cause the computer to perform the steps of the methods described herein. Other non-moving storage media that are packaged using leads or solder balls are also possible.

Any type of computer-readable medium is suitable for storing code comprising instructions in accordance with various embodiments.

server

The server stores a database of domains provided to VA developers in the platform menu. The server also stores a database of codes for grammars associated with the domain. The server also stores a database of pricing models associated with the domain.

17A illustrates a rack-mounted server blade multi-processor server system 170 in accordance with some embodiments. It includes multiple network-connected computer processors executing software in parallel.

17( b ) shows a block diagram of the server system 170 . It includes a multi-core cluster of computer processor (CPU) cores 171 and a multi-core cluster of graphics processor (GPU) cores 172 . The processor is coupled to a random access memory (RAM) device 174 via a board-level interconnect 173 for program code and data storage. Server system 170 also includes a network interface 178 for allowing the processor to access the Internet. By executing instructions stored in RAM device 174 , CPU 171 and GPU 172 perform the steps of the method as described herein.

SoC

18( a ) shows a bottom view of a packaged system on chip device 180 with a ball grid array for surface mount soldering to a printed circuit board. For various chip implementations, various package shapes and sizes are possible. System-on-chip (SoC) devices control many embedded system and IoT device embodiments as described herein.

18( b ) shows a block diagram of the system on chip 180 . It includes a multi-core cluster of computer processor (CPU) cores 181 and a multi-core cluster of graphics processor (GPU) cores 182 . The processor connects, via a network on chip 183 , to an off-chip dynamic random access memory (DRAM) interface 184 for volatile program and data storage and to non-volatile computer program code in a flash RAM non-transitory computer readable medium. It is connected to a flash interface 185 for storage. SoC 180 also has a display interface 186 for displaying a GUI and an I/O interface module 187 for connecting to various I/O interface devices as needed for different peripheral devices. do. The I/O interface enables sensors such as touchscreen sensors, geolocation receivers, microphones, speakers, Bluetooth peripherals, and USB devices such as keyboards and mice, among others. The SoC 180 also allows the processor to operate over a wired or wireless connection, such as Wi-Fi, 3G, 4G long-term evolution (LTE), 5G, and other air interface standard radios as well as Ethernet connection hardware. and a network interface 188 for allowing access to the Internet. By executing instructions stored in a RAM device via interface 184 or in a flash device via interface 185 , CPU 181 and GPU 182 perform the steps of the method as described herein. do.

Additional considerations

Those skilled in the art will recognize many modifications and variations. Modifications and variations include any relevant combination of the disclosed features.

Various embodiments are methods that use the behavior of any one or a combination of humans and machines. Method embodiments are completed wherever most of the construction steps occur. Some embodiments are one or more non-transitory computer-readable media arranged to store such instructions for the methods described herein. A non-transitory computer readable medium containing any of the necessary code maintains the complete embodiment, whatever the machine possesses. Some embodiments include physical devices such as semiconductor chips; a hardware description language representation of the logical or functional behavior of such a device; and one or more non-transitory computer-readable media arranged to store such hardware description language representations.

The description herein reciting principles, features, and embodiments encompasses both structural and functional equivalents thereof. Elements described herein as being coupled have an effective relationship that can be realized by direct connection with or indirectly with one or more intervening elements.

The examples shown and described use some spoken language. Various embodiments, likewise, operate for other languages or combinations of languages. The examples shown and described use certain domains of knowledge. Various embodiments operate similarly for other domains or combinations of domains.

Some embodiments are screenless without a display screen, such as an earpiece. Some embodiments are stationary, such as vending machines. Some embodiments are mobile, such as a motor vehicle. Some embodiments are portable, such as a mobile phone. Some embodiments include a manual interface, such as a keyboard or touchscreen. Some embodiments include neural interfaces that use human thoughts as a form of natural language expression.

Claims (10)

A method for providing integration of a virtual assistant domain in an app, the method comprising:
receiving a test query from an app integrator;
interpreting the test query in a plurality of domains, some of which have a cost to the app integrator;
indicates to the app developer a subset of domains capable of interpreting the test query, and for each indicated domain, the cost of using the indicated domain to respond to the test query to the app integrator comprising the steps of indicating
How to provide integration of virtual assistant domains in your app. A platform for constructing a domain for inclusion in a natural language virtual assistant, comprising:
means for entering a plurality of test queries;
means for inputting a selection of domains for inclusion in the natural language virtual assistant;
a display view showing which domain is selected from the menu of domains for inclusion in the natural language virtual assistant;
of aggregation costs, a cost component to be charged for the natural language virtual assistant for answering the plurality of test queries using the domain selected for inclusion in the natural language virtual assistant, each cost component selected from a menu of the domain due to the domain being - including a display view showing
The selection of combinations of domains for inclusion in the natural language virtual assistant is configurable so that the aggregation cost for responding to the test query is within a desired budget.
platform. 3. The method of claim 2,
and a display view showing a histogram indicating how many of the test queries will be answered by each domain selected for inclusion in the natural language virtual assistant.
platform. 3. The method of claim 2,
The display view showing which domains are selected from the menu of domains for inclusion in the natural language virtual assistant displays at least one promotional domain in a different way than other domains.
platform. 3. The method of claim 2,
When the domain is selected for inclusion in the natural language virtual assistant, the cost component of at least one domain varies non-linearly with respect to the number of test queries to which the domain will respond.
platform. A platform for configuring a virtual assistant, comprising:
a server-hosted platform for selecting a domain for inclusion in the virtual assistant;
a grammar for making a request to a domain provider and a plurality of selectable domains;
means for exporting code that, when executed by a processor in an embedded system independent of the platform server, causes an interpreter function to respond to a query to a local domain.
platform. 7. The method of claim 6,
The platform is
means for receiving a set of test queries;
and a display view enabled to indicate a cost for responding to the query in the set of test queries for a given domain selection.
platform. 7. The method of claim 6,
wherein the platform further comprises a plurality of pricing models associated with the selectable domain.
platform. receiving, by the platform, a domain;
generating, by the platform, a first domain grammar identifying a grammatical phrasing within the received domain;
associating, by the platform, the domain grammar to the domain before the domain is selected for inclusion in the virtual assistant.
method. 10. The method of claim 9,
associating with the domain a pricing model that identifies a cost per query for using the domain.
method.

Images