KR20220027220A - Predictive Similarity Scoring Subsystem in Natural Language Understanding (NLU) Frameworks - Google Patents

Abstract

The present embodiments include an agent automation framework having a similarity scoring subsystem that performs semantic expression similarity scoring to facilitate extraction of artifacts for handling utterances. The similarity scoring subsystem identifies the CCG form of the utterance-based semantic representation and queries the database to retrieve a list of comparison functions that enable quantifications of similarities between candidates and the semantic representation in the search space. Comparison functions enable the similarity scoring subsystem to perform computationally the least expensive and/or most efficient comparisons before other comparisons. The similarity scoring subsystem may determine an initial similarity score between the particular semantic expression and the candidates in the search space, and then omnipotent dissimilar candidates from the search space. The selective search space omnipotence allows the similarity scoring subsystem to iteratively compare more data of a semantic expression to the search space through increasingly complex comparison functions, while narrowing the search space to potentially matching candidates.

Description

Predictive Similarity Scoring Subsystem in Natural Language Understanding (NLU) Frameworks

cross references

This application claims the priority and benefits of U.S. Provisional Application No. 62/869,817, filed on July 2, 2019 under the title "PREDICTIVE SIMILARITY SCORING SUBSYSTEM IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK", and this provisional application is for all purposes For all purposes, it is incorporated herein by reference in its entirety. This application also includes US Provisional Application Serial Nos. 62/869,864 entitled "SYSTEM AND METHOD FOR PERFORMING A MEANING SEARCH USING A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK;" US Provisional Application No. 62/869,826 entitled "DERIVING MULTIPLE MEANING REPRESENTATIONS FOR AN UTTERANCE IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK"; and U.S. Provisional Application No. 62/869,811 entitled "PINNING ARTIFACTS FOR EXPANSION OF SEARCH KEYS AND SEARCH SPACES IN A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK," each of which was filed on July 2, 2019 and is incorporated herein by reference in its entirety for all purposes.

BACKGROUND This disclosure relates generally to the fields of natural language understanding (NLU) and artificial intelligence (AI), and more particularly to a predictive similarity scoring subsystem for NLU.

This section is intended to introduce the reader to various aspects of the technology that may be related to various aspects of the present disclosure described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it is to be understood that these statements are to be read in this light and are not admissions of prior art.

Cloud computing generally relates to the sharing of computing resources accessed via the Internet. In particular, a cloud computing infrastructure may enable users, such as individuals and/or businesses, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing-based services. do. In doing so, users may access customized computing resources located at remote locations, which may be used to perform various computing functions (eg, storage and/or processing of large amounts of computing data). there is. For enterprise and other organizational users, cloud computing provides access to cloud computing resources without incurring significant upfront costs, such as purchasing expensive network equipment or investing large amounts of time to build a private network infrastructure. Provides flexibility. Instead, by leveraging cloud computing resources, users can divert those resources to focus on the company's core functions.

This cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to problems at the client instance based on natural language requests from the user of the client instance. For example, a user can provide a request to a virtual agent for assistance with a password problem, which is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. am. NLP is a general area of computer science and AI that includes some form of natural language input processing. Examples of areas covered by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and the like. NLU is a sub-domain of NLP that specifically focuses on understanding user utterance. Examples of areas covered by the NLU include question-and-answer (eg, reading questions), article summaries, and the like. For example, an NLU may use algorithms to reduce human language (eg, spoken or written) into a known set of symbols for consumption by a downstream virtual agent. NLP is commonly used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, a type of AI that examines and uses patterns in data to improve the understanding of programs.

However, existing virtual agents that apply NLU techniques to identify intent and entity matches within a search space infer meaning from a received user utterance and attempt to determine an appropriate response therefor. Computational resources can be over-expanded. Indeed, during semantic retrieval, certain existing approaches may directly query the entire set of stored user utterances within the search space to derive meaning for the received user utterance, thereby requiring significant processing and processing over an extended period of time. It can consume memory resources. As such, existing approaches may not be able to efficiently handle complex user utterances in a manner suitable for real-time engagement with the user and/or may not be able to simultaneously generate appropriate and timely responses to a plurality of user utterances.

A summary of specific embodiments disclosed herein is presented below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these specific embodiments, and that these aspects are not intended to limit the scope of the disclosure. Indeed, the present disclosure may include various aspects that may not be set forth below.

The present embodiments relate to an agent automation framework designed to extract meaning from user utterances, such as requests received by a virtual agent, and to respond appropriately to such user utterances. To perform these tasks, the agent automation framework includes an intent-entity model and an NLU framework with defined intents and entities associated with sample utterances. The NLU framework includes a semantic extraction subsystem designed to generate semantic representations for sample utterances of an intent-entity model. Thus, the NLU framework creates a searchable understanding model or search space from semantic representations, each representing a different understanding or interpretation of an underlying utterance. The semantic extraction subsystem also generates a semantic representation based on the utterance received from the user, wherein the semantic representation is a search key to be compared with the search space. As such, the semantic retrieval subsystem of the disclosed NLU framework is designed to search the semantic representations of the understanding model to find semantic representation matches to the semantic representation of the received user utterance. The semantic extraction subsystem may subsequently extract intents and entities from matching semantic expressions to facilitate appropriate agent responses and/or actions.

For NLU frameworks that rely heavily on search to extract understandings from received user utterances, such as current semantic search-based NLU techniques, the search process applied to large understanding models or even combinations of understanding models is In certain cases it is possible to evaluate a significant number of semantic expressions. As such, pruning the search space to remove semantic expressions that are incompatible with the semantic expression of a user utterance on the surface of the search space allows the NLU framework to incorporate larger search spaces or allow present embodiments of search spaces. It is now recognized that it can lead to more skillful acceptance. In embodiments utilizing a search space generated from a plurality of models of understanding, the techniques disclosed herein may be used for a plurality of different business aspects (eg, sales, building information, service tickets) can deliver improved user satisfaction based on semantic searches. As discussed below, the present embodiments generally improve the operation of the semantic search subsystem by utilizing similarity scoring capabilities to prune large search spaces to improve identification of matching semantic expressions within an understanding model. In particular, the present embodiments use low-cost or less resource-intensive pruning criteria during earlier parts of the semantic match process to continuously shrink the search space, and then use the more resource-intensive pruning criteria in the reduced size search space. apply them As will be appreciated, the present techniques solve the difficult problem posed by the NLU by doing so, a manageable, less resource-intensive search problem that does not waste resources in evaluating semantic expressions in a search space that are not superficially related to a search key. This is solved by converting to .

More specifically, the present embodiments relate to a similarity scoring subsystem of a semantic search subsystem for efficiently comparing semantic expressions derived from received user utterances with semantic expressions derived from sample utterances of an intent-entity model. . As will be appreciated, based on an incremental set of mathematical comparison functions, the similarity scoring subsystem iteratively determines a similarity score for each pair of compared semantic expressions. For example, to compare a particular semantic expression to a set of semantic expressions in an understanding model, the similarity scoring subsystem may first determine a cognitive construction grammar (CCG) form for the particular semantic expression. As suggested by CCG techniques, the CCG form class membership of a specific semantic expression is established by the shape of the utterance tree structure of the specific semantic expression, as well as the part-of-speech annotation of the nodes of the semantic expression. Based on the CCG form of the particular semantic expression, the similarity scoring subsystem may then query the form class database to retrieve a list of mathematical comparison functions that enable comparisons between the particular semantic expression and the semantic expressions of the understanding model. . In addition, the similarity scoring subsystem may ignore or omnipotent semantic representations of an understanding model that do not have a compatible CCG form for that of a particular semantic representation.

As described herein, each list of mathematical comparison functions includes an ordered set of comparison functions that iteratively takes into account each number of nodes of the semantic expressions being compared. In particular, the comparison functions are ordered such that the similarity scoring subsystem first implements the computationally cheapest and/or most efficient function, and thus determines an initial or preliminary similarity score between the particular semantic representation and the semantic representations of the understanding model. For example, the similarity scoring subsystem may use the initial function to consider whether the root node of a particular semantic representation is appropriately similar to the root node of each comparable semantic representation of the understanding model. The similarity scoring subsystem then applies a subsequent function to account for both the root node and the first dependent node of each semantic expression, or applies any other more resource-intensive comparison function to appropriately similar semantic expressions. can be narrowed down to Thus, the iterative application of the selective node uncovering described herein and/or the application of more expensive comparison functions can create increasingly complex comparison functions while hone potentially matching semantic representation candidates of the understanding model. Iteratively considers more features of the compared semantic expressions through As such, the present techniques for predictive similarity scoring enable targeted discovery of semantic representation matches, whereby the semantic representation size (eg, number of nodes) and search space size can be large, NLU provides computational benefits to generative spaces such as

Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the drawings.
1 is a block diagram of an embodiment of a cloud computing system in which embodiments of the present technology may operate.
2 is a block diagram of an embodiment of a multi-instance cloud architecture in which embodiments of the present technology may operate.
3 is a block diagram of a computing device utilized in the computing system that may be in FIG. 1 or 2 , in accordance with aspects of the present technology.
4A is a schematic diagram illustrating an embodiment of an agent automation framework including an NLU framework that is part of a client instance hosted by a cloud computing system, in accordance with aspects of the present technology.
4B is a schematic diagram illustrating an alternative embodiment of an agent automation framework in which portions of the NLU framework are part of an enterprise instance hosted by a cloud computing system, in accordance with aspects of the present technology.
5 illustrates an embodiment of a process by which an agent automation framework including an NLU framework and a behavior engine framework extracts intents and/or entities from a user utterance and responds to a user utterance, in accordance with aspects of the present technology; is a flow chart that
6 is a block diagram illustrating an embodiment of an NLU framework including a semantic extraction subsystem and a semantic retrieval subsystem, wherein the semantic extraction subsystem generates semantic expressions from received user utterances, in accordance with aspects of the present technology. to calculate the utterance semantic model, generate semantic expressions from the sample utterances of the comprehension model to produce an comprehension model, and the semantic search subsystem compares the semantic expressions of the utterance semantic model with the semantic expressions of the comprehension model to compare the received user Extract artifacts (eg intents and entities) from the utterance.
7 is a diagram illustrating an example of an utterance tree generated for an utterance, according to an embodiment of the present approach.
8 is a semantic search analyzing a search space defined by an understanding model to determine or identify matching semantic expressions that enable an NLU framework to extract artifacts from a received user utterance, in accordance with aspects of the present technology; An information flow diagram illustrating an embodiment of a subsystem.
9 is a diagram of similarity that may be implemented within the semantic retrieval subsystem of the NLU framework to retrieve lists of mathematical comparison functions that enable efficient comparisons between any suitable number of semantic expressions, in accordance with aspects of the present technology; An information flow diagram illustrating an embodiment of a scoring subsystem.
10 is a flow diagram illustrating an embodiment of a process by which the similarity scoring subsystem retrieves lists of mathematical comparison functions that enable comparison between the utterance-based semantic representation of FIG. 8 and the search space, in accordance with aspects of the present technology.
11 is a diagram of an embodiment of a similarity scoring subsystem of a semantic search subsystem that uses a list of mathematical comparison functions to compare a first semantic representation to a second semantic representation, in accordance with aspects of the present technology.
12 is a schematic diagram illustrating an embodiment of a similarity scoring subsystem that applies a list of mathematical comparison functions to selectively narrow a search space to identify semantic expressions that match an utterance-based semantic expression, in accordance with aspects of the present technology.
13 is a flow diagram of an embodiment of a process by which a similarity scoring subsystem implements a list of mathematical comparison functions to identify matching semantic expressions from a search space, in accordance with aspects of the present technology.

One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. When developing any such actual implementation, such as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which may vary from implementation to implementation. It should be understood that It should also be understood that such a development effort can be complex and time consuming, but will nevertheless become the routine tasks of design, fabrication, and manufacture for those skilled in the art having the benefit of this disclosure.

As used herein, the term “computing system” or “computing device” refers to an electronic computing device such as, but not limited to, a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device. , or a plurality of electronic computing devices that work together to perform a function described as being performed on or by the computing system. As used herein, the term “machine-readable medium” refers to a single medium or a plurality of media (eg, a centralized or distributed database, and/or an associated cache) storing one or more instructions or data structures. and servers). The term “non-transitory machine-readable medium” may also store, encode, or carry instructions for execution by a computing system and cause the computing system to perform any one or more of the methodologies of the present subject matter, or such instructions It should be considered to include any tangible medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. The term “non-transitory machine-readable medium” should accordingly be considered to include, but is not limited to, solid state memories, and optical and magnetic media. Specific examples of non-transitory machine-readable media include, for example, semiconductor memory devices (eg, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), and flash memory devices). , magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and non-volatile memory including CD-ROM and DVD-ROM disks.

As used herein, the terms “application,” “engine,” and “plug-in” refer to computer software instructions (eg, a computer program) executable by one or more processors of a computing system to provide a particular function. and/or scripts). Computer software instructions may be written in any suitable programming languages, such as C, C++, C#, Pascal, Fortran, Perl, MATLAB, SAS, SPSS, JavaScript, AJAX, and JAVA. These computer software instructions may include standalone application with data input and data display modules. Alternatively, the computer software instructions disclosed may be classes instantiated as distributed objects. The computer software instructions disclosed may also be component software, eg, JAVABEANS or ENTERPRISE JAVABEANS. Additionally, the disclosed applications or engines may be implemented in computer software, computer hardware, or a combination thereof.

As used herein, the term “framework” refers to a system of applications and/or engines that cooperate to perform one or more overall functions, as well as any other supporting data structures, libraries, modules, and any other supporting function. In particular, a "Natural Language Understanding Framework" or "NLU Framework" refers to a set of computer programs designed to process and derive meaning (e.g., intents, entities, artifacts) from natural language utterances based on an understanding model. include As used herein, “behavioral engine” or “BE”, also known as an inference agent or RA/BE, refers to a rule-based agent, such as a virtual agent, designed to interact with users based on a conversational model. For example, a “virtual agent” may refer to a particular example of a BE designed to interact with users via natural language requests in a particular conversation or communication channel. With this in mind, the terms "virtual agent" and "BE" are used interchangeably herein. As a specific example, the virtual agent may be or include a chat agent that interacts with users via natural language requests and responses in a chat room environment. Other examples of virtual agents may include an email agent, forum agent, ticketing agent, phone call agent, etc. that interacts with users in connection with automated responses to email, forum posts, service tickets, phone calls, etc. there is.

As used herein, “intent” refers to a desire or goal of a user that may relate to a primary purpose of a communication, such as an utterance. As used herein, "entity" refers to some other parameterization of an object, subject, or intent. Note that in the present embodiments, specific entities are treated as parameters of the corresponding intent. More specifically, certain entities (eg, time and location) may be globally recognized and extracted for all intents, while other entities are intent-specific (eg, a product entity associated with purchase intents). ) and are generally extracted only when found within the intents that define them. As used herein, “artifact” refers collectively to both intents and entities of an utterance. As used herein, an “understanding model” is a set of models used by the NLU framework to infer the meaning of natural language utterances. The understanding model may include a lexical model that associates specific tokens (eg, words or phrases) with specific word vectors, an intent-entity model, an entity model, or a combination thereof. As used herein, "intent-entity model" refers to a model that associates particular intents with particular sample utterances, and the entities associated with the intent may be encoded as a parameter of the intent within the sample utterances of the model. . As used herein, the term “agents” may refer to computer-generated personas (eg, chat agents or other virtual agents) interacting with each other within a conversation channel. As used herein, “corpus” refers to a captured body of source data that includes interactions between various users and virtual agents, the interactions comprising one or more suitable tangible media (eg, communications or conversations within a help line, chat room or message string, e-mail string). As used herein, “utterance tree” refers to a data structure that stores a semantic representation of an utterance. As discussed, the utterance tree has a tree structure (eg, a dependency parse tree structure) representing the syntactic structure of the utterance, wherein the nodes of the tree structure contain vectors encoding the semantic meaning of the utterance (eg, word vectors). , subtree vectors).

As used herein, “source data” or “conversation logs” support chat logs, email strings, documents, help documentation, frequently asked questions (FAQs), forum entries, ticketing any suitable captured interactions between the various agents including, but not limited to, items made, records of telephone counseling service calls, and the like. As used herein, “utterance” refers to a single natural language statement made by a user or agent that may contain one or more intents. As such, the utterance may be part of a previously captured corpus of source data, and the utterance may also be a new statement received from the user as part of an interaction with the virtual agent. As used herein, “machine learning” or “ML” refers to any suitable statistical form of artificial intelligence that can be trained using machine learning techniques, including supervised, unsupervised, and unsupervised learning techniques. can be used to For example, in certain embodiments, ML techniques may be implemented using a neural network (NN) (eg, a deep neural network (DNN), a recurrent neural network (RNN), a recursive neural network). As used herein, “vector” (eg, word vector, intent vector, subject vector, subtree vector) means the meaning of part of an utterance (eg, word or phrase, intent, entity, token). Refers to a linear algebra vector that is an ordered n -dimensional list (eg, a 300-dimensional list) of floating-point values (eg, a 1×N or N×1 matrix) that provides a mathematical representation of the logical meaning. As used herein, “domain specificity” refers to how the system is tuned for accurately extracting intents and entities expressed in real conversations in a given domain and/or conversation channel. As used herein, “understanding” of an utterance refers to the interpretation or construction of an utterance by the NLU framework. As such, it can be seen that different understandings of an utterance may be associated with different semantic expressions with different structures (eg, different nodes, different relationships between nodes), different part-of-speech tagging, and the like.

As noted, the computing platform may include a chat agent, or other similar virtual agent, designed to automatically respond to user requests to perform functions or solve problems on the platform via NLU technologies. The disclosed NLU framework is based on the principles of cognitive building grammar (CCG), in which aspects of the meaning or understanding of a natural language utterance can be determined based on the form (eg, syntactic structure, shape) and semantic meaning of the utterance. based on The disclosed NLU framework may generate a plurality of semantic representations for an utterance, each semantic representation being an utterance tree representing a particular understanding of the utterance. As such, the disclosed NLU framework is capable of generating an understanding model having multiple semantic representations for particular sample utterances, which expands the search space of semantic search, thereby improving the operation of the NLU framework. However, when attempting to derive user intent from natural language utterances, certain NLU frameworks may have a search space of excessively large size due to their inclusion of semantic expressions that are incomparable and/or dissimilar to that of user-based utterances. It is now recognized that searches can be performed. As such, for the entire search space, certain NLU frameworks may employ non-customized or single-cost comparison functions that limit the performance of semantic search to search spaces that are above a certain scale threshold based on available processing and memory resources. there is.

Accordingly, the present embodiments generally relate to an agent automation framework having a semantic retrieval subsystem designed to utilize CCG techniques to enhance semantic retrieval. As discussed herein, the semantic search subsystem can iteratively and incrementally narrow the search space searched for matching semantic expressions. Indeed, these matching semantic expressions define a subset of the search space that enables identification of intent or entity matches for received user utterances. More specifically, the present embodiments relate to a similarity scoring subsystem of a semantic search subsystem that efficiently compares semantic expressions determined based on a received user utterance with semantic expressions of sample utterances defining a search space. As will be appreciated, based on a progressive set of mathematical comparison functions, the similarity scoring subsystem iteratively determines progressively accurate similarity scores between a pair of compared semantic expressions and searches based on the various iterations of the similarity score. Occupy the space

For example, to compare a particular semantic expression to a set of semantic expressions defining a search space, the similarity scoring subsystem may first determine a CCG form for the particular semantic expression. As previously mentioned, the CCG form class membership of each particular semantic expression is established by the shape and semantic meanings of the nodes that form a tree structure (eg, utterance tree) of the particular semantic expression. Based on the CCG form of a particular semantic expression, the similarity scoring subsystem is designed to query the form class database to retrieve a list of mathematical comparison functions, enabling quantitative comparisons between each pair of semantic expressions. Indeed, the list of mathematical comparison functions contains nested functions that individually specify how increasingly precise and/or precise determinations of the similarity between each part of a particular semantic expression and the semantic expressions of the search space can be performed. do. For a compared pair of semantic expressions that do not have a mathematical comparison function list, the similarity scoring subsystem prunes the associated semantic expressions from the search space (e.g., immediately determines the lowest possible score without performing any comparisons), It may conserve resource utilization for remaining potentially matching semantic expressions.

More specifically, each list of mathematical comparison functions can contain comparison functions (eg, vector algebra, cosine similarities, progressive functions, other databases or calls to structures). As will be appreciated herein, the comparison functions are ordered to allow the similarity scoring subsystem to perform the computationally cheapest and/or most efficient comparison first. The similarity scoring subsystem may determine an initial similarity score between the particular semantic representation and comparable semantic representations remaining within the search space. For example, the similarity scoring subsystem may use the computationally cheapest function to consider whether a particular semantic representation is appropriately similar to each semantic representation in the search space. Such a determination may generally provide the least accurate and most efficient prediction of which areas of the search space are worthy of further investigation. The similarity scoring subsystem may then prune dissimilar semantic expressions from the search space before applying a subsequent function to further consider each remaining comparable pair of compared semantic expressions. Thus, progressive data utilization of comparison functions allows the similarity scoring subsystem to iteratively consider additional features of compared semantic expressions through increasingly complex comparison functions, while narrowing the search space to potentially matching candidates. As such, the present techniques for predictive similarity scoring enable targeted discovery of semantic expression matches, thereby reducing computational overhead for agent automation systems implementing these techniques and reducing computational benefits to improve efficiency. to provide. Further, as the search capacity of an agent automation system is increased, the search space may be constructed from multiple understanding models that enable natural language agent responses to address multiple different aspects of the business supporting the agent automation system.

With the above in mind, the following figures relate to various types of generalized system architectures or configurations that may be used to provide services to an organization within a multi-instance framework and in which the present approaches may be employed. Correspondingly, such system and platform examples may also relate to systems and platforms in which the techniques discussed herein may be implemented or otherwise utilized. Referring now to FIG. 1 , illustrated is a schematic diagram of an embodiment of a cloud computing system 10 in which embodiments of the present disclosure may operate. The cloud computing system 10 may include a client network 12 , a network 18 (eg, the Internet), and a cloud-based platform 20 . In some implementations, the cloud-based platform 20 may be a configuration management database (CMDB) platform. In one embodiment, the client network 12 may be a local private network, such as a local area network (LAN), having various network devices including, but not limited to, switches, servers, and routers. In other embodiments, client network 12 represents an enterprise network, which may include one or more LANs, virtual networks, data centers 22, and/or other remote networks. 1 , a client network 12 may be connected to one or more client devices 14A, 14B, and 14C such that the client devices may communicate with each other and/or with the network hosting the platform 20 . there is. Client devices 14 access cloud computing services, for example, via a web browser application or via edge device 16 , which may act as a gateway between client devices 14 and platform 20 . computing systems and/or other types of computing devices commonly referred to as Internet of Things (IoT) devices. 1 also illustrates a management, instrumentation, and Discovery) exemplifies including an administration or managerial device, agent, or server, such as the server 17 . Although not specifically illustrated in FIG. 1 , the client network 12 may also include an access network device (eg, a gateway or router) or a combination of devices implementing a customer firewall or intrusion prevention system.

In the illustrated embodiment, FIG. 1 illustrates that a client network 12 is coupled to a network 18 . Network 18 may include other LANs, wide area networks (WANs), the Internet, and/or other remote networks for transferring data between client devices 14A-C and the network hosting platform 20 . It may include one or more computing networks such as Each of the computing networks in network 18 may include wired and/or wireless programmable devices operating in the electrical and/or optical domain. For example, network 18 includes wireless networks such as cellular networks (eg, Global System for Mobile Communications (GSM) based cellular network), IEEE 802.11 networks, and/or other suitable radio based networks. can do. Network 18 may also utilize any number of network communication protocols, such as Transmission Control Protocol (TCP) and Internet Protocol (IP). Although not explicitly shown in FIG. 1 , network 18 may include various network devices such as servers, routers, network switches, and/or other network hardware devices configured to transmit data over network 18 . may include

1 , the network hosting platform 20 may be a remote network (eg, a cloud network) capable of communicating with client devices 14 via client network 12 and network 18 . The network hosting the platform 20 provides additional computing resources to the client devices 14 and/or the client network 12 . For example, by utilizing a network hosting platform 20 , users of client devices 14 may build and run applications for various enterprise, IT, and/or other organizational related functions. In one embodiment, the network hosting the platform 20 is implemented on one or more data centers 22 , where each data center may correspond to a different geographic location. each of the data centers 22 includes a plurality of virtual servers 24 (also referred to herein as application nodes, application servers, virtual server instances, application instances, or application server instances); wherein each virtual server 24 is on a physical computing system, such as a single electronic computing device (eg, a single physical hardware server), or across multiple computing devices (eg, a plurality of physical hardware servers). can be implemented. Examples of virtual servers 24 are a web server (eg, a single Apache installation), an application server (eg, a single JAVA virtual machine), and/or a database server (eg, a single relational database). management system (RDBMS) catalog).

To utilize the computing resources within the platform 20 , network operators may choose to configure the data centers 22 using a variety of computing infrastructures. In one embodiment, one or more of the data centers 22 are configured using a multi-tenant cloud architecture such that one of the server instances 24 handles requests from and serves the plurality of customers. . Data centers 22 with a multi-tenant cloud architecture mix and store data from multiple customers, and multiple customer instances are assigned to one of the virtual servers 24 . In a multi-tenant cloud architecture, a specific virtual server 24 separates and separates data and other information of various customers. For example, a multi-tenant cloud architecture may assign a specific identifier to each customer to identify and isolate data from each customer. In general, implementing a multi-tenant cloud architecture may suffer from various disadvantages, such as failure of a particular one of the server instances 24 causing failure for all customers assigned to that particular server instance.

In another embodiment, one or more of the data centers 22 are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture may provide each customer instance with its own dedicated application server and dedicated database server. In other examples, a multi-instance cloud architecture may provide, for each customer instance, a single physical or virtual server 24 and/or physical and/or physical and/or virtual servers such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers. Or other combinations of virtual servers 24 may be deployed. In a multi-instance cloud architecture, a plurality of customer instances may be installed on one or more respective hardware servers, where each customer instance is allocated specific portions of physical server resources such as computing memory, storage, and processing power. . In doing so, each customer instance will have its own unique software stack that provides the benefits of data segregation, relatively less downtime for customers to access platform 20 , and customer-centric upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to FIG. 2 .

2 is a schematic diagram of an embodiment of a multi-instance cloud architecture 40 in which embodiments of the present disclosure may operate. 2 illustrates a client network 12 and network 18 in which a multi-instance cloud architecture 40 connects to two (eg, paired) data centers 22A and 22B that may be geographically separated from each other. shown to include Using FIG. 2 as an example, a network environment and service provider cloud infrastructure client instance 42 (also referred to herein as client instance 42 ) is configured with dedicated virtual servers (eg, virtual servers 24A, 24B, 24C, and 24D) and dedicated database servers (eg, virtual database servers 44A and 44B) (eg, supported and enabled by them). In other words, virtual servers 24A-24D and virtual database servers 44A and 44B are not shared with other client instances and are specific to each client instance 42 . In the illustrated example, to facilitate availability of client instance 42, virtual servers 24A-24D and virtual database servers 44A and 44B are assigned to two different data centers 22A and 22B. , one of the data centers 22 serves as a backup data center. Other embodiments of the multi-instance cloud architecture 40 may include other types of dedicated virtual servers, such as web servers. For example, client instance 42 may be associated with dedicated virtual servers 24A-24D, dedicated virtual database servers 44A and 44B, and additional dedicated virtual web servers (not shown in FIG. 2 ). are (eg, supported and enabled by them).

1 and 2 illustrate specific embodiments of cloud computing system 10 and multi-instance cloud architecture 40, respectively, the present disclosure is not limited to the specific embodiments shown in FIGS. 1 and 2 , respectively. . For example, while FIG. 1 shows platform 20 being implemented using data centers, other embodiments of platform 20 are not limited to data centers and may utilize other types of remote network infrastructures. there is. In addition, other embodiments of the present disclosure may combine one or more different virtual servers into a single virtual server, or conversely, use a plurality of virtual servers to perform operations attributed to a single virtual server. For example, using FIG. 2 as an example, virtual servers 24A, 24B, 24C, 24D and virtual database servers 44A, 44B may be combined into a single virtual server. Further, the present approaches may be adapted to other architectures or configurations, including but not limited to multi-tenant architectures, generalized client/server implementations, and/or even to perform some or all of the operations discussed herein. It can also be implemented on a single configured physical processor-based device. Similarly, although virtual servers or machines may be referenced to facilitate discussion of implementation, physical servers may be used instead as appropriate. The use and discussion of FIGS. 1 and 2 are examples for facilitating convenience of description and description, and are not intended to limit the present disclosure to the specific examples shown herein.

As will be appreciated, each of the architectures and frameworks discussed in connection with FIGS. 1 and 2 are used throughout various types of computing systems (eg, servers, workstations, client devices, laptops). , tablet computers, cellular phones, etc.). For completeness, a brief, high-level overview of the components typically found in such systems is provided. As will be appreciated, this summary is only intended to provide a high-level, generalized view of components typical of such computing systems, and should not be considered limiting in terms of components discussed or omitted from discussion.

As background, it can be appreciated that the present approach may be implemented using one or more processor-based systems such as those shown in FIG. 3 . Likewise, applications and/or databases used in the present approach may be stored, used, and/or maintained on such processor-based systems. As will be appreciated, such systems as shown in FIG. 3 may reside in a distributed computing environment, a networked environment, or other multi-computer platform or architecture. Likewise, systems such as that shown in FIG. 3 may be used to support or communicate with one or more virtual environments or compute instances in which the present approach may be implemented.

With this in mind, an exemplary computer system may include some or all of the computer components illustrated in FIG. 3 . 3 generally depicts a block diagram of exemplary components of computing system 80 and its potential interconnections or communication paths, such as along one or more buses. As shown, computing system 80 includes one or more processors 82 , one or more buses 84 , memory 86 , input devices 88 , power supply 90 , network interface 92 , and a user interface. (94), and/or other computer components useful for performing the functions described herein, such as, but not limited to, various hardware components.

One or more processors 82 may include one or more microprocessors capable of executing instructions stored in memory 86 . Additionally or alternatively, the one or more processors 82 may perform the functions discussed herein without invoking instructions from application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and/or memory 86 . It may include other devices designed to perform some or all of these.

With respect to the other components, the one or more buses 84 include electrical channels suitable for providing data and/or power between the various components of the computing system 80 . Memory 86 may include any tangible, non-transitory, and computer-readable storage media. Although shown as a single block in FIG. 1 , memory 86 may be implemented using a plurality of physical units of the same or different types in one or more physical locations. Input devices 88 correspond to structures for inputting data and/or instructions to one or more processors 82 . For example, input devices 88 may include a mouse, touchpad, touchscreen, keyboard, and the like. Power source 90 may be any suitable source for power of the various components of computing device 80 , such as line power and/or a battery source. Network interface 92 may include one or more transceivers capable of communicating with other devices over one or more networks (eg, communication channels). The network interface 92 may provide a wired network interface or a wireless network interface. User interface 94 may include a display configured to display text or images transmitted from one or more processors 82 to it. In addition to and/or alternatively to a display, user interface 94 may include other devices for interfacing with a user, such as lights (eg, LEDs), speakers, and the like.

It should be understood that the cloud-based platform 20 discussed above provides an example of an architecture that may utilize NLU techniques. In particular, the cloud-based platform 20 may contain or store a large corpus of source data that may be mined to facilitate the generation of multiple outputs, including an intent-entity model. For example, the cloud-based platform 20 may support requests for changes or repairs of certain systems, a conversation between the requestor and a service technician or administrator attempting to solve the problem, a description of how the ticket was eventually resolved, etc. It may include ticketing source data with The generated intent-entity model can then serve as a basis for classifying intents in future requests, and automatically address future issues within the cloud-based platform 20 based on natural language requests from users. It can be used to create and improve dialog models to support virtual agents that can be solved with As such, in certain embodiments described herein, the disclosed agent automation framework is integrated into the cloud-based platform 20 , while in other embodiments, the agent automation framework, as discussed below, comprises: may be hosted and executed (separately from the cloud-based platform 20 ) by a suitable system communicatively coupled to the cloud-based platform 20 for processing utterances.

With the above in mind, FIG. 4A illustrates an agent automation framework 100 (also referred to herein as agent automation system 100 ) associated with a client instance 42 . More specifically, FIG. 4A illustrates an example of a portion of a service provider cloud infrastructure, including the cloud-based platform 20 discussed above. The cloud-based platform 20 provides a user interface to the network applications running within the client instance 42 (eg, via a web browser of the client device 14D) via the network 18 to the client device ( 14D). A client instance 42 is supported by virtual servers similar to those described with respect to FIG. 2 , and is illustrated herein to demonstrate support for the disclosed functionality described herein within the client instance 42 . there is. The cloud provider infrastructure is generally configured to concurrently support a plurality of end user devices, such as client device 14D, where each end user device is communicating with a single client instance 42 . Further, the cloud provider infrastructure may be configured to concurrently support any number of client instances, such as client instance 42 , each of which is in communication with one or more end user devices. As noted above, the end user may also interface with the client instance 42 using an application running within a web browser.

The embodiment of the agent automation framework 100 shown in FIG. 4A includes a behavior engine (BE) 102 , an NLU framework 104 , and a database 106 , which communicate within a client instance 42 . possibly combined. BE 102 responds to natural language user requests 122 (also referred to herein as user utterances 122 or utterances 122) and agent responses 124 (also referred to herein as agent utterances 124). may host or include any suitable number of virtual agents or personas interacting with the user of the client device 14D via It may be noted that in actual implementations, the agent automation framework 100 may include a number of other suitable components, including a semantic extraction subsystem, a semantic retrieval subsystem, and the like, in accordance with the present disclosure.

In the embodiment shown in FIG. 4A , database 106 may be a database server instance (eg, database server instance 44A or 44B as discussed with respect to FIG. 2 ), or a collection of database server instances. there is. The illustrated database 106 stores an intent-entity model 108 , a dialog model 110 , a corpus of utterances 112 , and a set of rules 114 in one or more tables of the database 106 (eg, , a relational database table). The intent-entity model 108 stores associations or relationships between specific intents and specific entities through specific sample utterances. In certain embodiments, the intent-entity model 108 may be authored by a designer using an appropriate authoring tool. In other embodiments, the agent automation framework 100 generates an intent-entity model 108 from a corpus of utterances 112 stored in one or more tables of a database 106 and a set of rules 114 . . The intent-entity model 108 may also be determined based on a combination of authoring and ML techniques, in some embodiments. In any event, it should be understood that the disclosed intent-entity model 108 may associate any suitable combination of intents and/or entities with respective ones of the corpus of utterances 112 . In the embodiments discussed below, sample utterances of the intent-entity model 108 are used to generate semantic representations of the understanding model to define a search space for a semantic search.

In the embodiment shown in FIG. 4A , the dialog model 110 creates associations between the intents of the intent-entity model 108 and specific responses and/or actions that generally define the behavior of the BE 102 . Save. In certain embodiments, at least some of the associations in the dialog model are manually set by the designer of the BE 102 based on how the designer wants the BE 102 to respond to particular identified artifacts in the processed utterances. generated or predefined. In different embodiments, database 106 relates to compilation model template data (eg, class compatibility rules, class level scoring coefficients, tree model comparison algorithms, tree substructure vectorization algorithms), semantic expressions, etc. It should be noted that other database tables may include other database tables storing other information related to intent classification, such as tables storing information.

In the illustrated embodiment, the NLU framework 104 includes an NLU engine 116 and a vocabulary manager 118 . It will be appreciated that the NLU framework 104 may include any suitable number of other components. In certain embodiments, NLU engine 116 generates word vectors (eg, intent vectors, subject or entity vectors, subtree vectors) from words or phrases of utterances, as well as generating such vectors It is designed to perform a number of functions of the NLU framework 104 , including determining distances between them (eg, Euclidean distances). For example, the NLU engine 116 may generally generate a respective intent vector for each intent of the analyzed utterance. As such, a similarity measure or distance between two different utterances may be computed using respective intent vectors generated by the NLU engine 116 for the two intents, and the similarity measure in the meaning between the two intents. Provides an indication of similarity.

The vocabulary manager 118 handles extra-vocabulary words and symbols that the NLU framework 104 did not encounter during vocabulary training. For example, in certain embodiments, the vocabulary manager 118 (eg, based on the set of rules 114 ) may include words and abbreviations in utterances analyzed by the agent automation framework 100 (eg, based on the set of rules 114 ). can identify and replace synonyms and domain-specific meanings of , which can improve the performance of the NLU framework 104 to properly identify intents and entities within context-specific utterances. Additionally, to accommodate the tendency of natural language to adopt new uses for existing words, in certain embodiments, the vocabulary manager 118 may configure a word previously associated with other intents or entities based on a change in context. handle their re-creation. For example, vocabulary manager 118 may handle a situation in which the word “bike” actually refers to a motorcycle rather than a bicycle in the context of utterances from a particular client instance and/or conversation channel.

Once the intent-entity model 108 and dialog model 110 have been created, the agent automation framework 100 receives the user utterance 122 (in the form of a natural language request) and takes appropriate action to handle the request. designed to do For example, in the embodiment illustrated in FIG. 4A , BE 102 via network 18 , utterance 122 (eg, submitted by client device 14D disposed on client network 12 ) For example, a virtual agent that receives natural language requests in chat communication). BE 102 provides utterance 122 to NLU framework 104 , which NLU engine 116 is based on, along with various subsystems of NLU framework discussed below, based on intent-entity model 108 . to process the utterance 122 to derive artifacts (eg, intents and/or entities) within the utterance. Based on the artifacts derived by the NLU engine 116 as well as the associations within the dialog model 110 , the BE 102 performs one or more specific predefined actions. In the illustrated embodiment, the BE 102 also provides a response 124 (eg, a virtual agent utterance 124 or confirmation) to the client device 14D over the network 18 , which may include, for example, For example, it indicates actions performed by the BE 102 in response to the received user utterance 122 . Further, in certain embodiments, the utterance 122 may be added to the utterances 112 stored in the database 106 for continued learning within the NLU framework 104 .

It can be appreciated that in other embodiments, one or more components of the agent automation framework 100 and/or NLU framework 104 may be arranged, located, or hosted in other ways for improved performance. . For example, in certain embodiments, one or more portions of the NLU framework 104 are hosted by an instance that is separate from and communicatively coupled to the client instance 42 (eg, a shared instance, an enterprise instance). can be It is now recognized that such embodiments may advantageously reduce the size of the client instance 42 , thereby improving the efficiency of the cloud-based platform 20 . In particular, in certain embodiments, one or more components of the similarity scoring subsystem discussed below may provide an improved semantic search for appropriate matching semantic expressions within a search space to enable identification of artifact matches for an utterance 122 . may be hosted by the client instance 42 as well as a separate instance (eg, an enterprise instance) that is communicatively coupled to other client instances.

With the above in mind, FIG. 4B illustrates an agent in which portions of the NLU framework 104 are instead executed by a separate shared instance (eg, enterprise instance 125 ) hosted by the cloud-based platform 20 . An alternative embodiment of an automation framework 100 is illustrated. The illustrated enterprise instance 125 is configured to exchange data related to artifact mining and classification with any suitable number of client instances via an appropriate protocol (eg, via appropriate Representational State Transfer (REST) requests/responses). communicatively coupled. As such, for the design illustrated in FIG. 4B , by hosting a portion of the NLU framework as a shared resource accessible to a plurality of client instances 42 , the size of the client instance 42 is reduced (eg, FIG. 4A ). compared to the embodiment of the agent automation framework 100 illustrated in ) can be substantially reduced, and the overall efficiency of the agent automation framework 100 can be improved.

In particular, the NLU framework 104 shown in FIG. 4B is divided into three distinct components that perform separate processes within the NLU framework 104 . These components are the shared NLU trainer 126 hosted by the enterprise instance 125 , the shared NLU annotator 127 hosted by the enterprise instance 125 , and the NLU predictor hosted by the client instance 42 . (128). It should be noted that the organizations shown in FIGS. 4A and 4B are merely examples, and that in other embodiments, other organizations of the NLU framework 104 and/or the agent automation framework 100 may be used in accordance with the present disclosure. Able to know.

In the embodiment of the agent automation framework 100 illustrated in FIG. 4B , a shared NLU trainer 126 receives a corpus of utterances 112 from a client instance 42 , It is designed to perform semantic mining (including, for example, semantic parsing, grammar engineering, etc.) to facilitate creation. Once the intent-entity model 108 has been generated, when the BE 102 receives the user utterance 122 provided by the client device 14D, the NLU predictor 128 parses and annotates the utterance 122 . Pass utterance 122 and intent-entity model 108 to shared NLU annotator 127 for bookkeeping. The shared NLU annotator 127 performs semantic parsing, grammar manipulation, etc. of the utterance 122 based on the intent-entity model 108 and converts the annotated utterance trees of the utterance 122 to the client instance 42 . returns to the NLU predictor 128 of The NLU predictor 128 then uses these annotated structures of the utterance 122, discussed in more detail below, to identify matching intents from the intent-entity model 108, and thus the BE 102 may perform one or more actions based on the identified intents. It is understood that, as discussed below, the shared NLU annotator 127 may correspond to the semantic extraction subsystem 150 , and the NLU predictor may correspond to the semantic search subsystem 152 of the NLU framework 104 . can be

5 is a flow diagram illustrating a process 145 in which the behavior engine (BE) 102 and the NLU framework 104 perform their respective roles within an embodiment of the agent automation framework 100 . In the illustrated embodiment, the NLU framework 104 processes the received user utterance 122 to create artifacts 140 (eg, intents and/or entity) based on the intent-entity model 108 . ) are extracted. The extracted artifacts 140 may be implemented as a set of symbols representing the intents and entities of the user utterance 122 in a form consumable by the BE 102 . As such, these extracted artifacts 140 are provided to the BE 102 , which processes the received artifacts 140 based on the dialog model 110 , thereby providing a received user utterance 122 . ) in response to determining appropriate actions 142 (eg, changing a password, creating a record, purchasing an item, closing an account) and/or virtual agent utterances 124 . do. As indicated by arrow 144 , process 145 continues as agent automation framework 100 receives and handles additional user utterances 122 from the same user and/or other users in a conversational format. can be repeated.

As shown in FIG. 5 , it can be seen that, in certain circumstances, no further action or communication may occur once the appropriate actions 142 have been performed. Also, although user utterance 122 and agent utterance 124 are discussed herein as being delivered using a written conversation medium or channel (eg, chat, email, ticketing system, text messages, forum posts), , in other embodiments, speech-to-text and/or text-to-speech modules or plug-ins may convert oral user utterance 122 to text and/or text-based agent utterance 124 to speech in accordance with the present disclosure. It should be noted that it can be included to enable voice interactive systems by converting Further, in certain embodiments, both user utterances 122 and virtual agent utterances 124 are stored in database 106 ( For example, in the corpus of utterances 112 ).

As mentioned, the NLU framework 104 includes two major subsystems that work together to transform the difficult problem of NLU into a manageable search problem: a semantic extraction subsystem and a semantic search subsystem. For example, FIG. 6 is a block diagram illustrating the roles of the semantic extraction subsystem 150 and the semantic retrieval subsystem 152 of the NLU framework 104 in an embodiment of the agent automation framework 100 . In the illustrated embodiment, the right portion 154 of FIG. 6 includes the NLU framework 104 for receiving the intent-entity model 108, including sample utterances 155 for each of the various artifacts of the model. The semantic extraction subsystem 150 of The semantic extraction subsystem 150 generates an understanding model 157 that includes semantic representations 158 (eg, utterance tree structures) of sample utterances 155 of the intent-entity model 108 . . In other words, the understanding model 157 includes semantic expressions 158 for facilitating a search (eg, comparison and matching) by the semantic search subsystem 152 , as discussed in more detail below. is a transformed or augmented version of the intent-entity model 108 . As such, the right portion 154 of FIG. 6 is generally, for example routinely, performed prior to receiving the user utterance 122 on a scheduling basis or in response to updates to the intent-entity model 108 . it can be seen that

For the embodiment shown in FIG. 6 , the left portion 156 is a semantic extraction subsystem that also receives and processes user utterances 122 to generate a utterance semantic model 160 having at least one semantic representation 162 . (150) is shown. As discussed in greater detail below, these semantic expressions 158 and 162 are data structures having a form that captures the grammatical syntactic structure of one understanding of an utterance, and subtrees of data structures are semantic of parts of the utterance. Contains subtree vectors that encode semantics. As such, for a given utterance, the corresponding semantic representation is syntactically and semantically in a common semantic representation format that allows for retrieval, comparison, and matching by the semantic retrieval subsystem 152 , as described in more detail below. It captures both meaning. Thus, the semantic expressions 162 of the utterance semantic model 160 can be generally thought of as equivalent to a search key, whereas the semantic expressions 158 of the understanding model 157 can be searched for by a search key. define space. Accordingly, the semantic search subsystem 152 searches the semantic representations 158 of the understanding model 157 for one or more artifacts that match the semantic representations 162 of the utterance semantic model 160 , as described below. By finding , the extracted artifacts 140 are generated.

As an example of one of the semantic expressions 158 and 162 disclosed herein, FIG. 7 is a diagram illustrating an example of an utterance tree 166 generated for an utterance. As will be appreciated, the utterance tree 166 is a data structure generated by the semantic extraction subsystem 150 based on a user utterance 122 , or alternatively, based on one of the sample utterances 155 . . In the example shown in FIG. 7 , the firing tree 166 has an exemplary utterance, “I want to go to the store next to the mall today to buy a blue collared shirt and black pants and also return some defective batteries. "based on The illustrated utterance tree 166 is a set of nodes 202 arranged in a tree structure (eg, nodes 202A, 202B, 202C, 202D, 202E, 202F, 202G, 202H, 202I, 202J, 202K, 202L, 202M, 202N, and 202P)), each node representing a particular word or phrase of an exemplary utterance. It may be noted that each of the nodes 202 may also be described as representing a particular subtree of the utterance tree 166 , and a subtree may include one or more nodes 202 .

The shape or shape of the utterance tree 166 shown in FIG. 7 is determined by the semantic extraction subsystem 150 and represents the syntactic grammatical meaning of the exemplary utterance. More specifically, the prosody subsystem of the semantic extraction subsystem 150 decomposes the utterance into intent segments, while the structure subsystem of the semantic extraction subsystem 150 constructs the utterance tree 166 from these intent segments. do. Each of the nodes 202 stores or references a respective word vector (eg, token) determined by the vocabulary subsystem to indicate the semantic meaning of a particular word or phrase of the utterance. As noted, each word vector is an ordered n -dimensional list (eg, a 300-dimensional list) of floating-point values (eg, a 1×N or N×1 matrix) that provides a mathematical representation of the semantic meaning of a portion of an utterance. )am.

Further, in other embodiments, each of the nodes 202 is annotated by the structure subsystem with additional information about the word or phrase represented by the node to form an annotated embodiment of the utterance tree 166 . can be For example, each of the nodes 202 may include a respective tag, identifier, shading, or cross hatching indicating the class annotation of each node. In particular, for the example utterance tree 166 shown in FIG. 7 , certain subtrees or nodes (eg, nodes 202A, 202B, 202C, and 202D) are verb nodes with part-of-speech labels. or tags, and specific subtrees or nodes (e.g., nodes 202E, 202F, 202G, 202H, 202I, and 202J) may be annotated to be subject or object nodes. , and specific subtrees or nodes (eg, nodes 202K, 202L, 202M, 202N, and 202P) are defined by the structure subsystem as modifier nodes (eg, subject modifier nodes, object modifier nodes). nodes, verb modifier nodes). These class annotations may then be used by semantic retrieval subsystem 152 when comparing semantic expressions generated from the annotated utterance trees. As such, it can be seen that the utterance tree 166 in which semantic expressions are generated serves as a basis (eg, an initial basis) for artifact extraction.

As acknowledged herein, to facilitate generation of extracted artifacts 140 , semantic retrieval subsystem 152 may implement any two or more semantic representations, such as a semantic representation of utterance semantic model 160 . A degree of similarity between one or more of s 162 and one or more of semantic representations 158 of comprehension model 157 may be determined. For example, FIG. 8 is an information flow diagram illustrating an embodiment of a semantic search subsystem 152 operating within a search space 250 . As described above, the search space 250 of this embodiment is populated with and defined by the semantic representations 158 of the understanding model 157 . In other embodiments, the NLU framework 104 may provide a search space 250 based on the uncovering of semantic representations 158 generated by a plurality of understanding models, such as understanding models each appropriate to a particular context or domain. ) can be created. Accordingly, user utterances 122 are received (eg, by prosody subsystem) and segmented into potential artifacts (eg, by semantic extraction subsystem 150) and each semantic representations After being converted to 162 , the semantic search subsystem 152 compares the semantic expressions 162 of the user utterances 122 with the semantic expressions 158 in the search space 250 . Indeed, as discussed in greater detail below, semantic search subsystem 152 identifies any suitable matching semantic expressions 158 from search space 250 , from which NLU framework 104 extracts Artifacts 140 that have been identified can be identified.

In some embodiments, semantic search subsystem 152 identifies one or more semantic expressions 158 as appropriate matches for each of semantic expressions 162 of user utterance 122 . For example, in response to receiving three semantic expressions 162 corresponding to one or more user utterances 122 , the semantic search subsystem 152 of certain embodiments may be configured to: Returns one or more matching semantic representations 158 of the search space 250 . Semantic retrieval subsystem 152 also provides matching semantics to facilitate appropriate agent responses 124 and/or actions 142 for artifacts 140 most likely extracted from semantic expressions 158 . Representations 158 and/or artifacts therein may be scored with an accompanying confidence level.

As will be appreciated herein, semantic retrieval subsystem 152 intervenes between semantic representations 158 and 162 to facilitate determination of a targeted arsenal of semantic representations 158 and extracted artifacts 140 . A predictive similarity scoring scheme can be used to progressively render a more accurate similarity score of . In such embodiments, the predictive similarity scoring scheme may be used to evaluate searches over large-scale representations of search space 250 , such as those generated based on semantic representations 158 of multiple or extended understanding models 157 . make it possible to perform In general, the similarity scoring subsystem of the semantic search subsystem 152 is responsible for determining between a particular utterance-based semantic representation 162 and each semantic representation 158 (eg, search space semantic representation) within the search space 250 . It works by first identifying a list of mathematical comparison functions that enable comparison. For semantic representations 158 that do not have a comparable form to semantic representation 162 , the similarity scoring subsystem can identify the semantic representations 158 as incompatible and omnipotent them from the search space 250 . there is. The similarity scoring subsystem then constructs the widest or least expensive function therein to generate an initial similarity score between the particular semantic representation 162 and the remaining comparable semantic representations 158 of the search space 250 . By applying, each of the mathematical comparison function lists is implemented. As discussed below, semantic retrieval subsystem 152 may use more computationally expensive functions (eg, semantic expressions 158 ) because semantic expressions 158 that are not adequately similar to a particular semantic expression 162 in search space 250 are pruned. For example, one may proceed with using functions that take into account additional nodes, functions that take into account additional dimensions of the same number of nodes, functions that query databases such as dictionaries or external language models). Accordingly, the semantic search subsystem 152 systematically narrows down to the appropriate semantic representations 158 that provide the extracted artifacts 140 .

By way of example, FIG. 9 is an information flow diagram illustrating an embodiment of a similarity scoring subsystem 260 that may be implemented within the semantic search subsystem 152 of the NLU framework 104 . As discussed below, similarity scoring subsystem 260 searches for and uses mathematical comparison functions to iteratively compare any suitable number of semantic expressions to each other via increasingly expensive functions. As an example, this embodiment of FIG. 9 illustrates the functionality of similarity scoring subsystem 260 in which first semantic representation 262 and second semantic representation 264 are compared to semantic representations 158 of search space 250 . However, it should be understood that the techniques discussed below are applicable to each semantic representation of the NLU framework 104 . As will be appreciated, the first semantic representation 262 may correspond to a first one of the semantic expressions 162 discussed above, and the second semantic representation 164 is a second meaning of the semantic representations 162 . can respond to expressions. In other embodiments, semantic expressions 262 , 264 may each be derived from utterance 266 discussed primarily herein as corresponding to user utterance 122 , although sample utterances 155 discussed above. You can respond to one of them.

In general, each semantic expression 262 , 264 is assigned based on the shape of the semantic expression 262 , 264 (eg, utterance tree structure and part-of-speech tagging) 0, 1, or a plurality of cognitive constructing grammars. (CCG) belongs to the type class. That is, based on CCG techniques, the similarity scoring subsystem 260 provides nodes (eg, word vectors and/or word It is recognized that it has a shape or structure (eg, defined by an utterance tree or other suitable meta-structure) that includes parts-of-speech tags for a combination of vectors. Accordingly, the similarity scoring subsystem 260 based on the shapes of the semantic expressions 262 and 264 to identify the appropriate matching semantic expressions 158 that include artifact matches to the semantic expressions 262 and 264 . so that searches can be performed.

In the illustrated embodiment, similarity scoring subsystem 260 includes a shape class database 270 that includes a shape class table 272 therein. Although primarily discussed as a table, the shape class table 272 may be implemented in other embodiments with any suitable data structure. In some embodiments, the shape class database 270 and the shape class table 272 may be stored within the database 106 of the agent automation framework 100 . As will be appreciated herein, each entry 275 (eg, a shape class entry) in the shape class table 272 is a one-to-one shape class comparison (CCG shape class) supported by the semantic search subsystem 152 . Also called comparison). In particular, the shape class table 272 includes a first axis 273 associated with the CCG form of the first semantic expression and a second axis 274 associated with the CCG form of the second semantic expression being compared. Each axis label is associated with a shape pattern for each of the respective CCG forms supported by the similarity scoring subsystem 260, eg, a verb-driven phrase, a noun-driven phrase, etc., and supported CCG forms of f ₁ -f _N It is represented by an appropriate function identifier within the scope. Therefore, it should be understood that a shape pattern for a specific semantic representation defines CCG shape class membership for that specific semantic representation.

In this embodiment, the shape class table 272 contains entries for each intersection of two of the CCG types ( 275), respectively. It should be understood that the shape class table 272 may include any suitable number of entries 275 corresponding to each possible permutation of the compared CCG shape classes. In particular, the semantic representations each belonging to the same CCG shape class are themselves comparable to each other, and the comparison discussed below is shown within each entry 275 along the central diagonal 276 of the shape class table 272 . Represented by a function list. As currently illustrated, shape class table 272 has a reflective line of symmetry along central diagonal 276 indicating that the comparison functions of this embodiment of shape class table 272 are commutative. That is, comparing the first semantic expression with the second semantic expression produces the same result as comparing the second semantic expression with the first semantic expression. In other embodiments, the shape class table 272 may not include a line of reflection symmetry, so that the similarity scoring subsystem 260 is based on the order or direction in which the semantic expressions are being compared to the comparison function discussed below. Allows the list to be adjusted. As a specific example, one entry 275 of the form class table 272 indicates that a semantic expression having a verb-dominant CCG form may be compared with other semantic expressions having a verb-dominant CCG form, a noun-driven CCG form, and the like. It can be specified that there is In the present embodiments, similarity scoring subsystem 260 determines that the pair of semantic expressions are not comparable in response to determining that entry 275 for comparison is empty (eg, null, undefined). not, and thus do not perform comparisons between incomparable semantic expressions.

As mentioned, the entry 275 of the shape class table 272 for each supported CCG shape class comparison of the similarity scoring subsystem 260 also includes a similarity scoring subsystem 260 or one or more Directed to a mathematical comparison function list 278 (eg, form-algebraic function list, processing rules) having a function 280 (eg, comparison function) of Functions 280 of each mathematical comparison function list 278 allow each semantic representation of semantic expressions 262 , 264 to be compared with search space 250 , as described in more detail below. It is a set of nested functions that provide progressively more expensive scoring functions. A list of mathematical comparison functions 278 is used for vector algebra, cosine similarity functions, queries to external databases, and/or for similarity scoring subsystem 260 to determine similarity scores between any suitable number of semantic expressions. any other suitable mathematical functions or formulas available. It should be appreciated that functions 280 may further define a previous function of mathematical comparison function list 278 , or, alternatively, may be completely independent of previous functions 280 . In some embodiments, the mathematical comparison function list 278 for each entry 275 of the shape class table 272 is manually specified by linguists or users, derived by ML techniques, or the like.

In general, the functions 280 of the mathematical comparison function list 278 are responsive to the considered portions of the semantic expressions 262 , 264 that appropriately match the semantic expressions 158 of the search space 250 , respectively. The similarity between the semantic expressions 262 , 264 and the comparables in the search space 250 is scored, respectively, by giving a similarity score that exceeds a certain threshold score. In certain embodiments, functions 280 are, in response to each semantic expression 158 excluding or not matching significant or significant nodes of the corresponding search key semantic expression 262 , 264 , Zeros may be assigned to, or otherwise penalized, the similarity score associated with each semantic representation 158 of search space 250 . As will be appreciated, the similarity scoring subsystem 260 is configured to perform a comparison based on the shape class compatibility rules of the shape class database 270 , as indicated by empty entries 275 of the shape class table 272 . Do not compare the semantic expression with other semantic expressions that have inappropriate CCG forms.

In other embodiments, the similarity scoring subsystem may in some embodiments immediately assign a similarity score of zero to semantic representations of uncomparable pairs. In further embodiments, similarity scoring subsystem 260 may provide a mathematical comparison function having functions 280 that cause similarity scoring subsystem 260 to generate a similarity score of zero between non-comparable semantic expressions. Comparison may be performed by implementing list 278 . In such embodiments, the mathematical comparison function lists 278 naturally cause the similarity scoring subsystem 260 to either zero or Since null similarity scores may be assigned, the shape class table 272 may include a list of mathematical comparison functions 278 suitable for each entry 275 of the shape class table 272 .

Further, in certain embodiments, similarity scoring subsystem 260 may receive representations of a plurality of expressions of utterance 266 from utterance semantic model 160 . For example, semantic expressions 262 , 264 may be included in utterance semantic model 160 as representing alternative forms to utterance 266 . In general, each of the semantic expressions 262 , 264 (generated by the semantic extraction subsystem 150 and included in the utterance semantic model 160 ) is an appropriately distinct semantic representation corresponding to artifacts of the utterance 266 . indicates By considering each comparable pair of semantic expressions 262 , 264 , the similarity scoring subsystem 260 of this embodiment provides a more exhaustive search for the corresponding extracted artifacts 140 , or the corresponding extraction Multiple interpretations of the utterance 266 may be evaluated to create a larger net of artifacts.

6 to discuss a specific example, the semantic extraction subsystem 150 of certain embodiments may include a first alternative semantic expression in which the utterance 266 "book meeting" corresponds to "request reservation or scheduling a meeting" and It may be determined to have a second alternative semantic expression corresponding to “meeting about a book”. Semantic extraction subsystem 150 may then incorporate these alternative semantic expressions into utterance semantic model 160 . By propagating all of these alternative semantic expressions through semantic search subsystem 152 to later stages of the similarity scoring process, similarity scoring subsystem 260 provides an appropriate It is more likely to identify a list of mathematical comparison functions 278 .

With the above description of the components of the similarity scoring subsystem 260 in mind, FIG. 10 shows that the similarity scoring subsystem 260 illustrates the first semantic representation 262 of FIG. 9 and the search space 250 of FIG. 8 . A flow diagram illustrating an embodiment of a process 300 for retrieving one of a list of mathematical comparison functions 278 that enables comparison between semantic expressions 158 . It should be understood that process 300 may be iterated in parallel or processed individually for each utterance-based semantic representation 162 considered by similarity scoring subsystem 260 of semantic search subsystem 152 . As described above, the semantic search subsystem 152 , including a similarity scoring subsystem 260 therein, is configured to provide semantics for utterances 266 from the utterance semantic model 160 defined by the semantic extraction subsystem 150 . Representation 262 may be received. Similarity scoring subsystem 260 may retrieve semantic representation 262 directly from semantic extraction subsystem 150 in other embodiments. The steps illustrated and/or described as part of the semantic extraction subsystem 150 may be stored in a suitable memory (eg, memory 86 ), and may be stored in a client instance (eg, within data center 22 ). 42 or a suitable processor (eg, processor 82 ) associated with enterprise instance 125 (eg, hosted by cloud-based platform 20 ).

As such, similarity scoring subsystem 260 beginning the illustrated embodiment of process 300 determines a CCG form 304 (eg, a cognitive building grammar form) of the semantic representation 262 (block 302 ). ). As described above, the CCG form 304 is formed by the nodes of the semantic expression 262 , parsed with respect to the part-of-speech annotations or tagging of tokens (eg, words or phrases) within the nodes. It can be any suitable description of the shape formed. For example, similarity scoring subsystem 260 determines that semantic expression 262 has a CCG form 304 corresponding to verb-pronoun-noun phrase, noun-verb-direct adjective phrase, noun-verb-adverb phrase, etc. can be identified

In this embodiment, similarity scoring subsystem 260 applies a set of processing rules 306 (eg, shape processing rules) and evaluates its correspondence to a set of processing rules 306 . determines the CCG shape 304 for the semantic representation 262 by assigning the CCG shape 304 to the semantic representation 262 based on the In other embodiments, similarity scoring subsystem 260 may be configured in any other suitable manner, for example by receiving CCG form 304 from semantic extraction subsystem 150 or by ML based pattern matching techniques. The CCG form 304 may be determined. As an example of ML-based pattern matching, the similarity scoring subsystem 260 compares the shape with the part-of-speech tags of the semantic representation 262 to the semantic representations 158 of the understanding model 157 to identify similarities between them. may determine an appropriate or closest matching CCG form 304 for the semantic representation 262 . As will be appreciated, similarity scoring subsystem 260 may include any suitable plug-in or other processing components that enable determination of CCG form 304 of semantic representation 262 .

Once the CCG form 304 for the semantic representation 262 is identified, the similarity scoring subsystem 260 retrieves zero, one, or multiple matching form classes for the CCG form 304 from the form class database 270 ( 312) is determined (block 310). Matching shape classes 312 may be identified by pattern matching by querying table-type embodiments or the like of the shape class database 270 . More specifically, the similarity scoring subsystem 260 of the present embodiments finds the corresponding entries 275 in the shape class table 272 having the mathematical comparison function lists 278 therein, thereby meaning Determines which shape classes the representation 262 can be compared to. Indeed, as noted above, each entry 275 of the shape class table 272 for a particular CCG type 304 is each CCG that can be compared with the CCG type 304 of the semantic representation 262 . One of the mathematical comparison function lists 278 for shapes (eg, shape class compatibility) can, of course, include a set of functions 280 for which similarity can be calculated.

Accordingly, the similarity scoring subsystem 260 following the process 300 is associated with the CCG form 304 of the semantic expression 262 and each entry 275 (eg, each matching form class 312 ). A respective list of mathematical comparison functions 278 representing the identified matches between CCG forms of different semantic expressions is retrieved from each entry 275 of the form class table 272 (block 314). In the present embodiment, similarity scoring subsystem 260 identifies one match type class 312 and outputs one mathematical comparison function list 278 , but any other suitable number of mathematical comparison function lists 278 . ) can be searched according to the number of shape class matches. For example, similarity scoring subsystem 260 may provide a first list of mathematical comparison functions 278 indicating how to compare a semantic representation 262 with a semantic representation having the same CCG form 304, as well as a semantic representation ( 262 ) and a second list of mathematical comparison functions 278 enabling comparison between other semantic expressions having different CCG forms 304 .

To illustrate the use of mathematical comparison function list 278 , FIG. 11 shows one mathematical comparison to compare semantic representation 262 with search space semantic representation 330 (eg, in search space 250 ). A diagram of an embodiment of a similarity scoring subsystem 260 of a semantic search subsystem 152 using a function list 278 . In the present embodiment, search space semantic representation 330 is one of semantic representations 158 of search space 250 , although it should be understood that the present techniques may be used to compare any suitable semantic representations. As described above, the mathematical comparison function list 278 may have more computational resources and/or from the compared semantic expressions 262 , 330 in the functions 280 located deeper within the mathematical comparison function list 278 . It contains an ordered set of functions 280 that progressively use data (eg, a larger number of nodes). As will be appreciated herein, the mathematical comparison function list 278 is first ordered into the less computationally expensive functions 280 , and as potential semantic expressions 158 to be included in the search space 250 are omnipresent, more Proceed to use expensive functions. As such, similarity scoring subsystem 260 generally determines similarity scores 340 (eg, predictive similarity scores) that are subsequently used and more accurate via more expensive functions 280 .

For example, during first comparison 350 , similarity scoring subsystem 260 compares root node 354 of semantic representation 262 with search space root node 356 of search space semantic representation 330 . For this purpose, a first function 352 may be implemented. As noted above, the first function 352 is the least expensive function 280 of the mathematical comparison function list 278 . Similarity scoring subsystem 260 may then determine a first similarity score 360 that describes a local (eg, low-depth, cheapest to compute) similarity between semantic representations 262 , 330 . there is. Considering that the first function 352 is the least expensive function 352 of the list of mathematical comparison functions 278 , the root nodes 354 and 356 have semantic expressions compared via the first function 352 ( It should be understood that other portions, which are examples of portions 262 , 330 , including the entirety of each semantic expression 262 , 330 , may be compared in the first comparison 350 . The similarity scores 340 indicate the degree of similarity or correspondence between forms including parts-of-speech tags for two semantic expressions, such as values assigned within a similarity scoring range of 0 to 1, 0 to 5, 0 to 10, etc. It may be any suitable mathematical expression. Since at least similarity scores 340 describe generic semantic and form-based similarity between considered portions of two semantic expressions, similarity scores 340 described herein are determined between two intent vectors. It should be understood that this is separate from the similarity measures described above.

In the present embodiment of FIG. 11 , the root nodes 354 , 356 evaluated during the first comparison 350 are shown as hollow circles. As shown by the solid circles, the remaining dependent nodes of the semantic expressions 262 , 330 are effectively “covered” (eg, not considered) during the first comparison 350 . Extending this methodology to subsequent comparisons discussed below, in each comparison nodes shown with hollow circles (eg, considered nodes 362 ) are evaluated in each comparison, whereas solid nodes are evaluated in each comparison. Nodes shown as circles (not considered nodes 364) are not evaluated.

In general, similarity scoring subsystem 260 provides the least expensive Utilizes the order of functions 280 from most expensive to most expensive. In some embodiments, similarity scoring subsystem 260 selectively "uncovers" (e.g., more nodes of semantic expressions 262, 330) as similarity scoring subsystem 260 progresses through subsequent comparisons. for example) can be considered.

For example, after determining the first similarity score 360 , the similarity scoring subsystem 260 of this embodiment generates a more accurate second similarity score 372 based on the increased number of considered nodes 362 . Proceed to implement a second function 368 within the second comparison 370 it creates. The similarity scoring subsystem 260 then considers the same number of nodes and considers a greater number of dimensions of the word vectors associated with the considered nodes 362, thereby making it more expensive, in an external database, or any other A third comparison 374 is performed which produces an additional accurate third similarity score 376 via a third function 378 referencing the appropriate functions or operations. Similarity scoring subsystem 260 subsequently performs a final comparison 380 that produces the most accurate final similarity score 382 via a fourth function 384 , which may be the most expensive function 280 commonly used. . More details regarding exemplary comparisons 350 , 370 , 374 , 380 between semantic expressions 262 , 330 are provided below.

Also, although not shown, similarity scoring subsystem 260 may use weights (eg, derived from ML techniques) within comparisons or It should be understood that coefficients can be implemented. For example, to process search engine requests, similarity scoring subsystem 260 may increase the contribution of modifiers to nouns or verbs. Alternatively, during purchase order requests provided to a chatbot embodiment of the NLU framework 104 , the similarity scoring subsystem 260 may increase the contribution of nouns to modifiers or verbs. In some embodiments, similarity scoring subsystem 260 learns weights from semantic representations 158 within NLU framework 104 during operation, and thus may be developed to become more discoursed over time.

As will be appreciated herein, similarity scoring subsystem 260 of certain embodiments may also analyze the similarity between two semantic representations having different sizes by comparing appropriate combinations of subtree vectors of semantic representations. For example, similarity scoring subsystem 260 may determine a combination (eg, a centroid or weighted average) for a subtree vector of nodes of a longer semantic representation that do not have comparable counterparts in the shorter semantic representation. there is. Accordingly, the similarity scoring subsystem 260 of these embodiments may analyze the modified semantic representation (including comparisons instead of comparable nodes) for shorter semantic representations, in accordance with the present techniques.

In certain embodiments, similarity scoring subsystem 260 may be one based on the specific content of semantic expressions 262, 330, the specific context in which similarity scoring subsystem 260 is operating, the desired granularity of the similarity scoring process, etc.; Certain interdependent embodiments of functions 280 may be applied to expand the portion of the considered nodes for the other or both semantic representations 262 , 330 . For example, if a particular deep similarity scoring process is desired, the similarity scoring subsystem 260 does not do the same to the search space semantic representation 330 but rather the number of considered nodes 362 of the semantic representation 262 . can be expanded. During the next comparison, similarity scoring subsystem 260 de-expands considered nodes 362 of semantic representation 262 and search space semantic representation 330 to evaluate each permutation of considered nodes 362 . ) of the considered nodes 362 can be expanded. In other embodiments where a rapid similarity scoring process is performed, similarity scoring subsystem 260 alternatively expands considered nodes 362 of both semantic expressions 262 , 330 for each comparison. can do. Further, while semantic expressions 262 and 330 are shown as having the same shapes and lengths, similarity scoring subsystem 260 may iterate over any suitable semantic expressions having unequal forms and/or lengths. You have to understand that you can compare.

Also, in some situations, the semantic representation 262 of the utterance 266 may be formed in a plurality of CCG forms, such as both a broad noun-dominant CCG form and a more specific CCG form (eg, noun-adjective-verb). It is also recognized that corresponding to (304). As such, similarity scoring subsystem 260 of certain embodiments may identify a plurality of shape classes supported by shape class database 270 against which semantic expression 262 fits. In such embodiments, similarity scoring subsystem 260 may advantageously generate similarity scores for each CCG form 304 describing semantic expression 262 . The similarity scoring subsystem 260 may then aggregate similarity scores for each assigned CCG form of the semantic representation 262 to improve the operation of the agent automation system 100 by increasing the scope of the semantic search. there is.

In any case, similarity scoring subsystem 260 may apply an adjustment function to the similarity scores for each CCG form of the semantic expression, thus generating multiple CCG forms 304 interpretations of the semantic expression 262 . Each similarity scoring result may be adjusted (eg, consolidated) from the comparisons performed. The steering function may, in certain embodiments, be stored in the shape class database 270 . As will be appreciated herein, the adjustment function results from a single interpretation of the CCG form 304 of the semantic representation 262 by the similarity scoring subsystem 260 , for example, by maintaining a maximum or weighted average of the similarity scores. Generate and output aggregated refined similarity scores (eg aggregated similarity scores) that transcend limits. Further, the steering function of certain embodiments may be individually customized for each combination of CCG forms 304 , for each client, in each domain in which the similarity scoring subsystem 260 operates.

12 is a flow diagram illustrating an embodiment of a process 400 in which similarity scoring subsystem 260 may iteratively identify matching semantic expressions from search space 250 using mathematical comparison function list 278 . As described above, the search space 250 is defined by the semantic representations 162 of the at least one understanding model 157 . As will be appreciated, the process 400 allows the similarity scoring subsystem 260 to select each of the semantic representations 162 of the spoken semantic model 160 (corresponding to the semantic representation 262 introduced above) and the search space ( 250) to predictively evaluate the degree of similarity between a vast number of semantic expressions 158. Process 400 may be performed by an appropriate processor (eg, processor(s) 82 ) associated with client instance 42 or enterprise instance 125 , as described above with respect to FIGS. 3 , 4A and 4B . may be executed and stored in a suitable memory (eg, memory 86 ).

Specifically, the similarity scoring subsystem 260 of the illustrated embodiment iterates through each semantic representation 262 of the utterance semantic model 160 in a for-each loop (block 402). It should be appreciated that the similarity scoring subsystem 260 may implement any other suitable processing scheme in lieu of a for-each loop that enables the generation of a refined similarity score 414 for each CCG form 304 . For example, similarity scoring subsystem 260 may alternatively implement a do-while loop, a for loop, a while loop, a do-until loop, or the like. In any case, for each semantic representation 262 of the utterance semantic model 160 , the similarity scoring subsystem determines (block 404 ) the CCG shape of each semantic representation 262 , and determines the associated semantic representation 262 from the shape class database 270 . A list of mathematical comparison functions 278 is retrieved. Upon initializing the iteration parameters for the process 400 based on the CCG shape, the similarity scoring subsystem 260 also selects a first function 280 of the mathematical comparison function list 278 (block 406), of interest Define the search subspace in which there is initially to be the entire search space 250 .

In the illustrated embodiment, the process 400 compares the semantic representation 262 derived from the user utterance 122 with comparable semantic representations 158 of the search subspace (block 410), Generate a set 412 of similarity scores corresponding to comparisons of the semantic representation 262 to comparable semantic representations. In some embodiments, similarity scoring subsystem 260 can determine set 412 of similarity scores based on distances between semantic vectors (eg, subtree vectors) of the compared semantic expressions. As mentioned, the similarity scoring subsystem 260 implementing the first function 352 uses a minimal amount of computational resources. As such, the similarity scoring subsystem 260 is superior to other search systems that may use a single complexity comparison function to systematically compare the semantic representation of a user utterance with each semantic representation in the search space or comprehension model. Shape search and similarity scoring may be performed more quickly and/or efficiently.

For example, referring now to FIG. 13 , which is a similarity scoring subsystem 260 that applies a list of mathematical comparison functions 278 to selectively refine the search space 250 into appropriate search subspaces 502 . shows a schematic diagram of an embodiment of For example, during the first comparison 500 , the similarity scoring subsystem 260 applies a first function 352 to a comparison within the search subspace 502 , initialized to be the entirety of the search space 250 . Possible semantic expressions 158 and semantic expressions 262 may be compared. As will be appreciated herein, this application of the least accurate and most efficient function 352 allows the similarity scoring subsystem 260 to efficiently perform a first pass search across the search subspace 502 . . In the present embodiment, the first function 352 considers the root node 354 of the semantic representation 262 , but other suitable parts of the semantic representation 262 (eg, other nodes or combinations of nodes). It should be understood that this first function 352 may be parsed.

12 , similarity scoring subsystem 260 removes or omits semantic expressions 158 from search subspace 502 (block 414 ), and omniscient semantic expressions 158 obtain a threshold similarity score (eg, (eg, a predetermined threshold score) of the set 412 of similarity scores. In certain embodiments, the threshold similarity score is a predefined value within a range of possible similarity scores that is stored as a parameter in the similarity scoring subsystem 260 . Accordingly, the similarity scoring subsystem 260 can reduce the search subspace 502 to efficiently apply the subsequent functions 280 to the reduced number of semantic representations 158 of the search subspace 502 . there is. Indeed, returning to FIG. 14 , the search subspace 502 is narrowed (eg, shrunk, truncated) after the first comparison 500 so that semantic expressions associated with a similarity score of the set 412 that are below a threshold. Remove 158 from search subspace 502 .

In some embodiments, similarity scoring subsystem 260 receives a user-defined value as a threshold similarity score to remediate a threshold below which semantic expressions 158 of search subspace 502 are ignored. Similarity scoring subsystem 260 may also update a threshold similarity score based on ML techniques, such as based on the particular context in which similarity scoring subsystem 260 is currently operating. For example, similarity scoring subsystem 260 implements and/or predicts a relatively high or selective threshold similarity score (eg, at least 90% matches) for contexts in which a particular search subspace 502 is very large. The similarity scoring process can be further accelerated. Also, the threshold similarity score may be individually selected or updated after each function 280 is applied. More specifically, similarity scoring subsystem 260 disclosed herein is configured to, in response to determining that more than a threshold number of semantic expressions 158 in a previous comparison satisfy a threshold similarity score, for a subsequent comparison a threshold similarity score. It can reduce the value or selectivity of the score.

Returning to process 400 of FIG. 12 , similarity scoring subsystem 260 determines whether the CCG shape search presented by block 410 should continue (block 416 ). As will be appreciated herein, similarity scoring subsystem 260 may decide to continue the CCG shape search based on one or more appropriate stopping conditions being met. For example, similarity scoring subsystem 260 may determine that each semantic expression is omniscient from search subspace 502 (eg, indicating no matches), a threshold number of semantic expressions is remaining in (eg, representing the most probable matches), the recently applied function 280 indicating that the embedded stop conditions in the function 280 have been satisfied, and all of the mathematical comparison function list 278 The CCG shape search may end in response to which functions 280 have been applied or the like.

In response to determining at block 416 that the CCG shape search should continue, the similarity scoring subsystem 260 selects the next function 280 in the list of mathematical comparison functions 278 (block 420). Thereafter, as indicated by arrow 422 , similarity scoring subsystem 260 returns to block 410 with the remaining comparable semantic representations 158 and semantic representations 158 of search subspace 502 . 262) are compared. Accordingly, the similarity scoring subsystem 260 uses the more expensive function 280 of the mathematical comparison function list 278 to provide a set 412 of similarity scores associated with the remaining comparable semantic representations of the search subspace 502 . refines (eg, modifies, updates). After each comparison, similarity scoring subsystem 260 may store an array of various sets 412 of similarity scores generated via process 400 , or alternatively, each previous of set 412 . We can replace the similarity score generated in . Indeed, since more processing resources are used during application of subsequent functions 280 , the set of similarity scores 412 generally improves in accuracy and/or precision as more functions 280 are applied. Based on the set of similarity scores 412 , the similarity scoring subsystem 260 performing the illustrated process 400 generates semantic expressions 158 associated with respective similarity scores of the set 412 that are less than a threshold similarity score. Retrieve the search space 250 of (block 404).

Referring back to FIG. 13 , in order to refine the set 412 of similarity scores and prune the search subspace 502 , the similarity scoring subsystem 260 applies a second function 368 to the reduced size search. The remaining semantic representations 158 in the subspace 502 are compared with the semantic representations 262 . Accordingly, similarity scoring subsystem 260 may further reduce search subspace 502 to suitable candidates that satisfy a threshold similarity score. In certain embodiments, the structure of function 180 is essentially based, for example, on terms within functions 280 of mathematical comparison function list 278, covering or expanding the respective nodes of the compared semantic expressions. to guide For example, the first function 352 may include a single term that compares the root node 354 of the semantic expression with the semantic expressions 158 of the search subspace 502 , while the second function 368 . ) may include one or more terms comparing the expanded portion of the semantic representation 262 to the semantic representations 158 .

Accordingly, the similarity scoring subsystem 260 provides progressively more accurate and more expensive functions 280 to the persisting (eg, in-beam) semantic representations 158 of the search subspace 502 for a given comparison. is applied repeatedly. Continuing the discussion of process 400 of FIG. 12 with respect to FIG. 13 , similarity scoring subsystem 260 implements third function 378 during third comparison 516 to implement the same expression of semantic representation 262 . The covered portion 510 may be compared to an additionally omnipresent embodiment of the search subspace 502 and the like. As such, the similarity scoring subsystem 260 of this embodiment implements the final function 384 during the final comparison 524 to match the entire 526 of the semantic expression 262 with the final embodiment of the search subspace 502 . By designing to compare, it preserves the use of the most computationally intensive final function 384 for the significantly reduced number of remaining semantic representation 158 candidates. Indeed, in certain cases, the final comparison 524 of FIG. 13 utilizes all of the information available within the semantic representation 262 , the semantic representation from the search space 250 appropriately matching the semantic representation 262 . A set of 158 may be created. In other embodiments, as described above, the final function 384 may consider the portion of the semantic representation 262 through any other suitable resource intensive process, such as by querying an external language model.

As such, returning to FIG. 12 , the similarity scoring subsystem 260 determines at block 416 that the stopping conditions of the CCG form search have been met, and thereafter the set of matching semantic expressions identified from the search subspace 502 ( 430) can be printed. Accordingly, the similarity scoring subsystem 260 will efficiently identify the set of matching semantic expressions 430 for subsequent determination of the extracted artifacts 140 and provide them to other components of the NLU framework 104 . can

Technical effects of the present disclosure provide an agent automation framework implementing a semantic search subsystem capable of skillfully narrowing (eg, windowing) a search space filled with semantic expressions derived from sample utterances. thereby improving the identification of semantic expressions that properly match the semantic expression of the received user utterance. The present embodiments particularly relate to a similarity scoring subsystem of a semantic retrieval subsystem that efficiently and predictively compares semantic expressions via CCG techniques. That is, based on the identified CCG form of a particular semantic expression, the similarity scoring subsystem may determine a list of mathematical comparison functions that enable quantifications of the similarity between the semantic expressions. The comparison functions in this list are ordered chronologically to enable the similarity scoring subsystem to perform the most efficient and least expensive comparisons between semantic expressions and determine similar scores between them. Based on the similarity scores, the similarity scoring subsystem iteratively identifies particularly similar semantic expressions within the search space, prunes (eg, reduces, reduces) the search space for these semantic expressions, and then the semantic expression It is possible to implement more computationally intensive comparison functions for the same number of nodes or an increased number of nodes. That is, the iterative application of selective node uncovering and/or increased resource utilization typically results in narrowing down potentially matching semantic expressions within the search space, taking into account more data of compared semantic expressions, via increasingly complex comparison functions. all. As such, the present techniques for predictive similarity scoring enable targeted discovery of semantic expression matches that allow the NLU framework to efficiently extract artifacts, while reducing the computational overhead of enterprise-level natural language with multiple users. Reduce to an appropriate level for involvement.

The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments are capable of various modifications and alternative forms. It is further to be understood that the claims are not limited to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure.

The techniques presented and claimed herein clearly improve the technical field, and thus refer to and apply to material objects and specific examples of a practical nature that are not abstract, intangible, or purely theoretical. Also, if any claims appended to the end of this specification include one or more elements designated as "means for [performing] [the function]" or "steps for [performing] [the function]," These factors are 35 USC 112(f) is intended to be construed. However, for any claims that include elements designated in any other way, such elements are subject to 35 U.S.C. It is not intended to be construed under 112(f).

Claims (20)

An agent automation system comprising:
a memory configured to store a natural language understanding (NLU) framework comprising a similarity scoring subsystem having a form class database; and
a processor configured to execute instructions to cause a similarity scoring subsystem of the NLU framework to perform actions
Including, the actions are
receiving a semantic representation of a user utterance;
identifying a cognitive construction grammar (CCG) form of the semantic expression;
determining at least one shape class entry in the shape class database that matches a CCG shape of the semantic expression; and
retrieving a list of mathematical comparison functions from the at least one shape class entry, wherein the list of mathematical comparison functions is such that the similarity scoring subsystem compares at least a portion of the semantic expression to at least a search space portion of the search space semantic expression between them. to determine the similarity score of -
Including, agent automation system. According to claim 1,
wherein the list of mathematical comparison functions comprises an ordered set of functions that enable the similarity scoring subsystem to progressively use more computationally expensive functions to compare the semantic representation with the search space semantic representation. According to claim 1,
The search space semantic expression is one of a plurality of search space semantic expressions defining a search space of the NLU framework, and the instructions cause the similarity scoring subsystem to: and performing actions comprising searching the search space for a subset of spatial semantic representations. According to claim 1,
The instructions cause the similarity scoring subsystem to determine the similarity score between the semantic expression and the search space semantic expression by comparing the semantic expression to the search space semantic expression via the list of mathematical comparison functions. An agent automation system that allows to perform actions, including 5. The method of claim 4,
The similarity scoring subsystem comprises:
comparing a first root node of the semantic representation to a second root node of the search space semantic representation to determine the similarity score; and
comparing a first root node and a first dependent node of the semantic expression with a second root node and a second dependent node of the search space semantic expression to refine the similarity score;
and compare the semantic representation with the search space semantic representation by 6. The method of claim 5,
The instructions cause the similarity scoring subsystem to:
determining whether the similarity score is below a predetermined threshold score; and
disregarding the search space semantic expression from subsequent comparisons with the semantic expression in response to determining that the similarity score is below the predetermined threshold score.
An agent automation system for performing actions comprising: According to claim 1,
wherein the semantic representation comprises a utterance tree structure comprising a root node and at least one dependent node semantically coupled to the root node. 8. The method of claim 7,
at least a part of the semantic expression is the root node, and at least a search space part of the search space semantic expression is a search space root node of the search space semantic expression, so that the similarity score is a subtree vector of the root node and the search space An agent automation system that quantifies the similarity between subtree vectors of a root node. According to claim 1,
The CCG form comprises a first CCG form, the at least one form class entry comprises a first form class entry, the mathematical comparison function list comprises a first mathematical comparison function list, and the instructions include: the scoring subsystem,
identifying a second CCG form of the semantic expression;
determining a second shape class entry in the shape class database that matches a second CCG shape of the semantic expression; and
retrieving a second list of mathematical comparison functions from the second shape class entry.
An agent automation system for performing actions comprising: 10. The method of claim 9,
The instructions cause the similarity scoring subsystem to:
determining a first similarity score by comparing the semantic expression with the search space semantic expression through the first mathematical comparison function list;
determining a second similarity score by comparing the semantic expression with the search space semantic expression through the second list of mathematical comparison functions; and
aggregating the first similarity score and the second similarity score into an aggregated similarity score via an adjustment function, wherein the aggregated similarity score determines whether the similarity scoring subsystem determines whether the search space semantic expression is a match to the semantic expression. enable -
An agent automation system for performing actions comprising: A method of operating a similarity scoring subsystem of an agent automation system, comprising:
For each search space semantic expression of a plurality of search space semantic expressions defining a search space of the agent automation system, a first portion of the semantic expression is divided into a plurality of the plurality of search space semantic expressions through a plurality of comparison functions. generating a plurality of initial similarity scores compared to the portion of the search space;
pruning the search space of dissimilar search space semantic expressions of the plurality of search space semantic expressions, each of the dissimilar search space semantic expressions being less than a predetermined score threshold for each of the plurality of initial similarity scores provides a similarity score;
increasing the number of considered nodes between the plurality of search space semantic expressions and the semantic expressions remaining in the search space;
For each of the plurality of search space semantic expressions remaining in the search space, a plurality of refined similarity scores by comparing the plurality of search space semantic expressions with the considered nodes of the semantic expression through the plurality of comparison functions creating a;
pruning a search space of dissimilar search space semantic expressions that provides a similarity score for each of the plurality of refined similarity scores that is less than the predetermined score threshold; and
identifying a semantic expression match for the semantic expression from any search space semantic expressions of the plurality of search space semantic expressions remaining in the search space;
A method comprising 12. The method of claim 11,
wherein increasing the number of considered nodes comprises considering additional nodes of the semantic representation and additional search space nodes of the plurality of search space semantic representations remaining in the search space. 12. The method of claim 11,
After pruning the search space of any dissimilar search space semantic expressions having a similarity score for each of the plurality of refined similarity scores that is less than the predetermined score threshold, determining whether each node of the semantic expression has been considered. step; and
in response to determining that at least one node of the semantic representation has not been considered, returning to increasing the number of considered nodes between the semantic representation and the plurality of search space semantic representations remaining in the search space. step
A method comprising 12. The method of claim 11,
identifying a Cognitive Construction Grammar (CCG) form of the semantic expression;
determining at least one shape class entry in a shape class database that matches a CCG shape of the semantic expression, and
retrieving a list of mathematical comparison functions including the plurality of comparison functions from the at least one shape class entry;
determining the plurality of comparison functions by 12. The method of claim 11,
The determining of at least one form class entry matching the CCG form of the semantic expression may include: comparing, by the form class database, the CCG form of the semantic expression to at least one search space semantic expression among the plurality of search space semantic expressions. and identifying that it contains a list of mathematical comparison functions for comparison with possible CCG forms. 12. The method of claim 11,
determining that the semantic expression includes a plurality of CCG forms;
determining a respective refined similarity score for comparing the semantic expression having a respective CCG form interpretation from the plurality of CCG forms to the plurality of search space semantic expressions; and
aggregating the respective refined similarity score for each CCG shape interpretation into an aggregated similarity score, via an adjustment function of the similarity scoring subsystem;
A method comprising 17. The method of claim 16,
wherein the adjustment function specifies that the aggregate similarity score is a maximum or weighted average of the respective refined similarity scores for each CCG form. A non-transitory computer-readable medium having stored thereon instructions, comprising:
The instructions, when executed by one or more processors of an agent automation system, cause the agent automation system to cause the similarity scoring subsystem to:
identify a cognitive building grammar (CCG) form of the semantic expression corresponding to the received user utterance;
determine at least one shape class entry in a shape class database that matches a CCG shape of the semantic expression;
retrieve a list of mathematical comparison functions corresponding to the CCG form of the semantic expression, wherein the list of mathematical comparison functions enables the similarity scoring subsystem to progressively compare the semantic expression and the search space semantic expression;
iteratively applying each function of the list of mathematical comparison functions to determine a similarity score that quantifies the degree of similarity between a first considered portion of the semantic expression and a second considered portion of the search space semantic expression; and
successive functions in the list of mathematical comparison functions until the similarity score indicates a degree of similarity between all of the semantic expressions and at least a portion of the search space semantic expressions, or until the subsequent function is the most expensive function in the list of mathematical comparison functions. Refining the similarity score by iteratively applying
comparing the semantic expression with the search space semantic expression by
A non-transitory computer-readable medium that makes it tangible. 19. The method of claim 18,
wherein the search space semantic representation is one of a plurality of search space semantic representations defining a search space of the agent automation system. 20. The method of claim 19,
The instructions cause the agent automation system to cause the similarity scoring subsystem to:
after each refinement of the similarity score, narrow the search space to similar search space semantic representations of the plurality of search space semantic representations that provide respective similarity scores above a predetermined score threshold;
identifying the similar search space semantic expressions among the plurality of search space semantic expressions remaining in the search space as matches to the semantic expression after each similarity score indicates a degree of similarity between all of the semantic expressions;
A non-transitory computer-readable medium configured to be implemented in

Images