US20220222441A1 - Machine learning based named entity recognition for natural language processing - Google Patents
Machine learning based named entity recognition for natural language processing Download PDFInfo
- Publication number
- US20220222441A1 US20220222441A1 US17/202,183 US202117202183A US2022222441A1 US 20220222441 A1 US20220222441 A1 US 20220222441A1 US 202117202183 A US202117202183 A US 202117202183A US 2022222441 A1 US2022222441 A1 US 2022222441A1
- Authority
- US
- United States
- Prior art keywords
- natural language
- model
- ner
- context
- language sentence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010801 machine learning Methods 0.000 title claims description 16
- 238000003058 natural language processing Methods 0.000 title abstract description 13
- 238000000034 method Methods 0.000 claims description 55
- 238000003860 storage Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 abstract description 105
- 238000004519 manufacturing process Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 26
- 230000014509 gene expression Effects 0.000 description 25
- 230000004044 response Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 6
- 238000011144 upstream manufacturing Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 230000006872 improvement Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000013434 data augmentation Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 241000590419 Polygonia interrogationis Species 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
- H04L51/18—Commands or executable codes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/237—Lexical tools
- G06F40/247—Thesauruses; Synonyms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
- G06F40/35—Discourse or dialogue representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/253—Grammatical analysis; Style critique
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/55—Rule-based translation
- G06F40/56—Natural language generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
Definitions
- the disclosure relates in general to machine learning based natural language processing, and more specifically to machine learning based named entity recognition for natural language processing.
- Named entity recognition is a commonly used operation performed by many natural language processing (NLP) tasks.
- Named entity recognition is used in NLP services such as conversational BOTs that allow users to interact with online systems using unstructured text, for example, natural language sentences.
- Named entity recognition is performed using linguistic grammar-based techniques as well as machine learning based models such as neural networks.
- Machine learning based techniques for NER typically require a large amount of manually annotated training data. This requires significant resources in generating the training data.
- trained machine learning based models for performing named entity recognition have lower accuracy in production since they are trained using training data that is different from real conversations. For example, real conversations may include short utterances such as single word responses. Conventional techniques typically fail to perform accurate named entity recognition in such situations.
- FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment.
- FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment.
- FIG. 4 shows a flowchart illustrating the process of generating context expressions according to an embodiment.
- FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment.
- FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment.
- FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment.
- FIG. 8 shows a flowchart illustrating the overall process of using the NER model for processing sentences according to an embodiment.
- FIG. 9 is a block diagram illustrating a functional view of a typical computer system according to one embodiment.
- NER Named entity recognition
- chatbots or conversation engines Conversational engines or allow users to interact with web services through text or speech.
- an NER model is invoked to help extract entity information and formalize free text to structured text.
- the structured text is further processed by the system, for example, to determine the user intent and to automatically generate a response for providing to the user.
- Named entity recognition is also referred to as named entity identification or entity extraction.
- Named entity recognition includes locating and classifying named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, time expressions, quantities, monetary values, percentages, and so on.
- An example of unstructured text is an utterance referring to a sentence, a phrase, or a group thereof.
- NER models may fail to determine that “April” is a name of a person for various reasons. This is so, because most NER models are trained using public open datasets. These training datasets typically have full sentences. Conventional NER models typically learn from other words within the sentence to help identify the entity. However, in typical interactions with chatbots, user reply may have very short sentences, for example single word utterances. During the training phase, the NER model may never see such short answer data. A typical NER model may mis-classify the word “April” as a time expression or date since most training datasets typically include a Date entity “April.” However, in the above example, “April” is the name of a person.
- Embodiments use context information in named entity recognition.
- Embodiments include the context of a sentence during NER model training and execution.
- the system according to various embodiments generates high quality contextual data for training NER models.
- the system utilizes labeled and unlabeled contextual data for training NER models.
- the NER models are provided for execution in production environments.
- the system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information.
- Context information helps the system determine the intent of an utterance and boosts the performance for named entity recognition. Accordingly, the accuracy of the context-based NER model is higher than conventional NER models, thereby allowing the online system 120 to determine the named entities in a sentence of phrase more accurately.
- FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment.
- the system environment 100 includes an online system 120 that communicates with users via client devices 115 .
- the online system 120 includes a conversation engine 130 , a named entity recognition model 140 , and a data store 160 .
- the online system 120 may include other components not shown in FIG. 1 , for example, various applications, other types of data stores, and so on.
- the system environment 100 may include other elements not shown in FIG. 1 , for example, a network.
- the conversation engine 130 stores the instructions that allow the online system 120 to perform natural language conversations with users 110 .
- a conversation engine may also be referred to as a chatbot, a dialog system, virtual assistant, or artificial intelligence (AI) assistant.
- the conversation engine 130 receives a natural language request from a user and determines the action that the user requested via the natural language request.
- the conversation engine 130 uses the named entity recognition model 140 for performing NER tasks for use in analyzing user utterances to determine how to respond to user utterances.
- the conversation engine 130 invokes NER model 140 to perform NER for extracting key information from user utterances and to formalize free text in user utterances to structured text.
- the conversation engine 130 determines a context based on the current conversation with a user.
- the context of a conversation is represented by a sequence of recent utterances that may have been provided by the user or automatically generated by the online system 120 .
- the conversation engine 130 analyzes a natural language text by combining the text with the context to obtain a combined string 135 .
- the combined string 135 is provided as input to the named entity recognition model 140 .
- the named entity recognition model 140 provides an output text 145 annotated with entity information.
- the named entity recognition model 140 may remove the context from the combined string 135 .
- the output text 145 annotated with entity information is used by the conversation engine 130 for further processing.
- the conversation engine 130 may use the entity information to accurately determine an intent of the user. Accordingly, the conversation engine 130 may analyze the utterance based on the entity information to determine a type of information that the user may be requesting. The conversation engine 130 may perform processing to determine the information and provide to the user as a generated response.
- the online system 120 is a multi-tenant system that stores data of multiple tenants. Chatbots are often used by multi-tenant systems to allow tenants of the multi-tenant systems to configure chatbots for specific purposes.
- Each tenant may be an enterprise.
- a tenant may be a company that employs people that use the multi-tenant system to manage their processes.
- a multi-tenant system may store data for multiple tenants in the same physical database but enforce a logical separation of data across tenants. As a result, one tenant does not have access to another tenant's data, unless such data is expressly shared.
- the online system 120 stores data in data store 160 that may be processed in order to respond to requests from users. For example, a permitted action may update data stored in data store 160 , search for data stored in data store 160 , and so on.
- the data store 160 stores data for users of the multi-tenant system.
- the data store 160 stores information describing entities represented in the multi-tenant system. Each entity has an entity type. Each entity stores values in fields or data fields. Examples of entity types include account, case, opportunity, and so on.
- An entity representing an account stores information describing a user or an enterprise; an entity representing an opportunity represents a potential customer and stores fields representing information identifying the potential customer, a status of the opportunity, the last interaction with the potential customer; an entity representing a case stores information describing an interaction between an enterprise and a customer and stores fields comprising the date of the interaction, the type of the interaction, and other details of the interaction.
- the data store 160 may be implemented as a relational database storing one or more tables. Each table contains one or more data categories logically arranged as columns or fields. Each row or record of a table contains an instance of data for each category defined by the fields.
- a data store 160 may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc.
- multi-tenant system implements a web-based customer relationship management (CRM) system.
- the multi-tenant system may include application servers configured to implement and execute CRM software applications as well as provide related data, code, forms, webpages and other information to and from client devices 115 and to store to, and retrieve from, a database system related data, objects, and webpage content.
- CRM customer relationship management
- the online system 120 and client devices 115 shown in FIG. 1 represent computing devices.
- a computing device can be a conventional computer system executing, for example, a MicrosoftTM WindowsTM-compatible operating system (OS), AppleTM OS X, and/or a Linux OS.
- a computing device can also be a device such as a personal digital assistant (PDA), mobile telephone, video game system, etc.
- PDA personal digital assistant
- the client devices 115 may interact with the online system 120 via a network (not shown in FIG. 1 ).
- the network uses a networking protocol such as the transmission control protocol/Internet protocol (TCP/IP), the user datagram protocol (UDP), internet control message protocol (ICMP), etc.
- TCP/IP transmission control protocol/Internet protocol
- UDP user datagram protocol
- ICMP internet control message protocol
- the data exchanged over the network can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc.
- FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment.
- the online system 120 includes the conversation engine 150 , the named entity recognition model 140 , a training data generation module 210 , a training module 220 , and a training data store 230 .
- Other embodiments can have different and/or other components than the ones described here, and that the functionalities can be distributed among the components in a different manner.
- the training data generation module 210 generates training data for training the named entity recognition model 140 .
- a tenant of a multi-tenant system may not have sufficient data to train the named entity recognition model 140 .
- the training data generation module 210 can take a small amount of training data and generates additional training data for training the training data generation module 210 .
- the training data is stored in the training data store 230 .
- the training module 220 trains the named entity recognition model 140 using the training data.
- the various modules of the online system 120 execute the various processes further described herein.
- FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment.
- Other embodiments can perform the operations of FIG. 3 in different orders.
- Other embodiments can include different and/or additional steps than the ones described herein.
- the training data generation module 210 generates data by performing the steps 310 , 315 , and 320 .
- the training data generation module 210 generates 310 context expressions.
- the training data generation module 210 generates 315 answer expressions.
- the training data generation module 210 concatenates 320 the generated context expressions and the generated answer expressions to generate training data that is stored in the training data store 230 .
- the training module 220 uses the generated training data to train 325 the named entity recognition model 140 . The details of these steps are further described herein.
- FIG. 4 shows a flowchart illustrating the process of generating 310 context expressions according to an embodiment.
- Other embodiments can perform the operations of FIG. 4 in different orders.
- Other embodiments can include different and/or additional steps than the ones described herein.
- the training data generation module 210 uses context information for contextual data generation. Accordingly, the training data generation module 210 uses high quality context data that: (1) conveys the information relevant for understanding the user's answer, and (2) covers as many expressions as possible for comprehensive training and evaluation. The use of high quality data improves NER performance.
- the process illustrated in FIG. 4 is a two phase context generation method.
- the training data generation module 210 generates a high quality context seed list.
- the training data generation module 210 receives manually generated commonly used context expressions based on domain knowledge. For example, to improve for PERSON entity detection performance, the training data generation module 210 starts with common expressions that ask for a person's name, such as, “what's your name?”, “May I know your name”, and so on. For most domains, in this stage, the training data generation module 210 may use 10 to 30 common expressions.
- the training data generation module 210 diversifies context expressions.
- the training data generation module 210 diversifies the context expressions with data augmentation techniques.
- the training data generation module 210 trains a better generalized NER model and evaluates the model more fairly.
- Phrase re-generation the training data generation module 210 generates an alternative statement that carries the same meaning but with different words. The system may use a store of equivalent words and phrases to convert a sentence to alternative sentences.
- the training data generation module 210 gets high quality context data as it is based on manual generation. However, the number of the expressions may be limited. During the second stage, the training data generation module 210 extends the number of expressions while not losing its information. After the second stage, the training data generation module 210 may get a few hundred meaningful context expressions.
- FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment.
- the training data generation module 210 builds a pipeline to generate answers.
- the training data generation module 210 receives 510 a training dataset comprising utterances.
- the training data generation module 210 extracts 515 entities from the training dataset.
- the training data generation module 210 generates 520 answers based on the extracted entities.
- the training data generation module 210 uses (1) positive answers, which contains a potential entity, and (2) negative answers, which are common user responses but do not contain any entities. Negative data is used during data generation, to prevent the model from overfitting and avoiding learning that any user answer after context contains an entity.
- the training data generation module 210 may start from open public NER datasets. From the open dataset, the training data generation module 210 extracts entities based on the label for specific domains. The extracted entities (e.g., names like “John Doe”) could either be used directly as single/short word utterance or combined with simple utterance prefixes, for example, “my name is”, “I am”, to build a full sentence containing entities.
- entities e.g., names like “John Doe”
- simple utterance prefixes for example, “my name is”, “I am”, to build a full sentence containing entities.
- the training data generation module 210 executes a process similar to that discussed above for context generation. For first stage, the training data generation module 210 manually generates common negative expressions not containing any entities, e.g. “Ok, sure”, “Why?”. For second stage, the training data generation module 210 extends the number of negative expressions via data augmentation.
- the context may include text that occurs before a term as well as text that occurs after the term. For example, for processing a document the system may look at context after answers.
- FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment.
- the training data generation module 210 receives 610 a list of contexts and answers.
- the training data generation module 210 assigns each context in the list to different answers and concatenates 630 the respective contexts and answer pairs.
- the NER model is transformer based and the training data generation module 210 puts a separator ([SEP]) token between context and answer.
- a NER model input is “[CLS] context [SEP] answer [SEP]”.
- the training data generation module 210 may add additional token or tokens to distinguish between context and answer, for example, “[CLS] [CTX] context [SEP] [ASW] answer [SEP]”.
- the additional tokens help the model to focus on the answer for named entity recognition and simplify post-processing.
- the training data generation module 210 directly concatenates the context and the answer via ⁇ space> or another white space character, for example, “context ⁇ space> answer”, and lets the NER model learn to detect the entity.
- the training data generation module 210 may use various methods for assigning context to user answer. In an embodiment, the training data generation module 210 performs random assignment. If a system has enough context and answer data, the training data generation module 210 randomly chooses a context to assign to an answer. In an embodiment, the training data generation module 210 performs cross product assignment. If the system has less than a threshold amount of context and answer data, the system iteratively assigns each context to each answer. The generated context answer pair is a cross product of the set of contexts and the set of answers.
- Random Assignment may generate two inputs “[CLS] what's your name [SEP] Jim” and “[CLS] may I know your name? [SEP] John”.
- Cross Product Assignment generates four inputs: “[CLS] what's your name [SEP] John”, “[CLS] may I know your name? [SEP] Jim”, “[CLS] what's your name [SEP] John”, and “[CLS] may I know your name? [SEP] Jim”.
- the training module 220 implements an end-to-end model training pipeline based on deep learning NLP (natural language processing) techniques.
- FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment.
- the training module 220 performs an upstream pre-training 710 of the NER model using unlabeled data.
- the training module 220 further performs downstream training 720 that results in fine-tuning of NER model for specific tasks.
- the training module 220 provides 730 the trained model for execution for various NLP tasks.
- Upstream pre-training 710 is used to train a task agnostic base NER model which can be used for multiple downstream tasks. Since the pre-training is task agnostic, the model training uses (1) language model type of loss to train the model to understand the text, and (2) next sentence prediction and contrastive loss type of loss to train the model to understand the relationship between two sentences. An upstream base model pre-trained with such losses generalizes well on multiple different downstream tasks.
- the training module 220 may use unlabeled contextual data directly. For example, the training module 220 starts with open public conversational datasets.
- the system fine-tunes the NER model for specific downstream task.
- the labeled data may be generated via the above described methods performed for contextual data generation by the training data generation module 210 .
- labeled data is used as an additional dataset.
- the system may use an NLP model, for example, a transformer based deep learning model.
- NLP model for example, a transformer based deep learning model.
- the model encodes the input in the format of “[CLS] context [SEP] answer [SEP]”.
- the encoded input is processed by the model and for each token of the encoded input, the model generates an output embedding.
- the output embedding is used as input for another softmax layer to help decide the entity type tag for each token.
- the number of the softmax layer outputs corresponds to the label space, the number of the entity types.
- the NER model architecture is a stacked model of multiple transformer encoder layers.
- the system uses unlabeled data for pre-training of a raw model to obtain a base model that is trained to process conversation and context.
- the system fine-tunes the base NER model to obtain the final context based NER model.
- the model uses both unlabeled data as well as labeled data.
- the system performs following steps (described in further details below): (1) Passing context as part of the request to the NER service; (2) Making decision whether to use context; (3) Passing context along with input text to the model; (4) Preparing the final response.
- An API application programming interface
- An optional parameter that has the context and is passed to the service from the client along with the actual text/utterance being processed.
- An example request body is as follows:
- modelId “ENTITIES”, // additional optional parameter - context list “context”: [“May I have the name of your company to locate your account?”], “utterance”: “MyCompany” ⁇
- the system may use the different heuristics to determine whether or not to pass the context to the model.
- the system may use one or more of (1) utterance length or (2) context intent to determine whether to use the context or not.
- the system uses the length of the utterance to determine if the system should pass context to the model.
- Short utterances may not contain enough information to enable the model to predict entities.
- the short utterance may include only one keyword corresponding to the entity. There is no and no other text surrounding it.
- the system adds context to utterances that are short in length (typically less than 3 words but the threshold is configurable). The advantage to this approach is that it is simple and easy to implement.
- the system examines the context and determines whether the information present in the context is required by the model to make a correct prediction for the actual utterance. For each prediction request, the system first uses an ML (machine learning) model to predict whether the context is important for that particular prediction request.
- the system trains a relatively lightweight binary intent model such as a neural network based model on a curated dataset of contexts for this purpose. For example, the system may use a simple convolutional neural network based model, or even logistic regression.
- One of the use cases for this is conversational texts. For example, “What is your name?” is a context that has relevant information for the model to make a correct prediction on an utterance, “John Doe”.
- the advantage of this approach is that it is a more sophisticated heuristic, and it helps ensure that the system only adds context when necessary. This helps reduce overall NER model prediction times by supplying shorter utterances.
- the conversation engine 150 or any NLP task performed by a system that executes the NER model executes pre-processing phase to prepare the input to the NER model. If the system decides to use context, the system combines the context and the input text together and passes the combined text to the NER model for prediction. The process used for combining the context and input text depends on the process used for training the NER model. For example, if the NER model was trained using training data that was combined as “[CLS] context [SEP] text [SEP]” the system combines the context and answer using the same approach during execution of the NER model.
- the system uses the context as supplemental information to predict the entities present in the combined text.
- the system may store the length of the context that is being prepended to the utterance.
- the system uses the length information as explained in the section below to extract the processed input text from the output.
- the system executes a post-processing step that involves removing the entities present in the context from the output response generated using the NER model.
- the generated response includes the tags/entities present in context as well. Since the requester, for example, the user that invokes the service is interested in the entities present in the input text, the system removes the tags predicted by the model for the context (and any special tokens that were also passed).
- the system maintains the length of the context as described in connection with the pre-processing step.
- the system uses the length to strip off the predictions comprising the tags included in the context portion of the response generated using the NER.
- Remaining tags/entities represent the tags/entities in the input text/utterance and are provided to the requestor as the final response.
- the system improves upon the accuracy of conventional systems.
- a conventional NER model that does not use the context information to determine the entity information failed on utterances that were short, for example, for a conversation in which the agent asked a question ‘what is your name? and the user responded ‘A1’ or a conversation in which the agent asked a question ‘Which company do you work for?’ and the user responded salesforce.
- the conventional NER model failed to detect that the user utterance “A1” in the first example was a person and the user utterance “salesforce’ in the second example was an organization.
- the NER model according to embodiments disclosed was able to determine the correct entity with the help of context information.
- the model that uses the context information may be computationally intensive. As a result, the ability of embodiments to determine whether to use the context or use the utterance without the context for performing named entity recognition ensures that the computational expense of using the context is incurred only when necessary, thereby improving the computational efficiency of the process along with the accuracy.
- the accuracy of the NER models was measured in terms of F1 score that is determined as a combination of precision and recall for the model.
- a conventional NER model was compared with a base NER model that was re-trained with input starting with token “[CLS]” and an ending with token “[SEP]”. No contextual data was used for the base model.
- the base NER model performed better than conventional NER model for detecting person entities showing an improvement of 0.33 in F1 score for person entities.
- the model showed improvement for organization entities also, but not as noticeable.
- FIG. 9 is a high-level block diagram illustrating a functional view of a typical computer system for use as one of the entities illustrated in the environment 100 of FIG. 1 according to an embodiment. Illustrated are at least one processor 902 coupled to a chipset 904 . Also coupled to the chipset 904 are a memory 906 , a storage device 908 , a keyboard 910 , a graphics adapter 912 , a pointing device 914 , and a network adapter 916 . A display 918 is coupled to the graphics adapter 912 .
- the functionality of the chipset 904 is provided by a memory controller hub 920 and an I/O controller hub 922 .
- the memory 906 is coupled directly to the processor 902 instead of the chipset 904 .
- the storage device 908 is a non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device.
- the memory 906 holds instructions and data used by the processor 902 .
- the pointing device 914 may be a mouse, track ball, or other type of pointing device, and is used in combination with the keyboard 910 to input data into the computer system 900 .
- the graphics adapter 912 displays images and other information on the display 918 .
- the network adapter 916 couples the computer system 900 to a network.
- a computer 900 can have different and/or other components than those shown in FIG. 9 .
- the computer 900 can lack certain illustrated components.
- a computer system 900 acting as a multi-tenant system 120 may lack a keyboard 910 and a pointing device 914 .
- the storage device 908 can be local and/or remote from the computer 900 (such as embodied within a storage area network (SAN)).
- SAN storage area network
- the computer 900 is adapted to execute computer modules for providing the functionality described herein.
- module refers to computer program instruction and other logic for providing a specified functionality.
- a module can be implemented in hardware, firmware, and/or software.
- a module can include one or more processes, and/or be provided by only part of a process.
- a module is typically stored on the storage device 908 , loaded into the memory 906 , and executed by the processor 902 .
- a client device 115 may be a mobile phone with limited processing power, a small display 918 , and may lack a pointing device 914 .
- the multi-tenant system 120 may comprise multiple blade servers working together to provide the functionality described herein.
- any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment.
- the appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
- Coupled and “connected” along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.
- the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion.
- a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
- “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
Abstract
Description
- This application claims the benefits of U.S. Provisional Application No. 63/136,831, filed on Jan. 13, 2021, which is incorporated by reference in its entirety.
- The disclosure relates in general to machine learning based natural language processing, and more specifically to machine learning based named entity recognition for natural language processing.
- Named entity recognition (NER) is a commonly used operation performed by many natural language processing (NLP) tasks. Named entity recognition is used in NLP services such as conversational BOTs that allow users to interact with online systems using unstructured text, for example, natural language sentences.
- Named entity recognition is performed using linguistic grammar-based techniques as well as machine learning based models such as neural networks. Machine learning based techniques for NER typically require a large amount of manually annotated training data. This requires significant resources in generating the training data. Furthermore, trained machine learning based models for performing named entity recognition have lower accuracy in production since they are trained using training data that is different from real conversations. For example, real conversations may include short utterances such as single word responses. Conventional techniques typically fail to perform accurate named entity recognition in such situations.
-
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment. -
FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment. -
FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment. -
FIG. 4 shows a flowchart illustrating the process of generating context expressions according to an embodiment. -
FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment. -
FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment. -
FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment. -
FIG. 8 shows a flowchart illustrating the overall process of using the NER model for processing sentences according to an embodiment. -
FIG. 9 is a block diagram illustrating a functional view of a typical computer system according to one embodiment. - The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein.
- The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as “104A,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “104,” refers to any or all of the elements in the figures bearing that reference numeral.
- Named entity recognition (NER) models are used for various natural language processing tasks, for example, in chatbots or conversation engines. Conversational engines or allow users to interact with web services through text or speech. During a conversation between the chatbot and a user, an NER model is invoked to help extract entity information and formalize free text to structured text. The structured text is further processed by the system, for example, to determine the user intent and to automatically generate a response for providing to the user.
- Named entity recognition is also referred to as named entity identification or entity extraction. Named entity recognition includes locating and classifying named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, time expressions, quantities, monetary values, percentages, and so on. An example of unstructured text is an utterance referring to a sentence, a phrase, or a group thereof.
- Existing techniques for NER perform fail to perform accurate named entity recognition in various situations. For example, when the chatbot asks the user: “what's your name?”, and the user replies: “April”. In this scenario, conventional NER models may fail to determine that “April” is a name of a person for various reasons. This is so, because most NER models are trained using public open datasets. These training datasets typically have full sentences. Conventional NER models typically learn from other words within the sentence to help identify the entity. However, in typical interactions with chatbots, user reply may have very short sentences, for example single word utterances. During the training phase, the NER model may never see such short answer data. A typical NER model may mis-classify the word “April” as a time expression or date since most training datasets typically include a Date entity “April.” However, in the above example, “April” is the name of a person.
- Embodiments use context information in named entity recognition. Embodiments include the context of a sentence during NER model training and execution. The system according to various embodiments generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The NER models are provided for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information.
- Context information helps the system determine the intent of an utterance and boosts the performance for named entity recognition. Accordingly, the accuracy of the context-based NER model is higher than conventional NER models, thereby allowing the
online system 120 to determine the named entities in a sentence of phrase more accurately. - Although applications of NER are described in connection with conversational engines, the techniques disclosed are not limited to conversational engines but can be applied to other domains. For example, analysis of documents that processes an unstructured document and annotates it with additional information describing various keywords that occur within the document.
-
FIG. 1 is a block diagram of a system environment illustrating use of an NER model in a conversation engine, according to one embodiment. Thesystem environment 100 includes anonline system 120 that communicates with users viaclient devices 115. Theonline system 120 includes aconversation engine 130, a namedentity recognition model 140, and adata store 160. Theonline system 120 may include other components not shown inFIG. 1 , for example, various applications, other types of data stores, and so on. Thesystem environment 100 may include other elements not shown inFIG. 1 , for example, a network. - The
conversation engine 130 stores the instructions that allow theonline system 120 to perform natural language conversations with users 110. A conversation engine may also be referred to as a chatbot, a dialog system, virtual assistant, or artificial intelligence (AI) assistant. Theconversation engine 130 receives a natural language request from a user and determines the action that the user requested via the natural language request. Theconversation engine 130 uses the namedentity recognition model 140 for performing NER tasks for use in analyzing user utterances to determine how to respond to user utterances. During a conversation between theconversation engine 130 and user, theconversation engine 130 invokesNER model 140 to perform NER for extracting key information from user utterances and to formalize free text in user utterances to structured text. - The
conversation engine 130 determines a context based on the current conversation with a user. The context of a conversation is represented by a sequence of recent utterances that may have been provided by the user or automatically generated by theonline system 120. Theconversation engine 130 analyzes a natural language text by combining the text with the context to obtain a combinedstring 135. The combinedstring 135 is provided as input to the namedentity recognition model 140. The namedentity recognition model 140 provides anoutput text 145 annotated with entity information. The namedentity recognition model 140 may remove the context from the combinedstring 135. Theoutput text 145 annotated with entity information is used by theconversation engine 130 for further processing. For example, if the input text was an utterance from a user, theconversation engine 130 may use the entity information to accurately determine an intent of the user. Accordingly, theconversation engine 130 may analyze the utterance based on the entity information to determine a type of information that the user may be requesting. Theconversation engine 130 may perform processing to determine the information and provide to the user as a generated response. - In some embodiments, the
online system 120 is a multi-tenant system that stores data of multiple tenants. Chatbots are often used by multi-tenant systems to allow tenants of the multi-tenant systems to configure chatbots for specific purposes. Each tenant may be an enterprise. For example, a tenant may be a company that employs people that use the multi-tenant system to manage their processes. A multi-tenant system may store data for multiple tenants in the same physical database but enforce a logical separation of data across tenants. As a result, one tenant does not have access to another tenant's data, unless such data is expressly shared. - The
online system 120 stores data indata store 160 that may be processed in order to respond to requests from users. For example, a permitted action may update data stored indata store 160, search for data stored indata store 160, and so on. Thedata store 160 stores data for users of the multi-tenant system. In an embodiment, thedata store 160 stores information describing entities represented in the multi-tenant system. Each entity has an entity type. Each entity stores values in fields or data fields. Examples of entity types include account, case, opportunity, and so on. An entity representing an account stores information describing a user or an enterprise; an entity representing an opportunity represents a potential customer and stores fields representing information identifying the potential customer, a status of the opportunity, the last interaction with the potential customer; an entity representing a case stores information describing an interaction between an enterprise and a customer and stores fields comprising the date of the interaction, the type of the interaction, and other details of the interaction. - The
data store 160 may be implemented as a relational database storing one or more tables. Each table contains one or more data categories logically arranged as columns or fields. Each row or record of a table contains an instance of data for each category defined by the fields. For example, adata store 160 may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc. - In an embodiment, multi-tenant system implements a web-based customer relationship management (CRM) system. For example, the multi-tenant system may include application servers configured to implement and execute CRM software applications as well as provide related data, code, forms, webpages and other information to and from
client devices 115 and to store to, and retrieve from, a database system related data, objects, and webpage content. - The
online system 120 andclient devices 115 shown inFIG. 1 represent computing devices. A computing device can be a conventional computer system executing, for example, a Microsoft™ Windows™-compatible operating system (OS), Apple™ OS X, and/or a Linux OS. A computing device can also be a device such as a personal digital assistant (PDA), mobile telephone, video game system, etc. - The
client devices 115 may interact with theonline system 120 via a network (not shown inFIG. 1 ). The network uses a networking protocol such as the transmission control protocol/Internet protocol (TCP/IP), the user datagram protocol (UDP), internet control message protocol (ICMP), etc. The data exchanged over the network can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. -
FIG. 2 is a block diagram of the system architecture of an online system, according to an embodiment. Theonline system 120 includes theconversation engine 150, the namedentity recognition model 140, a trainingdata generation module 210, atraining module 220, and atraining data store 230. Other embodiments can have different and/or other components than the ones described here, and that the functionalities can be distributed among the components in a different manner. - The training
data generation module 210 generates training data for training the namedentity recognition model 140. For example, a tenant of a multi-tenant system may not have sufficient data to train the namedentity recognition model 140. The trainingdata generation module 210 can take a small amount of training data and generates additional training data for training the trainingdata generation module 210. The training data is stored in thetraining data store 230. Thetraining module 220 trains the namedentity recognition model 140 using the training data. The various modules of theonline system 120 execute the various processes further described herein. -
FIG. 3 shows a flowchart illustrating the overall process of generating training data for training the NER model according to an embodiment. Other embodiments can perform the operations ofFIG. 3 in different orders. Other embodiments can include different and/or additional steps than the ones described herein. - The training
data generation module 210 generates data by performing thesteps data generation module 210 generates 310 context expressions. The trainingdata generation module 210 generates 315 answer expressions. The trainingdata generation module 210concatenates 320 the generated context expressions and the generated answer expressions to generate training data that is stored in thetraining data store 230. Thetraining module 220 uses the generated training data to train 325 the namedentity recognition model 140. The details of these steps are further described herein. -
FIG. 4 shows a flowchart illustrating the process of generating 310 context expressions according to an embodiment. Other embodiments can perform the operations ofFIG. 4 in different orders. Other embodiments can include different and/or additional steps than the ones described herein. - The training
data generation module 210 uses context information for contextual data generation. Accordingly, the trainingdata generation module 210 uses high quality context data that: (1) conveys the information relevant for understanding the user's answer, and (2) covers as many expressions as possible for comprehensive training and evaluation. The use of high quality data improves NER performance. - The process illustrated in
FIG. 4 is a two phase context generation method. In the first stage, the trainingdata generation module 210 generates a high quality context seed list. For this stage, the trainingdata generation module 210 receives manually generated commonly used context expressions based on domain knowledge. For example, to improve for PERSON entity detection performance, the trainingdata generation module 210 starts with common expressions that ask for a person's name, such as, “what's your name?”, “May I know your name”, and so on. For most domains, in this stage, the trainingdata generation module 210 may use 10 to 30 common expressions. - In the second stage, the training
data generation module 210 diversifies context expressions. For this stage, the trainingdata generation module 210 diversifies the context expressions with data augmentation techniques. After this stage with more context expressions, the trainingdata generation module 210 trains a better generalized NER model and evaluates the model more fairly. - According to various embodiments, the data augmentation methods used by the system include: (1) Casing Optimization: the training
data generation module 210 converts the first letter of multiple words to uppercase, not just the first word of the sentence, e.g. transforming “What's your name”=>[“what's your name”, “What's Your Name”]. This helps NER model to focus on the context semantics rather than casing. (2) Question Mark: the trainingdata generation module 210 randomly removes question marks from context questions, e.g. transforming “what's your name?”=>[“what's your name”]. This helps the NER model avoid overfitting on question mark. (3) Word switching: The trainingdata generation module 210 switches certain words with other words that could occur in that context, whether or not they are equivalent words “May I know your first name?”=>[“May I know your last name?”, “Can I know your first name?”]. (4) Phrase re-generation: the trainingdata generation module 210 generates an alternative statement that carries the same meaning but with different words. The system may use a store of equivalent words and phrases to convert a sentence to alternative sentences. - After the first stage, the training
data generation module 210 gets high quality context data as it is based on manual generation. However, the number of the expressions may be limited. During the second stage, the trainingdata generation module 210 extends the number of expressions while not losing its information. After the second stage, the trainingdata generation module 210 may get a few hundred meaningful context expressions. -
FIG. 5 shows a flowchart illustrating the process of generating answer expressions according to an embodiment. In addition to generating context, the trainingdata generation module 210 builds a pipeline to generate answers. The trainingdata generation module 210 receives 510 a training dataset comprising utterances. The trainingdata generation module 210extracts 515 entities from the training dataset. The trainingdata generation module 210 generates 520 answers based on the extracted entities. - For user answers, the training
data generation module 210 uses (1) positive answers, which contains a potential entity, and (2) negative answers, which are common user responses but do not contain any entities. Negative data is used during data generation, to prevent the model from overfitting and avoiding learning that any user answer after context contains an entity. - For positive data generation, the training
data generation module 210 may start from open public NER datasets. From the open dataset, the trainingdata generation module 210 extracts entities based on the label for specific domains. The extracted entities (e.g., names like “John Doe”) could either be used directly as single/short word utterance or combined with simple utterance prefixes, for example, “my name is”, “I am”, to build a full sentence containing entities. - For negative data generation, the training
data generation module 210 executes a process similar to that discussed above for context generation. For first stage, the trainingdata generation module 210 manually generates common negative expressions not containing any entities, e.g. “Ok, sure”, “Why?”. For second stage, the trainingdata generation module 210 extends the number of negative expressions via data augmentation. - The context may include text that occurs before a term as well as text that occurs after the term. For example, for processing a document the system may look at context after answers.
- After the system generates context and user answers, the system concatenates context and answer.
FIG. 6 shows a flowchart illustrating the process of concatenating context and answer expressions to generate training data according to an embodiment. The trainingdata generation module 210 receives 610 a list of contexts and answers. The trainingdata generation module 210 assigns each context in the list to different answers and concatenates 630 the respective contexts and answer pairs. - In some embodiments, the NER model is transformer based and the training
data generation module 210 puts a separator ([SEP]) token between context and answer. For example, a NER model input is “[CLS] context [SEP] answer [SEP]”. Besides [SEP] token, the trainingdata generation module 210 may add additional token or tokens to distinguish between context and answer, for example, “[CLS] [CTX] context [SEP] [ASW] answer [SEP]”. The additional tokens help the model to focus on the answer for named entity recognition and simplify post-processing. In some embodiments, the trainingdata generation module 210 directly concatenates the context and the answer via <space> or another white space character, for example, “context <space> answer”, and lets the NER model learn to detect the entity. - The training
data generation module 210 may use various methods for assigning context to user answer. In an embodiment, the trainingdata generation module 210 performs random assignment. If a system has enough context and answer data, the trainingdata generation module 210 randomly chooses a context to assign to an answer. In an embodiment, the trainingdata generation module 210 performs cross product assignment. If the system has less than a threshold amount of context and answer data, the system iteratively assigns each context to each answer. The generated context answer pair is a cross product of the set of contexts and the set of answers. - As an example, if the training
data generation module 210 has two contexts: [“what's your name?”, “may I know your name?”] and two answers: [“John”, “Jim”], the above methods will generate corresponding results. Random Assignment may generate two inputs “[CLS] what's your name [SEP] Jim” and “[CLS] may I know your name? [SEP] John”. Cross Product Assignment generates four inputs: “[CLS] what's your name [SEP] John”, “[CLS] may I know your name? [SEP] Jim”, “[CLS] what's your name [SEP] John”, and “[CLS] may I know your name? [SEP] Jim”. - The
training module 220 implements an end-to-end model training pipeline based on deep learning NLP (natural language processing) techniques.FIG. 7 shows a flowchart illustrating the process of training the NER model according to an embodiment. Thetraining module 220 performs anupstream pre-training 710 of the NER model using unlabeled data. Thetraining module 220 further performsdownstream training 720 that results in fine-tuning of NER model for specific tasks. Thetraining module 220 provides 730 the trained model for execution for various NLP tasks. - Upstream Pre-Training with Unlabeled Data
-
Upstream pre-training 710 is used to train a task agnostic base NER model which can be used for multiple downstream tasks. Since the pre-training is task agnostic, the model training uses (1) language model type of loss to train the model to understand the text, and (2) next sentence prediction and contrastive loss type of loss to train the model to understand the relationship between two sentences. An upstream base model pre-trained with such losses generalizes well on multiple different downstream tasks. - Besides improved downstream performances, another advantage of upstream pre-training is that it does not require task specific labels. As the
pre-training step 710 uses language model and next sentence prediction or contrastive loss, thetraining module 220 may use unlabeled contextual data directly. For example, thetraining module 220 starts with open public conversational datasets. - Downstream Fine-Tuning with Labeled Data
- With the pre-trained upstream NER model and labeled data for specific downstream task, the system fine-tunes the NER model for specific downstream task. The labeled data may be generated via the above described methods performed for contextual data generation by the training
data generation module 210. For model downstream training for fine-tuning, labeled data is used as an additional dataset. - The system may use an NLP model, for example, a transformer based deep learning model. For the described contextual NER task, the model encodes the input in the format of “[CLS] context [SEP] answer [SEP]”. The encoded input is processed by the model and for each token of the encoded input, the model generates an output embedding. The output embedding is used as input for another softmax layer to help decide the entity type tag for each token. The number of the softmax layer outputs corresponds to the label space, the number of the entity types. In an embodiment, the NER model architecture is a stacked model of multiple transformer encoder layers.
- The system uses unlabeled data for pre-training of a raw model to obtain a base model that is trained to process conversation and context. The system fine-tunes the base NER model to obtain the final context based NER model. During training the model uses both unlabeled data as well as labeled data.
- To use context as part of the service for performing entity recognition, the system performs following steps (described in further details below): (1) Passing context as part of the request to the NER service; (2) Making decision whether to use context; (3) Passing context along with input text to the model; (4) Preparing the final response.
- An API (application programming interface) supports an optional parameter that has the context and is passed to the service from the client along with the actual text/utterance being processed. An example request body is as follows:
-
{ “modelId”: “ENTITIES”, // additional optional parameter - context list “context”: [“May I have the name of your company to locate your account?”], “utterance”: “MyCompany” } - The system may use the different heuristics to determine whether or not to pass the context to the model. The system may use one or more of (1) utterance length or (2) context intent to determine whether to use the context or not.
- According to one embodiment, the system uses the length of the utterance to determine if the system should pass context to the model. Short utterances may not contain enough information to enable the model to predict entities. In some cases, for example, the conversational use case, the short utterance may include only one keyword corresponding to the entity. There is no and no other text surrounding it. To overcome this, the system adds context to utterances that are short in length (typically less than 3 words but the threshold is configurable). The advantage to this approach is that it is simple and easy to implement.
- According to another embodiment, the system examines the context and determines whether the information present in the context is required by the model to make a correct prediction for the actual utterance. For each prediction request, the system first uses an ML (machine learning) model to predict whether the context is important for that particular prediction request. The system trains a relatively lightweight binary intent model such as a neural network based model on a curated dataset of contexts for this purpose. For example, the system may use a simple convolutional neural network based model, or even logistic regression. One of the use cases for this is conversational texts. For example, “What is your name?” is a context that has relevant information for the model to make a correct prediction on an utterance, “John Doe”. The advantage of this approach is that it is a more sophisticated heuristic, and it helps ensure that the system only adds context when necessary. This helps reduce overall NER model prediction times by supplying shorter utterances.
- These two techniques may be used together or separately, depending on an empirical evaluation.
- Passing Context Along with Input Text to the Model
- The
conversation engine 150 or any NLP task performed by a system that executes the NER model executes pre-processing phase to prepare the input to the NER model. If the system decides to use context, the system combines the context and the input text together and passes the combined text to the NER model for prediction. The process used for combining the context and input text depends on the process used for training the NER model. For example, if the NER model was trained using training data that was combined as “[CLS] context [SEP] text [SEP]” the system combines the context and answer using the same approach during execution of the NER model. - Since the NER model is trained to receive context, the system uses the context as supplemental information to predict the entities present in the combined text.
- Apart from preparing the input, the system may store the length of the context that is being prepended to the utterance. The system uses the length information as explained in the section below to extract the processed input text from the output.
- The system executes a post-processing step that involves removing the entities present in the context from the output response generated using the NER model.
- After the NER model makes the prediction on the combined text that also includes context, the generated response includes the tags/entities present in context as well. Since the requester, for example, the user that invokes the service is interested in the entities present in the input text, the system removes the tags predicted by the model for the context (and any special tokens that were also passed).
- To achieve this, the system maintains the length of the context as described in connection with the pre-processing step. The system uses the length to strip off the predictions comprising the tags included in the context portion of the response generated using the NER. Remaining tags/entities represent the tags/entities in the input text/utterance and are provided to the requestor as the final response.
- Following is an example illustrating output scores for various terms that occur in a sentence. The B-name score for the term John is high indicating that this term is a B-name named entity.
-
Score indicating Score indicating Score indicating keyword is keyword is keyword is Keywords Not entity B-Name I-name. what. 0.99 0.05 0.05 is 0.99 0.05 0.05 your 0.99 0.05 0.05 name 0.99 0.05 0.05 John 0.05 0.99 0.05 - The system according to various embodiments improves upon the accuracy of conventional systems. A conventional NER model that does not use the context information to determine the entity information failed on utterances that were short, for example, for a conversation in which the agent asked a question ‘what is your name? and the user responded ‘A1’ or a conversation in which the agent asked a question ‘Which company do you work for?’ and the user responded salesforce. The conventional NER model failed to detect that the user utterance “A1” in the first example was a person and the user utterance “salesforce’ in the second example was an organization. In contrast, the NER model according to embodiments disclosed was able to determine the correct entity with the help of context information. However, the model that uses the context information may be computationally intensive. As a result, the ability of embodiments to determine whether to use the context or use the utterance without the context for performing named entity recognition ensures that the computational expense of using the context is incurred only when necessary, thereby improving the computational efficiency of the process along with the accuracy.
- The accuracy of the NER models was measured in terms of F1 score that is determined as a combination of precision and recall for the model. The F1 score is calculated as F1=2*precision*recall/(precision+recall). A conventional NER model was compared with a base NER model that was re-trained with input starting with token “[CLS]” and an ending with token “[SEP]”. No contextual data was used for the base model. As shown in the table I, the base NER model performed better than conventional NER model for detecting person entities showing an improvement of 0.33 in F1 score for person entities. The model showed improvement for organization entities also, but not as noticeable.
-
TABLE I F1 score of F1 score Single Word # of conventional of base F1 score Entity entities NER model NER model improvement PERSON 1000 0.54 0.87 0.33 ORGANIZATION 1000 0.35 0.45 0.1 - Further performance was measured of the base NER model with NER model trained to use contextual data (referred to as the contextual NER model). As shown in table II significant increase (over 30%) in F1 score was found by using contextual NER model compared to base NER model.
-
TABLE II F1 score of F1 score of # of Base NER Contextual NER Absolute F1 Language entities Model Model Increase English 2031 0.69 0.99 0.3 French 2164 0.76 0.97 0.21 - The techniques disclosed that selectively use the contextual model based on the input utterance or the context improve the execution efficiency of the process. Since use of contextual NER is computationally expensive, the computational efficiency is improved as a result of techniques that avoid invoking the contextual model if equivalent accuracy can be achieved by conventional NER model.
-
FIG. 9 is a high-level block diagram illustrating a functional view of a typical computer system for use as one of the entities illustrated in theenvironment 100 ofFIG. 1 according to an embodiment. Illustrated are at least oneprocessor 902 coupled to achipset 904. Also coupled to thechipset 904 are amemory 906, astorage device 908, akeyboard 910, agraphics adapter 912, apointing device 914, and anetwork adapter 916. Adisplay 918 is coupled to thegraphics adapter 912. In one embodiment, the functionality of thechipset 904 is provided by amemory controller hub 920 and an I/O controller hub 922. In another embodiment, thememory 906 is coupled directly to theprocessor 902 instead of thechipset 904. - The
storage device 908 is a non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. Thememory 906 holds instructions and data used by theprocessor 902. Thepointing device 914 may be a mouse, track ball, or other type of pointing device, and is used in combination with thekeyboard 910 to input data into thecomputer system 900. Thegraphics adapter 912 displays images and other information on thedisplay 918. Thenetwork adapter 916 couples thecomputer system 900 to a network. - As is known in the art, a
computer 900 can have different and/or other components than those shown inFIG. 9 . In addition, thecomputer 900 can lack certain illustrated components. For example, acomputer system 900 acting as amulti-tenant system 120 may lack akeyboard 910 and apointing device 914. Moreover, thestorage device 908 can be local and/or remote from the computer 900 (such as embodied within a storage area network (SAN)). - The
computer 900 is adapted to execute computer modules for providing the functionality described herein. As used herein, the term “module” refers to computer program instruction and other logic for providing a specified functionality. A module can be implemented in hardware, firmware, and/or software. A module can include one or more processes, and/or be provided by only part of a process. A module is typically stored on thestorage device 908, loaded into thememory 906, and executed by theprocessor 902. - The types of
computer systems 900 used by the entities ofFIG. 1 can vary depending upon the embodiment and the processing power used by the entity. For example, aclient device 115 may be a mobile phone with limited processing power, asmall display 918, and may lack apointing device 914. Themulti-tenant system 120, in contrast, may comprise multiple blade servers working together to provide the functionality described herein. - It is to be understood that the Figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for the purpose of clarity, many other elements found in a typical distributed system. Those of ordinary skill in the art may recognize that other elements and/or steps are desirable and/or required in implementing the embodiments. However, because such elements and steps are well known in the art, and because they do not facilitate a better understanding of the embodiments, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.
- Some portions of the above description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
- As used herein any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
- Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.
- As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
- In addition, use of the “a” or “an” are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.
- Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for a system and a process for displaying charts using a distortion region through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/202,183 US20220222441A1 (en) | 2021-01-13 | 2021-03-15 | Machine learning based named entity recognition for natural language processing |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202163136831P | 2021-01-13 | 2021-01-13 | |
US17/202,183 US20220222441A1 (en) | 2021-01-13 | 2021-03-15 | Machine learning based named entity recognition for natural language processing |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220222441A1 true US20220222441A1 (en) | 2022-07-14 |
Family
ID=82321935
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/202,188 Pending US20220222489A1 (en) | 2021-01-13 | 2021-03-15 | Generation of training data for machine learning based models for named entity recognition for natural language processing |
US17/202,183 Pending US20220222441A1 (en) | 2021-01-13 | 2021-03-15 | Machine learning based named entity recognition for natural language processing |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/202,188 Pending US20220222489A1 (en) | 2021-01-13 | 2021-03-15 | Generation of training data for machine learning based models for named entity recognition for natural language processing |
Country Status (1)
Country | Link |
---|---|
US (2) | US20220222489A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220043985A1 (en) * | 2020-10-14 | 2022-02-10 | Beijing Baidu Netcom Science Technology Co., Ltd. | Role labeling method, electronic device and storage medium |
CN115422939A (en) * | 2022-10-14 | 2022-12-02 | 重庆邮电大学 | Fine-grained commodity named entity identification method based on big data |
CN116029299A (en) * | 2023-03-30 | 2023-04-28 | 北京融信数联科技有限公司 | Named entity recognition method, system and storage medium based on polysemous words |
CN116646041A (en) * | 2023-07-21 | 2023-08-25 | 北京惠每云科技有限公司 | Method and system for improving matching precision of clinical test subjects based on large model |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11880659B2 (en) | 2021-01-29 | 2024-01-23 | Salesforce, Inc. | Hierarchical natural language understanding systems |
US20230069587A1 (en) * | 2021-08-31 | 2023-03-02 | Paypal, Inc. | Named entity recognition in chat dialogues for customer relationship management systems |
US20230197067A1 (en) * | 2021-12-20 | 2023-06-22 | Rovi Guides, Inc. | Methods and systems for responding to a natural language query |
WO2024044491A1 (en) * | 2022-08-22 | 2024-02-29 | Oracle International Corporation | Adaptive training data augmentation to facilitate training named entity recognition models |
US20240078431A1 (en) * | 2022-08-25 | 2024-03-07 | Nec Laboratories America, Inc. | Prompt-based sequential learning |
US11822894B1 (en) * | 2022-12-30 | 2023-11-21 | Fmr Llc | Integrating common and context-specific natural language understanding processing in a virtual assistant application |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190012302A1 (en) * | 2014-10-20 | 2019-01-10 | Google Inc. | Annotations of textual segments based on user feedback |
CN110032634A (en) * | 2019-04-17 | 2019-07-19 | 北京理工大学 | A kind of Chinese name entity recognition method based on polymorphic type depth characteristic |
US20190251169A1 (en) * | 2017-02-12 | 2019-08-15 | Seyed Ali Loghmani | Convolutional state modeling for planning natural language conversations |
US10769374B1 (en) * | 2019-04-24 | 2020-09-08 | Honghui CHEN | Answer selection method for question answering system and the system |
US20200349919A1 (en) * | 2019-04-30 | 2020-11-05 | Microsoft Technology Licensing, Llc | Using dialog context to improve language understanding |
Family Cites Families (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8594996B2 (en) * | 2007-10-17 | 2013-11-26 | Evri Inc. | NLP-based entity recognition and disambiguation |
US8078638B2 (en) * | 2008-07-09 | 2011-12-13 | Yahoo! Inc. | Operations of multi-level nested data structure |
US20110131205A1 (en) * | 2009-11-28 | 2011-06-02 | Yahoo! Inc. | System and method to identify context-dependent term importance of queries for predicting relevant search advertisements |
US9710556B2 (en) * | 2010-03-01 | 2017-07-18 | Vcvc Iii Llc | Content recommendation based on collections of entities |
US9009134B2 (en) * | 2010-03-16 | 2015-04-14 | Microsoft Technology Licensing, Llc | Named entity recognition in query |
TW201310959A (en) * | 2011-08-31 | 2013-03-01 | Ibm | Method and computer system for dynamically providing algorithm-based password/challenge authentication |
US20140278362A1 (en) * | 2013-03-15 | 2014-09-18 | International Business Machines Corporation | Entity Recognition in Natural Language Processing Systems |
US9116940B1 (en) * | 2013-04-15 | 2015-08-25 | Google Inc. | Searching for join candidates |
US20200067861A1 (en) * | 2014-12-09 | 2020-02-27 | ZapFraud, Inc. | Scam evaluation system |
US20190180196A1 (en) * | 2015-01-23 | 2019-06-13 | Conversica, Inc. | Systems and methods for generating and updating machine hybrid deep learning models |
US11663409B2 (en) * | 2015-01-23 | 2023-05-30 | Conversica, Inc. | Systems and methods for training machine learning models using active learning |
US10216715B2 (en) * | 2015-08-03 | 2019-02-26 | Blackboiler Llc | Method and system for suggesting revisions to an electronic document |
AU2018214675B2 (en) * | 2017-02-06 | 2022-08-04 | Thomson Reuters Enterprise Centre Gmbh | Systems and methods for automatic semantic token tagging |
US9946789B1 (en) * | 2017-04-28 | 2018-04-17 | Shenzhen Cestbon Technology Co. Limited | Classifying electronic messages using individualized artificial intelligence techniques |
US10678816B2 (en) * | 2017-08-23 | 2020-06-09 | Rsvp Technologies Inc. | Single-entity-single-relation question answering systems, and methods |
US20200401794A1 (en) * | 2018-02-16 | 2020-12-24 | Nippon Telegraph And Telephone Corporation | Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs |
CN110472251B (en) * | 2018-05-10 | 2023-05-30 | 腾讯科技(深圳)有限公司 | Translation model training method, sentence translation equipment and storage medium |
WO2020072758A1 (en) * | 2018-10-03 | 2020-04-09 | Camelot Uk Bidco Limited | System and methods for training and employing machine learning models for unique string generation and prediction |
US10929392B1 (en) * | 2018-11-16 | 2021-02-23 | Amazon Technologies, Inc. | Artificial intelligence system for automated generation of realistic question and answer pairs |
US11580379B1 (en) * | 2019-01-29 | 2023-02-14 | Amazon Technologies, Inc. | Phased deployment of deep-learning models to customer facing APIs |
US10607042B1 (en) * | 2019-02-12 | 2020-03-31 | Live Objects, Inc. | Dynamically trained models of named entity recognition over unstructured data |
JP7358748B2 (en) * | 2019-03-01 | 2023-10-11 | 富士通株式会社 | Learning method, extraction method, learning program, and information processing device |
US11126678B2 (en) * | 2019-03-05 | 2021-09-21 | Corinne Chantal David | Method and system to filter out harassment from incoming social media data |
US11170064B2 (en) * | 2019-03-05 | 2021-11-09 | Corinne David | Method and system to filter out unwanted content from incoming social media data |
US11157707B2 (en) * | 2019-07-23 | 2021-10-26 | International Business Machines Corporation | Natural language response improvement in machine assisted agents |
US20210064821A1 (en) * | 2019-08-27 | 2021-03-04 | Ushur, Inc. | System and method to extract customized information in natural language text |
US11709873B2 (en) * | 2020-01-13 | 2023-07-25 | Adobe Inc. | Reader-retriever approach for question answering |
US11544472B2 (en) * | 2020-01-17 | 2023-01-03 | Samsung Electronics Co., Ltd. | Structured adversarial, training for natural language machine learning tasks |
US11423897B1 (en) * | 2020-01-30 | 2022-08-23 | Amazon Technologies, Inc. | Techniques for providing adaptive responses |
US11880411B2 (en) * | 2020-02-12 | 2024-01-23 | Home Depot Product Authority, Llc | Named entity recognition in search queries |
US11606613B2 (en) * | 2020-02-27 | 2023-03-14 | Rovi Guides, Inc. | Systems and methods for generating dynamic annotations |
US11847414B2 (en) * | 2020-04-24 | 2023-12-19 | Deepmind Technologies Limited | Robustness to adversarial behavior for text classification models |
US20210365837A1 (en) * | 2020-05-19 | 2021-11-25 | Arizona Board Of Regents On Behalf Of Arizona State University | Systems and methods for social structure construction of forums using interaction coherence |
KR20210142891A (en) * | 2020-05-19 | 2021-11-26 | 삼성에스디에스 주식회사 | Method and apparatus for customizing natural language processing model |
CN112199511B (en) * | 2020-09-28 | 2022-07-08 | 西南电子技术研究所(中国电子科技集团公司第十研究所) | Cross-language multi-source vertical domain knowledge graph construction method |
US11500830B2 (en) * | 2020-10-15 | 2022-11-15 | International Business Machines Corporation | Learning-based workload resource optimization for database management systems |
US11520829B2 (en) * | 2020-10-21 | 2022-12-06 | International Business Machines Corporation | Training a question-answer dialog sytem to avoid adversarial attacks |
US11669740B2 (en) * | 2021-02-25 | 2023-06-06 | Robert Bosch Gmbh | Graph-based labeling rule augmentation for weakly supervised training of machine-learning-based named entity recognition |
US11960842B2 (en) * | 2021-02-27 | 2024-04-16 | Walmart Apollo, Llc | Methods and apparatus for natural language understanding in conversational systems using machine learning processes |
US20220277290A1 (en) * | 2021-03-01 | 2022-09-01 | Bank Of America Corporation | Electronic system for automatically generating resource distributions based on sms-based instructions using machine learning |
US11321538B1 (en) * | 2021-10-15 | 2022-05-03 | Dovel Technologies, Llc | Ensemble natural language processing model with compliance verification |
-
2021
- 2021-03-15 US US17/202,188 patent/US20220222489A1/en active Pending
- 2021-03-15 US US17/202,183 patent/US20220222441A1/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190012302A1 (en) * | 2014-10-20 | 2019-01-10 | Google Inc. | Annotations of textual segments based on user feedback |
US20190251169A1 (en) * | 2017-02-12 | 2019-08-15 | Seyed Ali Loghmani | Convolutional state modeling for planning natural language conversations |
CN110032634A (en) * | 2019-04-17 | 2019-07-19 | 北京理工大学 | A kind of Chinese name entity recognition method based on polymorphic type depth characteristic |
US10769374B1 (en) * | 2019-04-24 | 2020-09-08 | Honghui CHEN | Answer selection method for question answering system and the system |
US20200349919A1 (en) * | 2019-04-30 | 2020-11-05 | Microsoft Technology Licensing, Llc | Using dialog context to improve language understanding |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220043985A1 (en) * | 2020-10-14 | 2022-02-10 | Beijing Baidu Netcom Science Technology Co., Ltd. | Role labeling method, electronic device and storage medium |
US11907671B2 (en) * | 2020-10-14 | 2024-02-20 | Beijing Baidu Netcom Science Technology Co., Ltd. | Role labeling method, electronic device and storage medium |
CN115422939A (en) * | 2022-10-14 | 2022-12-02 | 重庆邮电大学 | Fine-grained commodity named entity identification method based on big data |
CN116029299A (en) * | 2023-03-30 | 2023-04-28 | 北京融信数联科技有限公司 | Named entity recognition method, system and storage medium based on polysemous words |
CN116646041A (en) * | 2023-07-21 | 2023-08-25 | 北京惠每云科技有限公司 | Method and system for improving matching precision of clinical test subjects based on large model |
Also Published As
Publication number | Publication date |
---|---|
US20220222489A1 (en) | 2022-07-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20220222441A1 (en) | Machine learning based named entity recognition for natural language processing | |
WO2019200923A1 (en) | Pinyin-based semantic recognition method and device and human-machine conversation system | |
US8452772B1 (en) | Methods, systems, and articles of manufacture for addressing popular topics in a socials sphere | |
US9710829B1 (en) | Methods, systems, and articles of manufacture for analyzing social media with trained intelligent systems to enhance direct marketing opportunities | |
US11258902B2 (en) | Partial automation of text chat conversations | |
Khin et al. | University chatbot using artificial intelligence markup language | |
KR102041621B1 (en) | System for providing artificial intelligence based dialogue type corpus analyze service, and building method therefor | |
US10831990B1 (en) | Debiasing textual data while preserving information | |
US20220414463A1 (en) | Automated troubleshooter | |
CN112805715A (en) | Identifying entity attribute relationships | |
US11531821B2 (en) | Intent resolution for chatbot conversations with negation and coreferences | |
US20230186033A1 (en) | Guided text generation for task-oriented dialogue | |
US20220165257A1 (en) | Neural sentence generator for virtual assistants | |
Zhang et al. | Chatbot design method using hybrid word vector expression model based on real telemarketing data | |
CN111708870A (en) | Deep neural network-based question answering method and device and storage medium | |
CN114328902A (en) | Text labeling model construction method and device | |
Zafar | Developing Smart Conversation Agent ECOM-BOT for Ecommerce Applications using Deep Learning and Pattern Matching | |
Van Thin et al. | A Human-like Interactive Chatbot Framework for Vietnamese Banking Domain | |
Kaur et al. | Design and development of a ticket booking system using Smart bot | |
WO2020026229A2 (en) | Proposition identification in natural language and usage thereof | |
US20240126991A1 (en) | Automated interaction processing systems | |
Nenciu et al. | RASA Conversational Agent in Romanian for Predefined Microworlds. | |
US20240144921A1 (en) | Domain specific neural sentence generator for multi-domain virtual assistants | |
US20220215834A1 (en) | System and method for speech to text conversion | |
US20220343068A1 (en) | Intent detection via multi-hop unified syntactic graph |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SALESFORCE.COM, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIU, JINGYUAN;SHARMA, ABHISHEK;BAROT, SUHAIL SANJIV;AND OTHERS;SIGNING DATES FROM 20210316 TO 20210323;REEL/FRAME:055699/0011 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |