KR20240076977A - Device and method for extracting dialogue-based relation using cloze-style masking with prompts for entity types and relational information - Google Patents

Abstract

A method for generating an interactive relationship extraction model using prompts and blank reasoning for entity type and relationship information is disclosed. The interactive relationship extraction model generation method is performed by a computing device including at least a processor, and includes examples including a conversation, a subject, and an object, each including utterances by a plurality of speakers. It includes receiving, generating input for each of the examples, and learning a pre-trained language model (PLM) using the input.

Description

Dialog relationship extraction method and device using prompts and blank inference for entity type and relationship information {DEVICE AND METHOD FOR EXTRACTING DIALOGUE-BASED RELATION USING CLOZE-STYLE MASKING WITH PROMPTS FOR ENTITY TYPES AND RELATIONAL INFORMATION}

The present invention relates to a conversation-level relationship extraction method, and more specifically, by applying soft prompt and close-style blank reasoning methods for entity type and relationship information to a pre-trained language model to It relates to a method and device for extracting relationships between entities mentioned in.

Dialogue-level relationship extraction covered by the present invention is a different task from sentence, multi-sentence, and document relationship extraction. Conversation-level relationship extraction is the task of extracting all relationships between speakers and entities mentioned in the conversation in a text in which multiple speakers speak freely in colloquial language. Since this task deals with relationships that may arise in everyday conversations rather than with knowledge of common sense inference, the relationship between the speaker and the characters in the conversation must be inferred through context.

In contrast, relationship extraction within general sentences and/or documents not only deals with written language, but also allows relationships to be extracted through common sense reasoning rather than context understanding. For example, in a sentence such as “Samsung is a Korean company,” the relationship between “Samsung” and “Korea” can be sufficiently inferred through external knowledge and common sense inference without understanding the context. On the other hand, conversation-level relationship extraction tasks require us to infer knowledge beyond our common sense through context.

The technology applied in the present invention to extract conversation-level relationships is a technology that generates prompts about entity types and relationship information as virtual tokens and inputs them into the system along with the conversation, as well as a blank reasoning method (Masked Language Modeling, MLM) and dictionary learning. This is a relationship extraction learning methodology using a language model. Prompt technology is a methodology that inputs virtual tokens or actual natural language that are assumed to be helpful for the task into the system along with the task to be solved. Prompt technology is a methodology that allows you to easily instruct a model to perform tasks without requiring a task-specific layer or additional learning model. Prompts for entity type and relationship information applied in the present invention are provided in the form of virtual tokens along with the input conversation, and can act as a kind of hint when the model extracts relationships in the conversation.

Encoder-based pre-trained language models typically learn the context on large corpora using the blank inference method (MLM). In detail, for blank inference learning, [MASK] tokens are generally used, some of the input tokens are replaced with [MASK] tokens, and the model is pre-trained by restoring the replaced tokens. In general, in order to modify this pre-trained model to fit the desired task, additional neural network layers are added and fine-tuning is performed on a small task-specific corpus.

However, a methodology that has recently emerged with prompt technology models the task by providing the prompt virtual token, the task token to be solved, and the [MASK] token as input to the model, and then matching the tokens related to the task. This methodology is effective because the blank reasoning method used in pre-training is applied to subtasks almost as is, so the neural network does not need to adapt to new tasks. The present invention proposes a methodology for modeling conversation-level relationship extraction without an additional neural network layer using this blank inference learning methodology.

Republic of Korea Patent Publication No. 2019-0038258 (published on April 8, 2019) Republic of Korea Patent Publication No. 2022-0083414 (published on June 20, 2022) Republic of Korea Patent Publication No. 2021-0114256 (published on September 23, 2021)

Yu, Dian, et al. “Dialogue-Based Relation Extraction.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. Gao, Tianyu, Adam Fisch, and Danqi Chen. “Making Pre-trained Language Models Better Few-shot Learners.” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021. Chen, Xiang, et al. “Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction.” Proceedings of the ACM Web Conference 2022. 2022. Han, Xu, et al. "Ptr: Prompt tuning with rules for text classification." arXiv preprint arXiv:2105.11259 (2021).

Although the prior art utilizes a pre-learning language model, which is a recent technology, it is difficult to find an inventive technology that applies prompts or uses blank reasoning methodology (MLM) to perform relational inference. Additionally, although there are many technologies for general sentence-level, multi-sentence, and document-level relationship extraction, there are few technologies that deal with conversation-level relationship extraction.

In the present invention, we only change the method of using the language model without complex modeling by applying the blank reasoning methodology (MLM), which allows more effective use of pre-learning language models in conversation-level relationship extraction tasks, and the prompt methodology that provides hints for relationship inference. Efficient conversation relationship extraction can be performed.

The method of generating an interactive relationship extraction model according to an embodiment of the present invention is performed by a computing device including at least a processor, and generates a conversation, a subject, and an object, each including utterances by a plurality of speakers. It includes receiving examples containing an object, generating an input for each of the examples, and learning a pre-trained language model (PLM) using the input.

Through the methodology of the present invention, which provides a pre-learning language model with input dialogue and prompts for entity type and relationship information, and predicts relationship words using a blank inference method, relationship inference is possible using only the language model without an additional neural network layer. .

Methodologies that utilize prior distributions to define prompts for entity types and relationship information are useful information to help the model infer relationships.

By reusing the empty space inference method (MLM) that the pre-training language model uses for pre-training when inferring relationships, the model can adapt to the relationship extraction task more efficiently.

The elements considered in the present invention can take into account difficulties in conversation that cannot be considered in general sentences and documents. For example, in order to solve the characteristics of conversations that are long and have low information density, the present invention predicts key clues of relationships called “triggers.”

In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.
Figure 1 shows (1) a standard tweaking approach and (2) a prompt-based tweaking approach.
Figure 2 is a schematic diagram of a relationship extraction model generation method or relationship extraction method according to an embodiment of the present invention.
Figure 3 is a flowchart illustrating a method for generating an interactive relationship extraction model according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining the embodiments according to the concept of the present invention. They may be implemented in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can make various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component and similarly a second component The component may also be named a first component.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to that other component, but that other components may also exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly adjacent to" should be interpreted similarly.

The terms used in this specification are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in this specification, but are not intended to indicate the presence of one or more other features. It should be understood that it does not exclude in advance the existence or possibility of addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification. However, the scope of the patent application is not limited or limited by these examples. The same reference numerals in each drawing indicate the same members.

The dialogue-based relation extraction (DialogRE) task aims to predict relationships between argument pairs (which can be named topic pairs) that appear in a conversation. Most previous studies use fine-tuning of pre-trained language models (PLMs). In order to efficiently utilize the inherent knowledge of PLMs without introducing additional layers and to consider scattered semantic clues about relationships between arguments, we propose a technique called GRASP (Guiding model with RelAtional Semantics using Prompt). In the proposed invention, we adopt a prompt-based fine-tuning approach and capture semantic clues about relationships in a given conversation. Specifically, it includes 1) an argument-recognition prompt marker strategy and 2) a relational cue detection task. According to experiments, GRASP recorded very high F1 and F1c scores on the DialogRE dataset, despite using only PLMs without any additional layers.

Relationship extraction (RE) tasks aim to extract semantic relationships from unstructured text such as sentences, documents, or conversations. Because RE can extract structured relationship information, it plays an important role in information extraction and knowledge base construction. However, the use of sentence-level RE is limited because many relational facts are widely distributed across multiple sentences by one or more speakers within a conversation. Accordingly, the DialogRE task, which includes argument pairs and their corresponding relationships, can be understood as creating a model that captures the underlined meanings scattered within the dialogue, as shown in Table 1.

[Table 1]

Table 1 shows examples of DialogRE data. Arguments are shown in bold, and triggers are underlined. Arguments and triggers are scattered throughout the entire dialog. In other words, the information density is low. Triggers indirectly determine the correct direction of the relationship.

To efficiently utilize knowledge from PLMs, several studies involving prompt-based fine-tuning have been conducted. To bridge the gap between pre-training and fine-tuning, they used PLM directly as a predictor and completed a close task. As shown in Figure 1, prompt-based tweaking treats the subtask as a masked language modeling (MLM) problem by directly generating a textual response to a given template. In particular, prompt-based tweaking updates the original input based on a template and predicts label words with the [MASK] token. In the prompt-based fine-tuning approach of Figure 1, a set of label words is mapped to a class set by a specific mapping function. Ultimately, the model maps the predicted label words to a set of corresponding task-specific classes.

However, the prompt-based fine-tuning approach does not show sufficient performance compared to the fine-tuning-based approach. In the present invention, this was determined to be due to 1) the high frequency of person-pronouns and 2) the low information density by multiple speakers. Therefore, a prompt-based tweaking approach needs to understand the relationships between arguments.

In the present invention, we propose a new method for DialogRE called GRASP (Guiding model with RelAtional Semantics using Prompt). To maximize the benefits of the prompt-based fine-tuning approach, we propose an argument-aware prompt marking (APM) strategy and a relational clue detection (RCD) task. The APM strategy guides the model to important arguments scattered within the dialog by carefully considering arguments. In addition to the APM strategy, the proposed RCD task forces the model to consider important relationship clues. In particular, the model is trained to determine whether each token in a conversation belongs to a subject, object, or trigger.

Figure 2 is a schematic diagram of a relationship extraction model generation method or relationship extraction method according to an embodiment of the present invention.

First, input with a prompt template is generated using the APM strategy. The Pre-trained Language Model (PLM) then receives the generated input for prompt-based fine-tuning and uses the PLM's contextualized representations to determine the probability distribution of the model's vocabulary. Calculate (or estimate) probability distributions. The RCD task forces the model to predict the relational clue type of each token, and in the Masked Language Modeling (MLM) task, the model takes the [MASK] representation to predict the final relationship. . In other words, the proposed model is learned through multi-task learning that reads mutual communication between relationship clues for argument pairs and the final relationship.

Problem Formulation

Each example is a dialogue , subject a ₁ , and object a ₂ . is {s ₁ : u ₁ , s ₂ : u ₂ , … , s _N : u _N }. Here, s _n is the nth speaker (or speaker) and u _n is the corresponding utterance. In other words, s _n refers to the speaker of the nth utterance u _n . also, may include information about at least one trigger. Regarding, the purpose of dialogue-based relation extraction (DialogRE) is Relationship between arguments (a ₁ and a ₂ , meaning the relationship extraction target) using is to predict. To illustrate DialogRE with respect to prompt-based fine-tuning, each example Template function that maps to is defined. [MASK] is is inserted into the relationship It is used to predict label words. is the same as Equation 1.

[Equation 1]

Based on the structure of, the present invention uses a template function by applying two steps. Build. Both steps have an argument-aware prompt marker. cast It includes a conversion step and a prompt initialization step for [subj] and [obj]. A detailed description of these two steps will be provided later. Additionally, the present invention proposes a relationship clue extraction technique.

Argument-aware Prompt Marker

The present invention proposes an APM strategy that considers both speakers and non-speakers. In previous studies, dialogue is encoded by focusing on speaker information, without focusing on non-speaker arguments. However, in the DialogRE dataset, approximately 77.4% of relation triples contain at least one non-speaker argument, suggesting that consideration of non-speaker arguments is essential to improve the model's argument recognition.

In the present invention, the argument-recognition prompt marker token [p] is inserted as an entity marker. Argument-aware prompt markers allow the model to obtain information to determine which tokens are elements for relationship prediction. In the present invention, the feature of [p] is initialized by embedding a space token in the model dictionary. According to experiments, blank tokens can detect the start position of arguments. Therefore, the proposed prompt marker allows us to determine which parts of the conversation play an important role in predicting relationships.

Using constraining argument-aware prompt markers, BERTs (Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-based relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927 -4940, Online. Association for Computational Linguistics.)'s token replacement method has been strengthened. example Regarding BERTs, Build (or create) . here, ego, is the same as Equation 2.

[Equation 2]

[S ₁ ] and [S ₂ ] are special tokens for speakers. [S ₁ ] and [S ₂ ] are virtual tokens for speaker replacement. also, Is, If satisfies, it is defined as [S _m ], otherwise it is defined as a _m .

BERTs prevent models from overfitting and generally show high performance, but BERTs do not take non-speaker arguments into account. To explore arguments that were not considered, the APM strategy extends BERTs by considering both types of arguments in the DialogRE task. In the present invention, a function APM(·) is defined that encodes an utterance by inserting a virtual token [p] in front of each argument token. Conversations Built by BERTs Regarding, in the present invention By applying APM(u _n ) to each utterance within creates . Therefore, the APM strategy is based on creates . This can be expressed as Equation 3.

[Equation 3]

For example, APM(·) encodes the text "I am Tom Gordon" as [I, am, [p], Tom, Gordon] by inserting [p] in front of "Tom Gordon". Input of the pre-trained language model through the operations described above. This can be created.

PromptConstruction

In the present invention, constructed (or generated) input Updates with a template function and performs initialization. The present invention uses a prior distribution of argument types for initialization. Prompt tokens [subj] and [obj] are used to inject argument-type information. In the present invention, an argument type set (may also be named a type set of the entity under consideration) is created using predefined types in the dataset as shown in Table 1. Define. Here, “PER” means a person, “ORG” means an organization, “GPE” means a place, “VALUE” means numerical information, and “STRING” means other types. In the present invention, by using frequency statistics, Distribution of argument types for and Calculate . In other words, the prior distribution is calculated from the learning corpus for the defined type. To initialize the prompt tokens [subj] and [obj], each argument type Combine with the corresponding prior distribution. An exemplary initialization formula is shown in Equation 4.

[Equation 4]

In equation 4, e(·) is the embedding from PLM for the input token, is the initialized embedding of the prompt token (in other words, it can refer to the embedding initialization function of the pretrained language model). According to Equation 4, the embeddings of [subj] and [obj] are calculated using the prior distribution for the entity types of the subject and object and initialized in the embedding layer of the language model. Illustratively, the subject has a prior distribution Let's assume we have . [subj] The initial embedding of the token is a weighted average, i.e. It can be calculated as

thus, cast convert to can be formalized. is an example of argument-enhanced input. in other words, am. Then, the final input structure (final token sequence) for prompt-based tweaking is as shown in Equation 5.

[Equation 5]

Additionally, virtual relationship tokens are created for learning blank inference. Specifically, for relationship prediction by application of MLM, As shown in Figure 2, is defined as a set of label words in the model's dictionary, that is, a set of labeled words. Specifically, metadata for initializing each relationship expression can be used to inject meaning.

For example, a special token in the model's dictionary You can initialize these tokens by adding as a label word and clustering the word's embedding in the metadata. That is, class for . In other words, in the case of a relationship where "person:date_of_birth" is the correct answer, the present invention creates a virtual token [per:date_of_birth] and maps it to the same unit as the target of the [MASK] token. In order to give relational information to this virtual token, the embedding of [per:date_of_birth] calculates the average value of the embeddings of “person”, “date”, “of”, and “birth”, Initialize using a function. This operation is performed on all relationships existing in the corpus to generate and initialize the final 36 virtual relationship tokens.

Relational Clue Detection (RCD) task

RCD predicts whether each input token is a subject ([subject]), an object ([object]), a trigger ([trigger], a relationship clue), or another token ([outside]). That is, by introducing a prompt-based fine-tuning approach, we propose the RCD task to improve understanding of relational clues. A set of label words Define ={[subject],[object],[trigger],[outside]} and add it to the model dictionary. then, Each token within A sequence of label words for an RCD by assigning to the corresponding cue type word from, can be built. For example, if the token sequence is When given as , go It is built as follows. here, is the subject, is an object argument, is a trigger.

The RCD task uses the MLM head, which is used to predict [MASK] tokens. Except for the [MASK] token, each token is sequentially processed using the metadata provided in the dataset. It is tagged with In other words, the RCD task allows the model to identify which non-[MASK] tokens are associated with a particular relational cue type. RCD helps the model gather scattered information from the entire conversation by indicating where to focus within the conversation to predict relationships. In this respect, the model pays considerable attention to semantic clues of relationships, such as triggers. Additionally, the proposed model has low complexity by performing the RCD task without an additional classifier. each token The loss of the RCD task for is given in Equation 6.

[Equation 6]

Model Training

Before predicting the final relationship, based on the results of the RCD task, the position of the trigger is marked using the [p] token, i.e. the argument-aware prompt marker. example For all non-[MASK] tokens, the model Predict the label words of Additionally, [p] is inserted in front of the word predicted as a trigger to learn the model to identify key clues in determining the relationship.

Multitask Learning

is a set of classes A set of label words It is a mapping function that converts to (in other words, a function that maps the relational token corresponding to the [MASK] token). each input For , the goal of MLM is to fill [MASK] with relationship label words in the model's dictionary. Because the model predicts the correct label word at the position of [MASK], can be formalized. The learning objective of relationship prediction is to minimize Equation 7. In other words, the objective function in Equation 7 is It facilitates matching the virtual relationship token corresponding to the [MASK] token included in .

[Equation 7]

To improve the model's ability to capture relationship clues through the interaction between the MLM task and the RCD task for relationship prediction, the GRASP model was developed by using the joint loss expressed in Equations 6 and 7. Multi-task learning can be introduced. Therefore, the final learning goal is to minimize the combination loss in Equation 8. here class are hyperparameters.

[Equation 8]

Figure 3 is a flowchart illustrating a method for generating a relationship extraction model according to an embodiment of the present invention. The relationship extraction model creation method, which may also be named a relationship extraction method or the like, may be performed by a computing device that includes at least a processor and/or memory. Accordingly, at least some of the steps configuring the relationship extraction model generation method may be understood as operations of a processor included in the computing device. Computing devices may include personal computers (PCs), servers, tablet PCs, smartphones, head mounted devices (HMDs), navigation systems, smart glasses, smart watches, etc.

First, examples constituting the learning corpus are received (S110). Each of the examples includes a dialog, a subject, and an object. Additionally, each of the examples may include at least one trigger.

Next, input corresponding to each of the examples may be generated (S120). The input can be created using a template function, and since the specific input creation process is the same as the above-described process, the detailed description will be omitted.

Finally, a pre-trained language model can be learned using the generated input (S130). Here, the first loss ( ) and second loss ( ), which is the weighted sum of the final loss ( ) Learning can proceed in the direction of minimizing ).

In addition, before learning the pre-training language model using the learning corpus, operations such as initializing embeddings such as [subj] and [obj] and/or generating and initializing (virtual) relationship tokens can be performed. there is.

The device described above may be implemented as a hardware component, a software component, and/or a set of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an Arithmetic Logic Unit (ALU), a Digital Signal Processor, a microcomputer, a Field Programmable Array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a Programmable Logic Unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device may include multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes specially configured hardware devices to store and execute program instructions, such as magneto-optical media, ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

Claims (7)

At least, in a method of generating an interactive relationship extraction model performed by a computing device including a processor,
Receiving examples including a conversation, a subject, and an object, each including utterances by a plurality of speakers;
generating input corresponding to each of the examples; and
An interactive relationship extraction model generation method including the step of learning a pre-trained language model (PLM) using the input. According to paragraph 1,
Each of the examples includes a trigger for the relationship between the subject and the object,
How to create an interactive relationship extraction model. According to paragraph 2,
The step of generating the input is,
Inserting an entity marker in front of the subject and the object in the conversation,
How to create an interactive relationship extraction model. According to paragraph 3,
The step of generating the input is,
replacing a speaker identical to the subject in the conversation with a first special token for the speaker; and
Comprising replacing the same speaker as the object in the conversation with a second special token for the speaker,
How to create an interactive relationship extraction model. According to clause 4,
The learning step is the first loss ( ) and second loss ( ), which is the weighted sum of the final loss ( ), learn the PLM to minimize
The final loss is ego,
remind class is a predefined hyperparameter,
How to create an interactive relationship extraction model. According to clause 5,
The first loss is ego,
remind represents the input,
remind is a sequence of label words,
How to create an interactive relationship extraction model. According to clause 6,
The second loss is ego,
remind is a set of classes ( ) into a set of label words ( ), a mapping function that converts to
How to create an interactive relationship extraction model.

Images