KR102605709B1 - A pyramid layered attention model for nested and overlapped named entity recognition - Google Patents

Abstract

A pyramid layered attention model for recognizing nested and overlapped named entities is disclosed. An entity name recognition method performed by an entity name recognition system according to an embodiment includes receiving text data into a deep learning model for entity name recognition; And a step of recognizing an entity name from the input text data through the deep learning model, wherein the deep learning model uses the encoding result output from the encoder as input data of an attention-based decoder for each layer of the decoder. It may be constructed to reduce the output length of .

Description

Pyramid Layered Attention Model for Nested and Overlapped Named Entity Recognition {A PYRAMID LAYERED ATTENTION MODEL FOR NESTED AND OVERLAPPED NAMED ENTITY RECOGNITION}

The explanation below is about entity name recognition technology.

Named Entity Recognition (NER) is a subtask of information extraction that aims to identify the location of named entities (NEs) in text and classify them into defined categories such as people, organizations, locations, entities, etc. Name-entity recognition is not only a tool for information extraction, but also plays an important role in various natural language processing applications such as text understanding, automatic text summarization, question answering systems, machine translation, and knowledge base construction.

The initial entity name recognition system had high accuracy, but because it required a large number of people to design the rules, a large amount of rules had to be redesigned when switching to another field. In recent years, the rapid development of deep learning has had positive effects in each field, so many entity name recognition systems have adopted deep learning models to achieve the best performance.

A method and system for solving nested and overlapped named entity recognition can be provided using a deep learning model.

An entity name recognition method performed by an entity name recognition system includes the steps of inputting text data into a deep learning model for entity name recognition; And a step of recognizing an entity name from the input text data through the deep learning model, wherein the deep learning model uses the encoding result output from the encoder as input data of an attention-based decoder for each layer of the decoder. It may be constructed to reduce the output length of .

The deep learning model includes an encoder that performs word embedding and character embedding from text data; and a decoder that obtains output data of each layer through an operation of calculating attention scores of a plurality of input data adjacent to each other by using the encoding result output through the encoder as input data.

The decoder has a plurality of attention layers configured in a pyramid shape, multiplies the encoding results output from the encoder by different matrices, generates a plurality of vectors for the encoding results at each time, and divides the information between two adjacent input data. In order to obtain the correlation, the value of the correlation is obtained through the operation of calculating the attention score by querying the two input data and performing a multiplication operation with each other's key, and using the tanh function to obtain the obtained correlation The value can be converted to a value between -1 and 1.

The decoder obtains the output of each attention layer by adding the correlation value converted to a value between -1 and 1 multiplied by the value, and the obtained output of each attention layer is transmitted to the fully connected layer. Accordingly, classification prediction can be performed through the softmax function.

The encoder is composed of an embedding layer, an encoding layer, and a concatenate layer. In the embedding layer, word embedding and character embedding are performed from text data, and in the encoding layer, An encoding output is obtained as the embedding result connecting the word embedding result and the character embedding result is passed through a Bidirectional LSTM, and in the connection layer, the obtained encoding output is connected to the output of the pre-trained language model. Accordingly, the encoding result can be output through a fully connected layer.

The entity name recognition system includes a data input unit that receives text data into a deep learning model for entity name recognition; and an entity name recognition unit that recognizes an entity name from the input text data through the deep learning model, wherein the deep learning model uses the encoding result output from the encoder as input data for an attention-based decoder to It may be built so that the output length of each layer is reduced.

By utilizing character, word, and context-level features to learn the content of the text well, and using a pyramid-shaped attention-based decoder, it is possible to not only solve flat NER, but also recognize nested or duplicate names. It can be applied.

Figure 1 is a diagram for explaining the general operation of an entity name recognition system according to an embodiment.
Figure 2 is a diagram for explaining the operation of an entity name recognition system according to an embodiment.
Figure 3 is a diagram for explaining an attention layer, according to one embodiment.
Figure 4 is a block diagram for explaining the configuration of an entity name recognition system according to an embodiment.
Figure 5 is a flowchart for explaining an entity name recognition method in an entity name recognition system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

Figure 1 is a diagram for explaining the general operation of an entity name recognition system according to an embodiment.

The entity name recognition system 100 can solve nested and overlapped named entity recognition using the deep learning model 110. The entity name recognition system 100 may receive text data as input to the deep learning model 110. The entity name recognition system 100 can recognize the entity name 102 from text data 101 through a deep learning model 110.

At this time, the deep learning model 110 may be designed as a pyramid model based on an attention mechanism. When the text embedding and encoding results are input to the decoder based on a pyramid, they are stacked in a pyramid shape from bottom to top. Since the layer based on the designed attention mechanism calculates the attention of multiple (for example, two) adjacent to each other and outputs the result, the result output from each layer becomes 1 less than the input length. The L (L is a natural number) layer of the decoder is used to recognize entity names with a length of L+1, so they do not affect each other when recognizing nested or duplicate entity names. Therefore, it is suitable for flat entity name recognition, nested entity name recognition, and duplicate entity name recognition tasks.

For reference, nested entity name recognition (nested NER), overlapped named entity recognition, and flat entity name recognition (flat NER) recognition will be described. Many entity names themselves contain other entity names, and recognizing these is called nested entity name recognition. For example, University of Washington In this span, University of Washington is the organization and Washington is the location. This kind of work cannot be viewed simply as a series of labeling operations. Because when the word Washington is classified, it is an end and a location of the organization. Recently, a lot of work has been done in computer vision to solve nested object name recognition, where the model directly predicts the boundaries and classification of objects using object detection methods.

Additionally, the task of recognizing when multiple entity names overlap is called duplicate entity name recognition. Therefore, nested entity name recognition can be viewed as a special situation of duplicate entity name recognition. However, there are many cases where multiple entity names overlap, but none of them completely includes the other entity. Therefore, methods based on boundary detection are also suitable for recognizing duplicate entity names.

The flat entity name recognition task involves monotonously adding annotations to the entity name annotations. There is no overlap or overlap between all entities. The solution to this task is relatively simple, so the task can be considered directly as a sequence labeling task. A recently popular solution is to classify each word using long short-term memory (LSTM), conditional random fields (CRF), or simple softmax.

Figure 2 is a diagram for explaining the operation of an entity name recognition system according to an embodiment.

The entity name recognition system can recognize entity names from text data through the deep learning model 110. The entity name recognition system can design a deep learning model 110 to solve the task for entity name recognition. The deep learning model 110 may be composed of an encoder 210 and a decoder 220.

The encoder 210 may be composed of three components including an embedding layer, an encoding layer, and a concatenate layer. In detail, a method of combining embeddings in the embedding layer may be adopted. Character embedding and word embedding can be combined. The combined character embedding and word embedding approach combines word-level features and character-level features. The embedding result can be passed to the encoding layer. The purpose of the encoding layer is to combine context information. The output result of the encoding layer can be combined with a pre-trained language model, such as BERT's embedding result, and passed to the decoding layer.

The decoder 220 may be designed with a pyramid-shaped attention model. The output of the L (L is a natural number) th layer is a feature of the span of length L-1, and finally, the softmax method can be used for classification.

In detail, the entity name recognition system can input text data into a deep learning model for entity name recognition. For example, text data may be sentences. At this time, the input data input to the deep learning model is a text sentence of T length. The output is L tag sequences in IOB2 format, each of length T, T-1, ??, T-L+1. In the IOB2 format, B-{class} is used at the beginning of the entity name, I-{class} is used from the second token among one or more tokens corresponding to the entity name, and O is a token that does not correspond to the entity name.

The deep learning model 110 can be divided into two parts including an encoder 210 and a decoder 220. Encoding may be performed through the encoder 210 configured in the deep learning model 110. As encoding is performed, the obtained embedding sequence can be recursively supplied to the decoder (decoding layer) for entity name recognition. Each layer of the decoder 220 is a sequential labeling of features of length T-L.

First, sentences containing T tokens are It can be displayed as .

The encoder 210 may use Bidirectional LSTM for character embedding. For word embedding, the GloVe method can be used. The results of character embedding and word embedding can be combined and transmitted to the encoding layer. In the encoding layer, context information can be combined using Bidirectional LSTM. At the end of the encoder, the output of the encoding layer and the BERT embedding result are combined and used as the output of the encoder through a fully connected layer. Therefore, the input data The encoder formula for can be expressed as follows.

More specifically, word embeddings can be initialized with pre-trained word vectors. Here, a pre-trained GloVe can be selected. Words without GloVe can be initialized randomly. This problem is called out-of-vocabulary (OOV). To express it in a formula, This happens.

Additionally, character embeddings can be dynamically generated using Bidirectional LSTM, and weights can be updated during training. Introducing character embeddings allows the model to better handle OOV words. You can separate words into letters and perform embedding on the letters using embedding. Finally, the result of concatenating the last hidden state of the forward LSTM and the first hidden state of the reverse LSTM can be used as a word vector. To express it in a formula, , This happens.

Additionally, the encoding layer can concatenate the word embedding result and the character embedding result and transfer them to Bidirectional LSTM. This is to further utilize contextual information. To express it in a formula, , This happens.

Additionally, pre-trained language models can be used to utilize better context information. Here, BERT can be used. To express it in a formula, This happens.

The concatenate layer connects the output of the encoding layer and the output of the pre-trained language model, and then outputs the resulting data through the fully concatenated layer. To express it in a formula, This happens.

Referring to FIG. 3, it is a diagram to explain the attention layer 230. The decoder 220 may be composed of a pairwise attention layer. The decoder can obtain three vectors q, k, and v by multiplying each input by three different matrices. In other words, decoder 220 can multiply the output of the encoder portion by three different matrices to generate three vectors for the output at each point in time. The input data of the decoder is am. Here, q represents query, k represents key, and v represents value. The calculation method for vectors q, k, and v is as follows.

here, is the i-th input data of the L-th attention layer of the decoder. Is These are the key, query, and value. And d _k =d _q .

are the parameters to be learned, and can be shared between the attention layers of the decoder.

Then, in order to obtain a correlation between two adjacent input data, a query for the two input data can be multiplied with the other's key. At this time, the higher the value, the stronger the relationship between the two points. Then, the tanh function is used to scale the correlation value between [-1, 1]. The formula for calculating the attention score is as follows.

is the input data is a query of, is the input data It is the transpose of the key of . is the input data input data for Indicates the attention score of .

The output of the attention layer 230 configured in the decoder 220 can be obtained by multiplying the attention scores of two adjacent input data by the corresponding values and adding them. The calculation method is as follows:

here, is the i-th output of the L-th attention layer of the decoder, and is the i-th input of the L+1-th attention layer.

Each time the attention layer is output to the decoder, the output length is reduced by 1. And, the output of attention layer L is equivalent to setting a sliding window for text, and the size of the window is L+1.

Finally, classification tasks can be performed on each attention layer using fully connected layers and the softmax method. The output of each attention layer is sent to the fully connected layer, and the softmax function can be used to perform classification prediction. The formula is as follows:

here, is the prediction result of the ith output of the Lth attention layer of the decoder. d _class is the total number of tag classes.

FIG. 4 is a block diagram for explaining the configuration of an entity name recognition system according to an embodiment, and FIG. 5 is a flowchart for explaining an entity name recognition method in an entity name recognition system according to an embodiment.

The processor of the entity name recognition system 100 may include a data input unit 410 and an entity name recognition unit 320. These processor components may be expressions of different functions performed by the processor according to control instructions provided by program codes stored in the entity name recognition system. The processor and its components may control the entity recognition system to perform steps 510 to 520 included in the entity name recognition method of FIG. 5 . At this time, the processor and its components may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the entity name recognition method into memory. For example, when a program is executed in the entity name recognition system, the processor can control the entity name recognition system to load program code from the program file into memory under the control of the operating system. At this time, the processor, the data input unit 410 and the entity name recognition unit 420 each execute the instructions of the corresponding part of the program code loaded in the memory to execute the subsequent steps 510 to 520. These can be functional expressions.

In step 510, the data input unit 410 may receive text data as input into a deep learning model for entity name recognition. The data input unit 410 can receive text data in the form of sentences into a deep learning model for entity name recognition. At this time, the text data may include at least one sentence. Additionally, text data may include one language, such as Korean or English, or a mixture of several languages.

In step 520, the entity name recognition unit 420 may recognize the entity name from text data input through a deep learning model. Through this, entity name recognition suitable for flat entity name recognition, nested entity name recognition, and duplicate entity name recognition tasks is possible.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (6)

In the entity name recognition method performed by the entity name recognition system,
Inputting text data into a deep learning model for entity name recognition; and
Recognizing an entity name from the input text data through the deep learning model
Including,
The deep learning model is constructed to reduce the output length of each layer of the decoder by using the encoding result output from the encoder as input data of an attention-based decoder,
An encoder that outputs an encoding result by combining context information combined through word embedding and character embedding from text data and the output of a pre-trained language model; and
A decoder that obtains output data of each layer through an operation of calculating attention scores of a plurality of input data adjacent to each other by using the encoding result output through the encoder as input data,
The decoder is,
The encoding result output from the encoder is multiplied by different matrices to generate a plurality of vectors for the encoding result at each point in time, and a query for the two input data is respectively performed to obtain the correlation between the two adjacent input data. Obtaining the value of the correlation through the operation of calculating an attention score that performs a multiplication operation with the other key of , and converting the obtained correlation value to a value of -1 to 1 using the tanh function.
An entity name recognition method characterized by: delete delete According to paragraph 1,
The decoder is,
The output of each attention layer is obtained by adding the correlation value converted to a value between 1 and 1 multiplied by the value, and as the output of each attention layer obtained is transmitted to the fully connected layer, softmax is obtained. Performing classification prediction through functions
An entity name recognition method characterized by: According to paragraph 1,
The encoder consists of an embedding layer, an encoding layer, and a concatenate layer,
In the embedding layer, word embedding and character embedding are performed from text data,
In the encoding layer, an encoding output is obtained as the embedding result connecting the word embedding result and the character embedding result is passed through a Bidirectional LSTM,
In the connection layer, the obtained encoding output and the output of the pre-trained language model are connected, and the encoding result is output through a fully connected layer.
An entity name recognition method characterized by: In the entity name recognition system,
A data input unit that receives text data into a deep learning model for entity name recognition; and
Entity name recognition unit that recognizes the entity name from the input text data through the deep learning model
Including,
The deep learning model is constructed to reduce the output length of each layer of the decoder by using the encoding result output from the encoder as input data of an attention-based decoder,
An encoder that outputs an encoding result by combining context information combined through word embedding and character embedding from text data and the output of a pre-trained language model; and
A decoder that obtains output data of each layer through an operation of calculating attention scores of a plurality of input data adjacent to each other by using the encoding result output through the encoder as input data,
The decoder is,
The encoding result output from the encoder is multiplied by different matrices to generate a plurality of vectors for the encoding result at each time point, and a query is made on the two input data to obtain the correlation between the two adjacent input data. Obtaining the value of the correlation through the operation of calculating an attention score that performs a multiplication operation with the other key of , and converting the obtained correlation value to a value of -1 to 1 using the tanh function.
An entity name recognition system characterized by:

Images