KR20230146398A - Sequence text summary processing device using bart model and control method thereof - Google Patents

Abstract

The present invention relates to a sequential text summary processing device using the BART model and a control method thereof. The sequential text summary processing device according to the present invention uses a BART model including an encoder and a decoder, inputs sequential text into a topic model to generate topic modeling data, and models the generated topic. a document topic embedding calculation unit that calculates a document topic embedding value for data; and a topic attention layer disposed between an encoder of the BART model and an input layer that transmits an input value to the encoder, wherein the topic attention layer receives the document topic embedding value calculated by the document topic embedding calculation unit, Cross attention processing is performed based on the received document topic embedding value and the input value input from the input layer.

Description

Sequential text summary processing device and control method using BART model {SEQUENCE TEXT SUMMARY PROCESSING DEVICE USING BART MODEL AND CONTROL METHOD THEREOF}

The present invention relates to a sequential text summary processing device and a control method thereof, and more specifically, to a sequential text summary processing device and a control method using the BART model.

Recently, Deep Learning, which was introduced as a method of machine learning, has shown excellent results, and attempts are being made to introduce artificial intelligence in various fields.

In particular, natural language processing is a meaningful and important field in that it enables machines to understand human language.

Natural language refers to language used in daily life, and natural language processing is a general term for analyzing the meaning of natural language and allowing computers to process it.

Natural language processing can be used in a variety of fields, including voice recognition, content summarization, translation, user sentiment analysis, text classification tasks (spam email classification, news article category classification), question and answer systems, and chatbots.

Various models that perform natural language processing have been proposed, the representative one being the seq2seq (sequence-to-sequence) model using a recurrent neural network (RNN).

The seq2seq model is a model that converts one sentence (sequence) into another sentence (sequence), and is internally composed of an encoder and a decoder.

Here, the encoder encodes (encodes) the input data, and the decoder decodes (decodes) the encoded data. For example, as a specific sentence passes through the encoder, a context vector containing condensed information of the meaning of the sentence is generated. ) is created, and as these context vectors pass through the decoder again, an output sentence (for example, a translated sentence) corresponding to a specific sentence is created.

However, in this seq2seq model, the information that the encoder transmits to the decoder is a vector of fixed length. The term 'fixed length' means that no matter how long the sentence is, it is always converted into a vector of the same length, and accordingly, the necessary A problem occurred in which all information could not be completely contained in a fixed vector of limited length, that is, the necessary information was not properly included in the context vector. In other words, the longer the input sentence, the less efficiently learning takes place.

To solve this, an attention mechanism was introduced. Attention refers to the entire input sentence in the encoder once again at each time-step when the decoder predicts an output word.

At this time, rather than referring to all input sentences at the same rate, the part of the input word that is related to the word to be predicted at that point is given more weight.

Accordingly, even if the input sentence is long, the condensed meaning of the input sentence can be transmitted to the output stage, thereby solving the problem of information loss.

However, even if the attention mechanism was applied to the seq2seq model, the natural language processing ability did not reach a satisfactory level, and therefore a new model called 'Transformer' was proposed.

Transformer is a model derived from the paper “Attention is all you need” published by Google in 2017. It follows the existing seq2seq structure of encoder-decoder, but as the name of the paper suggests, it is a model implemented only with attention. .

This model showed better performance than RNN in translation performance even though this model did not use RNN and designed an encoder-decoder structure.

Various models expanded based on this Transformer model structure include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and BART (Bidirectional Auto-Regressive Transformer) models.

Here, the BERT model is a pre-trained model released by Google in 2018, and its basic structure is a structure that stacks transformer encoders.

In addition, the GPT model is a model that excludes the encoder from the transformer and utilizes only the decoder structure. In contrast to the BERT model, which has strengths in extracting the meaning of sentences, GPT is a model that has strengths in sentence generation.

In addition, the BART model has a similar form to the above-mentioned BERT and GPT combined into one, and is a model that combines the existing Sequence to Sequence transformer model by learning it through a new pre-training objective.

By using this BART model, significant progress has been made in natural language processing in various fields, but it is somewhat inadequate in terms of generating summaries for sequential text.

Here, sequential text is a text in which time-series context analysis is essential, such as human conversation or consultation. Generating a summary from such sequential text is very important in manual or automatic analysis of conversation or consultation content.

Therefore, there is a request for the development of a modified model that utilizes the BART model, which has recently shown prominence in natural language processing, while improving performance in generating summary sentences, or a device that includes such a modified model.

Registered Patent No. 10-2256007

The present invention was conceived to respond to the above-described conventional requests, and its purpose is to provide a sequential text summary processing device and a control method thereof that improve automatic summarization performance for sequential text containing content such as conversation or consultation. will be.

In order to achieve the above object, a sequential text summary processing device according to the present invention utilizes a BART model including an encoder and a decoder, and generates topic modeling data by inputting sequential text into a topic model. a document topic embedding calculation unit that calculates a document topic embedding value for the generated topic modeling data; and a topic attention layer disposed between an encoder of the BART model and an input layer that transmits an input value to the encoder, wherein the topic attention layer receives the document topic embedding value calculated by the document topic embedding calculation unit, Cross attention processing can be performed based on the received document topic embedding value and the input value input from the input layer.

In addition, in order to achieve the above object, the control method of the sequential text summary processing device according to the present invention is a method of controlling the sequential text summary processing device using a BART model including an encoder and a decoder, where the sequential text is converted into a topic model. generating topic modeling data by inputting it into , and calculating a document topic embedding value for the generated topic modeling data; Placing a topic attention layer between the encoder of the BART model and an input layer that transmits an input value to the encoder; It may include the step of transmitting the calculated document topic embedding value to the topic attention layer so that cross attention processing is performed by the topic attention layer based on the document topic embedding value and the input value input from the input layer. .

As described above, according to the present invention, an improved summary can be automatically generated for sequential data.

1 is a functional block diagram of a sequential text summary processing device according to an embodiment of the present invention;
Figures 2 to 5 are examples of data types generated at each stage in the process of processing in the document topic embedding calculation unit of Figure 1;
FIG. 6 is a detailed functional block diagram of the topic attention layer of FIG. 1.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Each embodiment according to the present invention below is only an example to aid understanding of the present invention, and the present invention is not limited to these embodiments. In particular, the present invention may be comprised of a combination of at least one of the individual components, individual functions, or individual steps included in each embodiment.

In particular, for convenience, alphabet letters such as '(a)' are included in some claims, but these alphabet letters do not specify the order of each step.

As shown in FIG. 1, the sequential text summary processing device 100 according to an embodiment of the present invention includes an encoder input unit 110, an encoder unit 120, a decoder input unit 130, a decoder, and a decoder output unit 150. , a document topic embedding calculation unit 160, and a topic attention layer 170.

The encoder input unit 110 receives sequential text, processes it into a form that can be processed by the encoder, and transmits it to the encoder.

Here, the sequential text is a text that includes conversation content according to a predetermined order, such as the content of a consultation phone call. In this embodiment, it may be the conversation content itself, or only a specific part of speech (for example, a 'noun') from such conversation content. It may be that they were extracted in order.

Furthermore, the sequential text may include the speaker's utterance point token. In other words, when two parties are having a conversation, information about when each party speaks can be included.

In particular, the encoder input unit 110 can perform embedding processing on the input sequential text.

In the field of natural language processing, embedding processing means expressing each word as a vector, and these vectors may consist of numbers with tens to thousands of dimensions.

Furthermore, the encoder input unit 110 can reflect information on the position of each word included in the sequential text and then transmit it to the encoder unit 120. Reflecting the position information of the word is called positional encoding. .

Meanwhile, the document topic embedding calculation unit 160 is configured to include a topic model, inputs sequential text into the corresponding topic model to generate topic modeling data, and generates a document topic embedding value for the generated topic modeling data. After calculating, it performs the function of transmitting to the topic attention layer 170, which will be described later.

Here, the topic model performs statistical processing to discover the abstract "topic" of a document set in the field of machine learning and natural language processing, including latent semantic analysis (LSA) and probabilistic latent semantic indexing (PLSI: It includes the Probabilistic latent semantic indexing (LDA) model and the Latent Dirichlet Allocation (LDA) model.

However, in this embodiment, the topic model is an example of a latent Dirichlet allocation model.

Since the technology itself for generating topic modeling data by applying sequential text to a latent Dirichlet allocation model is a known technology, a more detailed description will be omitted.

In other words, topic proportion data for each document (FIG. 2(a)) and word distribution data for each topic (FIG. 2(b)) can be extracted from predetermined data using a latent Dirichlet allocation model, as shown in FIG. 2. , this processing algorithm corresponds to known technology.

In calculating the document topic embedding value, the document topic embedding calculation unit 160 uses data generated by the latent Dirichlet allocation model described above (i.e., topic proportion data for each document and word distribution data for each topic) and sequential text. Based on word embedding processing data can also be considered.

Here, word embedding means processing each word in the form of a dense vector. Since this dense vector corresponds to the result of the word embedding process, it is also called an embedding vector.

Word embedding processing methods include LSA, Word2Vec, FastText, and Glove, among which FastText is used as an example in the present invention.

FastText was developed by Facebook, and the mechanism itself can be said to be an extension of Word2Vec. If Word2Vec considers words as indivisible units, FastText considers multiple words to exist within one word, and internal words . In other words, the difference is that it is learned by considering subwords.

As a specific example of calculating the document topic embedding value, the document topic embedding calculation unit 160 calculates the distribution data of words for each topic by using the topic ratio data for each document as a weight among the topic modeling data generated using the latent Dirichlet allocation model. is adjusted (for example, a value is generated by multiplying the topic proportion value for each document and the proportion value for each topic) (Figure 3), and generated using the adjusted word distribution data for each topic and FastText. The document topic embedding value can be calculated by vector multiplying the word embedding data (Figure 4).

Here, vector multiplication may correspond to a multiplication process between each matrix.

An example of the document topic embedding finally generated through this process is shown in Figure 5.

The topic attention layer 170 performs a processing function using the document topic embedding value received from the document topic embedding calculation unit 160 and the input value input from the input layer.

Specifically, the topic attention layer 170 performs cross attention processing using the document topic embedding value received from the document topic embedding calculation unit 160 and the input value input from the input layer, The results of residual connection and layer normalization (aka "Add & Norm") processing can be passed through a feed-forward neural network, and then residual connection and layer normalization processing can be performed again.

The encoder unit 120 processes input data received from the topic attention layer 170 and performs a function of generating information containing the meaning of the sequential text.

To this end, the encoder unit 120 may include at least one multi-head attention module, a residual connection and layer normalization (Add & Norm) processing module, and a feed forward module.

A plurality of such encoder units 120 may be configured with the same structure.

The decoder input unit 130 receives data and performs embedding processing and positional encoding processing, similar to the above-described encoder input unit 110, and transmits the processed data to the decoder unit 140.

The data input to the decoder input unit 130 may be a document summary in the machine learning stage, and may be control text that controls the start of decoding in the summary generation stage for the actual new sequential text.

The decoder unit 140 processes the data input from the decoder input unit 130 and the data input from the encoder together to generate summary data corresponding to sequential text.

To this end, the decoder unit 140 includes a Masked Multi-Head Attention module, a Multi-Head Attention module, a residual connection and layer normalization (Add & Norm) processing module, and a feed forward. At least one (Feed Forward) module may be included.

A plurality of such decoder units 140 may be configured with the same structure.

The decoder output unit 150 performs the function of processing the data generated by the decoder unit 140 so that it becomes data in a desired form or range.

For example, the decoder output unit 150 may include a dance layer or a linear layer, and the decoder output unit 150 normalizes all input values to values between 0 and 1 as output, and uses softmax to output them. Activation function functions such as (Softmax) can also be performed.

As described above, the sequential text summary processing device 100 according to an embodiment of the present invention includes various functional units, including an encoder input unit 110, an encoder unit 120, a decoder input unit 130, and a decoder unit. (140), the structure of the decoder output unit 150 and its unique function are the same as the conventional BART model.

However, the present invention differs from the prior art in that a document topic embedding calculation unit 160 is additionally configured, and a topic attention layer 170 is disposed between the encoder input unit 110 and the encoder unit 120. will be.

That is, conventionally, there was no separate function for processing cross attention in the area of the encoder unit 120, but in the present invention, after additionally arranging the topic attention layer 170, the document topic embedding calculation unit 160 The difference is that cross attention processing is performed on the input document topic embedding value and the input value input from the input layer.

The specific structure of this topic attention layer 170 is as shown in FIG. 6.

As shown in the figure, the topic attention layer 170 includes, from the bottom, a cross attention processing unit 171, a residual concatenation and layer normalization processing unit 172, a feed forward processing unit 173, and a residual concatenation and layer normalization processing unit 174. It may be arranged sequentially.

Here, the residual concatenation and layer normalization processing unit and the feed forward processing unit perform the same functions as those included in the encoder unit 120 and the decoder unit 140, and the cross attention processing unit can perform processing according to the following equation.

Here, Query is the embedding data received from the input unit, Key is the document topic embedding value (token embedding value) input from the document topic embedding calculation unit, Value is the document topic embedding value input from the document topic embedding calculation unit, is the dimension of token and document topic embedding.

As a result of evaluating the processing results according to the present invention and the results according to the conventional BART model through the Rouge score, which is a performance indicator of a representative summary model, the Rouge score according to the present invention is 10% to 15% higher than the conventional one. was able to confirm.

The experiment was conducted on 12,000 customer center interactive data, and the summaries of correct answers were labeled directly by the agent.

Meanwhile, of course, the process of performing each of the above-described embodiments can be performed by a program or application stored in a predetermined recording medium (eg, computer-readable). Here, recording media include electronic recording media such as RAM (Random Access Memory), magnetic recording media such as hard disks, and optical recording media such as CDs (Compact Disk).

At this time, the program stored in the recording medium can be executed on hardware such as a computer or smartphone to perform each of the above-described embodiments. In particular, at least one of the functional blocks of the sequential text summary processing device according to the present invention described above may be implemented by such a program or application.

In addition, the present invention is not limited to the specific embodiments described above, but can be implemented with various changes and modifications without departing from the gist of the present invention. It will be apparent that such changes and modifications are included in the present invention if they fall within the scope of the appended claims.

100: Sequential text summary processing device 110: Encoder input unit
120: encoder unit 130: decoder input unit
140: decoder unit 150: decoder output unit
160: Document topic embedding calculation unit 170: Topic attention layer

Claims (16)

In a sequential text summary processing device using a BART model including an encoder and a decoder,
a document topic embedding calculation unit that inputs sequential text into a topic model to generate topic modeling data, and calculates a document topic embedding value for the generated topic modeling data;
It includes a topic attention layer disposed between the encoder of the BART model and an input layer that transmits an input value to the encoder,
The topic attention layer receives the document topic embedding value calculated from the document topic embedding calculation unit, and performs cross attention processing based on the received document topic embedding value and the input value input from the input layer. A sequential text summary processing device. According to paragraph 1,
A sequential text summary processing device, wherein the topic model is a Latent Dirichlet Allocation (LDA) model. According to paragraph 2,
The document topic embedding calculation unit adjusts the distribution data of words for each topic using the topic proportion data for each document as a weight among the topic modeling data generated using the latent Dirichlet allocation model, and performs word embedding based on the sequential text. and calculating a document topic embedding value by vector multiplying the adjusted word distribution data for each topic and the generated word embedding data. According to paragraph 3,
A sequential text summary processing device, characterized in that the word embedding processing uses a Fasttext method. According to paragraph 1,
The sequential text is extracted from the conversation text based on a specific part of speech, and includes a speaker's utterance point token. According to paragraph 1,
The topic attention layer performs cross attention processing using the document topic embedding value and the input value input from the input layer, and then passes the result of residual concatenation and layer normalization processing through a feed forward neural network and then performs cross attention processing again. A sequential text summary processing device characterized in that it performs residual concatenation and layer normalization processing. According to clause 6,
A sequential text summary processing device, characterized in that the cross attention processing is performed using the following formula.

Here, Query is the embedding data received from the input unit,
Key is the document topic embedding value (token embedding value) input from the document topic embedding calculation unit,
Value is the document topic embedding value input from the document topic embedding calculation unit,
is the dimension of token and document topic embeddings. In the control method of a sequential text summary processing device using a BART model including an encoder and a decoder,
(a) inputting sequential text into a topic model to generate topic modeling data, and calculating a document topic embedding value for the generated topic modeling data;
(b) placing a topic attention layer between the encoder of the BART model and an input layer that transmits an input value to the encoder;
(c) The document topic embedding value calculated in step (a) is transmitted to the topic attention layer, and cross attention processing is performed by the topic attention layer based on the document topic embedding value and the input value input from the input layer. A control method of a sequential text summary processing device, comprising the step of allowing According to clause 8,
A control method for a sequential text summary processing device, wherein the topic model is a Latent Dirichlet Allocation (LDA) model. According to clause 9,
In step (a),
(a1) adjusting the distribution data of words for each topic using topic proportion data for each document as a weight among the topic modeling data generated using the latent Dirichlet allocation model;
(a2) performing word embedding based on the sequential text;
(a3) calculating a document topic embedding value by vector multiplying the word distribution data for each topic adjusted in step (a1) and the word embedding data generated in step (a2). Control method of sequential text summary processing device. According to clause 10,
A control method of a sequential text summary processing device, characterized in that in step (a2), word embedding is performed using the Fasttext method. According to clause 8,
The sequential text is extracted from the conversation text based on a specific part of speech, and includes a speaker's utterance point token. According to clause 8,
In step (c), cross attention processing is performed using the document topic embedding value and the input value from the input layer, and the result of residual concatenation and layer normalization processing is passed through a feed forward neural network. A control method for a sequential text summary processing device, characterized in that residual concatenation and layer normalization processing are performed again. According to clause 13,
A control method of a sequential text summary processing device, characterized in that the cross attention processing is processed using the following equation.

Here, Query is the embedding data received from the input unit,
Key is the document topic embedding value (token embedding value) input from the document topic embedding calculation unit,
Value is the document topic embedding value input from the document topic embedding calculation unit,
is the dimension of token and document topic embeddings. A computer-readable recording medium recording a program for executing the method of any one of claims 8 to 14. An application program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 8 to 14.