KR20230014034A - Improving classification accuracy using further pre-training method and device with selective masking - Google Patents

Abstract

The present invention relates to a method and device for the further pre-training based on selective masking for improving classification accuracy. The method comprises: a step of performing the emotional classification on training data of a training data population for pre-training; a step of generating an effective word corpus on the words of the training data population in accordance with the result of the emotional classification; a step of applying the effective word corpus to a tokenizer built through the pre-training, and building an extended tokenizer; a step of masking the training data by using the extended tokenizer; and a step of performing a further pre-training by using the masked training data.

Description

Optional masking-based additional pre-learning method and device for improving classification accuracy

The present invention relates to an additional pre-learning technique, and more particularly, to a selective masking-based additional pre-learning method and apparatus for improving classification accuracy capable of performing additional pre-learning specialized for a specific classification task through selective masking.

Recent breakthroughs in natural language processing (NLP) and text mining (Text Mining) technologies have led to attempts to apply text analysis techniques to domain problem solving in various fields. Natural language processing is a process of converting human language such as text into numbers so that a computer can process it and expressing it as a vector. Text data has various linguistic features and domain information together, and the core of natural language processing technology is to reflect these features and information as much as possible and represent them in a given vector space. It is called (embedding). The vector extracted through the embedding process is used for Sentiment Analysis, which predicts opinions, emotions, and attitudes in the text as polarity, Named Entity Recognition, which recognizes the type of entity such as a person's name, place name, etc., and automatically recognizes language. It is used as an element technology in various research fields that utilize text, such as machine translation.

Along with the development of deep learning technology, research on improving embedding performance by applying deep learning technologies has been actively conducted in the field of natural language processing. Recently, a text embedding method using a pre-trained language model, which is a model that learns a large-capacity corpus based on deep learning, is mainly used for analysis. As interest in transfer learning, which can bring high performance by performing additional training on a small amount of data based on a pre-learning language model, has increased, various pre-learning language models have been released. In particular, a representative model As a result, BERT (Pre-training of Deep Bidirectional Transformers for Language Understanding) is actively used.

BERT is a model that improves the one-way structure, which is a limitation of existing pre-learning language models, into a bi-directional structure. One of BERT's pre-training methods, MLM (Masked Language Model), randomly ), and then masking, the context information is learned by an unsupervised learning method that predicts the correct answer of the corresponding word. Unlike the existing Left-to-Right language model, MLM can extract high-quality text expressions because masking words are inferred by reflecting both left and right context information. The pre-learned BERT can be applied and utilized to various tasks in the field of natural language processing through fine-tuning that updates weights according to analysis tasks (downstream tasks).

As the performance of BERT has been proven, research to build models suitable for specific domains or specific tasks using BERT continues in various fields. Research on further pre-training is attracting attention. This method can be described as learning the general meaning of a word through prior learning and learning the special meaning or nuance of a word in a corresponding domain and task through additional learning.

Korean Patent Registration No. 10-0766169 (2007.10.04)

An embodiment of the present invention performs emotion classification on movie comments through attention-based LSTM (Long Shot-Term Memory), and classifies clue words and surrounding words according to the level of contribution to emotion classification of each comment. Later, in the additional pre-learning step, it is intended to provide a selective masking-based additional pre-learning method and apparatus for improving classification accuracy that performs selective masking in which masking is applied only to neighboring words.

Among the embodiments, a selective masking-based additional pre-learning method includes performing emotion classification with learning data of a training data population for pre-training; generating a valid word corpus of words of the learning data population according to a result of the sentiment classification; constructing an extended tokenizer by applying the valid word corpus to the tokenizer built through the prior learning; masking the training data using the extended tokenizer; and performing further pre-training using the masked training data.

The performing of the emotion classification may include performing the emotion classification by using sentences to which emotional tags are attached as the learning data.

The performing of the sentiment classification may include extracting attention weights of the words through an attention-based language model.

The performing of the sentiment classification may include comparing the attention weight with a preset threshold and classifying the words into a clue term, a surrounding term, and an excluded term. there is.

In the step of performing the sentiment classification, words having the attention weight higher than the upper 1-Sigma of the average are classified as clue words, words having the attention weight lower than the average are classified as the neighboring words, and the remaining words are classified as the excluded words. A classification step may be included.

The performing of the sentiment classification may include independently setting the threshold for each of the sentences.

The generating of the valid word corpus may include constructing the valid word corpus from words classified as the clue word and the neighboring words.

The masking of the learning data may include segmenting each of the learning data in word units using the extended tokenizer; and selectively masking any one of the words classified as the neighboring words for each of the learning data.

The performing of the additional pre-training may include applying the masked training data to the pre-trained BERT model to perform the additional pre-training.

The method may further include fine-tuning the learning model built through the pre-learning and the additional pre-learning.

Among the embodiments, an optional masking-based additional pre-learning device includes an emotion classification performer configured to perform emotion classification with learning data of a learning data population for pre-training; a corpus generating unit generating a valid word corpus of words of the learning data population according to a result of the sentiment classification; a tokenizer extension unit constructing an extended tokenizer by applying the effective word corpus to the tokenizer built through the prior learning; an optional masking processor for masking the learning data using the extended tokenizer; and an additional pre-training unit performing further pre-training using the masked training data.

The emotion classifier may extract attention weights of the words through an attention-based language model.

The sentiment classification performer may compare the attention weight with a preset threshold and classify the words into clue terms, surrounding terms, and excluded terms.

The corpus generator may configure the effective word corpus with words classified as the clue word and the neighboring word.

The masking processor may segment each of the training data into word units using the extended tokenizer and selectively mask one of the words classified as the neighboring words for each of the training data.

The apparatus may further include a transfer learning performing unit fine-tuning the learning model built through the pre-learning and the additional pre-learning.

The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all of the following effects or only the following effects, so it should not be understood that the scope of rights of the disclosed technology is limited thereby.

A selective masking-based additional pre-learning method and apparatus for improving accuracy according to an embodiment of the present invention performs emotion classification on movie comments through attention-based LSTM (Long Shot-Term Memory), and through this, each comment's After classifying clue words and neighboring words according to the level of contribution to sentiment classification, selective masking may be performed in which masking is applied only to neighboring words in an additional pre-learning step.

1 is a diagram illustrating an additional pre-learning system according to the present invention.
Figure 2 is a diagram explaining the functional configuration of the additional pre-learning device of Figure 1.
3 is a flowchart illustrating an embodiment of a selective masking-based additional pre-learning process according to the present invention.
4 is a diagram illustrating a selective masking-based additional pre-learning method according to the present invention.
5 is a diagram illustrating emotional sentences and tags.
6 is a diagram illustrating a learning process of an attention-based language model.
7 is a diagram for explaining results of classification according to attention weights and thresholds.
8 is a diagram illustrating word words, surrounding words, and valid word corpus according to the present invention.
9 is a diagram illustrating a BERT pre-learning process.
10 is a diagram illustrating BERT tokenizer extension.
11 is a diagram for explaining an embodiment of MLM learning to which selective masking is applied according to the present invention.
12 to 19 are diagrams illustrating experimental results of the selective masking-based additional pre-learning method according to the present invention.

Since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

Meanwhile, the meaning of terms described in this application should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may exist in the middle. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

Expressions in the singular number should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as “comprise” or “having” refer to an embodied feature, number, step, operation, component, part, or these. It should be understood that it is intended to indicate that a combination exists, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identification code (eg, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step clearly follows a specific order in context. Unless otherwise specified, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as consistent with meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

Deep learning may correspond to an algorithm that learns weights connecting each hidden layer based on an artificial neural network in which several hidden layers are stacked. Natural language processing using deep learning can have the advantage of better expressing information and features contained in text through learned weights as vectorized word information passes through a hidden layer. Representative deep learning-based word embedding methods include Word2Vec and FastText. Since then, sequence embedding methods that apply deep learning to statistical-based language models have emerged. Representatively, RNN (Recurrent Neural Network), a language model that learns by reflecting previous word information in the next word information according to the order in a sentence , LSTM, and GRU (Gated Recurrent Unit) showed excellent performance in Seq2Seq (Sequence-to-Sequence) models such as machine translation. However, these methods have limitations such as the problem of not processing words that are not in the learned text corpus (Out Of Vocabulary) and the long-term dependency problem (Long Term Dependency), in which information on words in the past is not delivered until the end as the length of the sentence increases. Have.

To overcome these limitations, active research has been conducted to improve performance by using language models and attention mechanisms. The attention mechanism may correspond to a technique for learning which word information the language model should pay more attention to by obtaining an attention weight for each input word regardless of the length of the sentence. Through the attention function used in the attention mechanism, it is possible to know which part of the source data the target data concentrates more on and reflects the information. can interpret the learning process of Since then, as a Transformer model based on Self-Attention, built using only the attention algorithm without a language model, was proposed, there was a breakthrough in the field of natural language processing.

In recent years, studies have been actively conducted to build a pre-learning language model using a transformer model. The pre-learning language model may correspond to a model designed to overcome limitations of not being able to express unique information of text due to lack of data. Pre-learning language models can extract richer text expressions than existing embedding methods in the process of inferring small-scale data through a series of transfer learning methods after learning general expressions of text through large-scale text corpus like Wikipedia. there is. As a method of applying a pretrained language model to an analysis task, there can be a feature-based approach and a fine-tuning approach. The feature-based approach learns by adding pre-trained language expressions to a model that performs an analysis task, and ELMo (Embedding from Language Model) is a representative example. Fine-tuning approaches can include BERTs, Generative Pre-trained Transformers (GPTs), etc., by adjusting the weights of pre-trained language models using minimal parameters to suit the analysis task.

Existing pre-learning language models may have limitations due to a unidirectional structure in which only information on previous words is reflected and learned. BERT builds a bidirectional model by utilizing the encoder structure of the transformer, and overcomes the limitations of the existing unidirectional model through NSP (Next Sentence Prediction) and unsupervised learning of the MLM method. Among the learning methods of BERT, NSP can learn text representation by giving two sentences as input at the same time and predicting the next sentence after the first sentence. MLM learns by randomly selecting a word from a set of words in a sentence, masking it, and then predicting the word. Only 15% of the total tokens are masked, and 80% of them are [Mask] tokens, respectively. 10% can be learned by replacing it with another word or leaving it as it is. BERT is capable of two-way learning that reflects both contexts through two types of unsupervised learning, and with these advantages, it has been able to achieve SOTA (State-of-the-Art) in most natural language processing fields.

Among BERT's pre-learning methods, MLM has the advantage of being able to learn rich information by bi-directionally predicting masked words through left/right context, and this has led to efforts to improve performance by utilizing MLM in various natural language processing tasks. A follow-up study was conducted. Specifically, there are studies on improving one-way decoders through MLM in machine translation, studies on domain emotion transfer using MLM, and studies on improving the efficiency of pre-learning models by applying continuous MLM.

Meanwhile, recently, studies have been actively conducted to improve the performance of analysis tasks through fine-tuning using pre-learned BERT. Performance improvement through fine-tuning was mainly achieved by adjusting hyperparameters inside BERT or extracting context vectors by fusing with other language models. However, the existing BERT has a limitation in that it is limited to general data such as Wikipedia. Therefore, when domain-specific data is fine-tuned with the existing BERT, there is a limitation that information cannot be completely expressed because the distribution of general corpus learned in BERT and the distribution of words in the domain corpus are different. Accordingly, many studies have been conducted on methods of constructing large-scale domain data and reconstructing domain-specific pre-learning language models. Specifically, there are studies that have learned biomedical texts, studies that have learned scientific knowledge and clinical information, and the like.

On the other hand, attempts to improve BERT by additionally learning domain data on pretrained BERT are also actively being made, and studies are being conducted to additionally learn domain data using MLM, which is mainly used for pretraining. However, since most of these attempts proceed with additional learning in traditional MLM, that is, by randomly selecting and masking words, they may have limitations in that information for improving the performance of analysis tasks is not sufficiently utilized.

Meanwhile, according to the present invention, a selective masking scheme may be disclosed in which words are divided into clue words and neighboring words according to the effect each word has on the performance of an analysis task, and masking is applied only to these words.

1 is a diagram illustrating an additional pre-learning system according to the present invention.

Referring to FIG. 1 , the additional pre-learning system 100 may include a user terminal 110 , an additional pre-learning device 130 and a database 150 .

The user terminal 110 may correspond to a computing device that is connected to the additional pre-learning device 130 to provide information or to use information. That is, the user terminal 110 may provide the additional pre-learning device 130 with learning data necessary for learning, and use the learning data collected by the additional pre-learning device 130 or the additional pre-learning device 130 ) can be used. In addition, the user terminal 110 may be implemented as a smart phone, a laptop computer, or a computer, but is not necessarily limited thereto, and may also be implemented as various devices such as a tablet PC. The user terminal 110 may be connected to the additional pre-learning device 130 through a network, and a plurality of user terminals 110 may be connected to the additional pre-learning device 130 at the same time.

The additional pre-learning device 130 may be implemented as a server corresponding to a computer or program that performs the optional masking-based additional pre-learning method for improving classification accuracy according to the present invention. The additional pre-learning device 130 may be connected to the user terminal 110 through a wired or wireless network and may exchange data with each other.

In one embodiment, the additional pre-learning device 130 may operate in conjunction with various external systems (or servers) in the process of performing the optional masking-based additional pre-learning method for improving classification accuracy according to the present invention. For example, the additional pre-learning device 130 can access related content through SNS services, portal sites, Wikipedia, blogs, etc., and receive data necessary for collecting learning data and building a learning model. can

The database 150 may correspond to a storage device for storing various information necessary for the operation process of the additional pre-learning device 130 . For example, the database 150 may store learning data collected from various sources, and may store learning algorithms and model information for building a learning model, but are not necessarily limited thereto, and the additional pre-learning device 130 may In the process of performing the optional masking-based additional pre-learning method for improving classification accuracy according to the present invention, collected or processed information may be stored in various forms.

Figure 2 is a diagram explaining the functional configuration of the additional pre-learning device of Figure 1.

Referring to FIG. 2 , the additional pre-learning device 130 includes a sentiment classification performer 210, a corpus generator 220, a tokenizer expander 230, an optional masking processor 240, an additional pre-learning performer ( 250), a transfer learning performing unit 260, and a control unit (not shown in FIG. 2).

The emotion classification performer 210 may perform emotion classification with learning data of a learning data population for pre-training. Here, the learning data population may correspond to a data set composed of learning data. Also, the learning data may correspond to text for emotion classification. For example, learning data may include sentences and may be composed of various sentences collected from various sources, such as messages from SNS services, comments registered in blogs or postings, and the like.

In one embodiment, the emotion classification performer 210 may perform a preprocessing operation on the learning data population. For example, the emotion classification performer 210 may remove learning data in which a specific word is used through filtering on the learning data population, and may also remove a predetermined icon or image. In addition, the emotion classification performer 210 may classify and cluster the learning data for the learning data population according to a predetermined criterion. For example, the emotion classification performer 210 may classify learning data for each professional field with respect to the learning data population.

In an embodiment, the emotion classification performer 210 may perform emotion classification by using sentences to which the emotion tags are attached as learning data. The learning data may be composed of text including sentences, and when the learning data is composed of sentences, the emotion classification performer 210 may assign emotion tags by classifying emotions in advance for each sentence. That is, the emotion tag may correspond to a summary expression of emotion felt in the corresponding sentence with a predetermined emotion word or phrase. The emotion classification performer 210 may perform emotion classification on sentences having emotion tags, and may analyze accuracy of emotion classification by comparing emotion classification results and emotion tags.

In an embodiment, the emotion classification performer 210 may extract attention weights of words through an attention-based language model. For example, the sentiment classification performer 210 may use an attention-based LSTM classifier as an attention-based language model. That is, sentences with sentiment tags attached may be input to an attention-based LSTM classifier, and the classifier may generate a sentiment classification result for each sentence as an output. In this case, the emotion classification performer 210 may obtain an attention weight for each word in the sentence in the emotion classification process. That is, the emotion classification performer 210 may collect and store the attention weight for each word in sentence units.

In one embodiment, the emotion classification performer 210 compares the attention weight with a preset threshold value to classify words into a clue term, a surrounding term, and an excluded term. . At this time, the threshold value to be compared with the attention weight may be set in advance and utilized. The emotion classification performer 210 may classify each word in the sentence as one of a clue word, a peripheral word, and a control word based on the attention weight determined for each word.

Here, the clue term may correspond to a word with a high contribution when classifying a sentence in the sentiment classification process, and the surrounding term may correspond to a word with a low classification contribution. Also, an excluded term may correspond to a word having an intermediate level of contribution.

In one embodiment, the sentiment classification performer 210 classifies words with an attention weight higher than the upper 1-Sigma of the average as clue words, and classifies words with an attention weight lower than the average as neighboring words, excluding the other words. can be classified as words. That is, the sentiment classification performer 210 may apply the average of the attention weights and the upper 1-Sigma of the average as reference thresholds for classifying words, respectively.

In one embodiment, the emotion classification performer 210 may independently set threshold values for each sentence. The emotion classification process may be performed in units of learning data, that is, sentences, and when an attention-based LSTM classifier is used, an attention score may be calculated for each word within each sentence. Accordingly, the attention score may be independently determined according to word distribution in a sentence, and a threshold value for word classification may also be independently determined for each sentence. Accordingly, even the same word may have different attention weights according to the word distribution of included sentences, and as a result, word classification results may be different for each sentence.

The corpus generator 220 may generate a valid word corpus for words in the training data population according to the sentiment classification result. In one embodiment, the corpus generator 220 may construct a valid word corpus from words classified as clue words and neighboring words. That is, the valid word corpus may correspond to a word set composed of specific words according to the sentiment classification result. More specifically, the words classified by the sentiment classification performer 210 may be generated in the form of a clue word list, a neighboring word list, and an excluded word list according to the classification result. The corpus generator 220 may generate a valid word corpus by integrating the clue word list and the neighboring word list and then removing duplicate words. As a result, the valid word corpus can be defined as a set of words classified as clue words and peripheral words according to sentiment classification.

The tokenizer expansion unit 230 may construct an extended tokenizer by applying a valid word corpus to a tokenizer built through prior learning. For example, if a BERT model is used in the pretraining process, a BERT tokenizer can be built. The tokenizer extension unit 230 may extend the tokenizer by adding valid word corpus instead of using the previously constructed BERT tokenizer as it is. That is, the tokenizer extension unit 230 may add valid word corpus to the words of the tokenizer built through prior learning, and build an extended tokenizer by removing redundancies.

The selective masking processor 240 may mask the learning data using the extended tokenizer. In one embodiment, the selective masking processor 240 may segment each of the learning data into word units using an extended tokenizer, and selectively select one of words classified as neighboring words for each of the learning data. can be masked with When each sentence corresponding to the learning data is segmented in word units using an extended tokenizer, the selective masking processor 240 may obtain a set of tokens including clue words and neighboring words for each sentence. . Thereafter, the selective masking processor 240 may exclude a clue word from among the segmented words from the masking target, and perform masking by selecting at least one of neighboring words. At this time, the masking operation may be performed by replacing the word to be masked with the word 'MASK'.

The additional pre-training unit 250 may perform further pre-training using the masked training data. In one embodiment, the additional pre-training unit 250 may perform additional pre-learning by applying the masked training data to the pre-trained BERT model. For example, the additional pre-learning unit 250 may perform additional pre-learning through MLM for sentences in which selective masking is performed for surrounding words.

The transfer learning performer 260 may fine-tune the learning model built through pre-learning and additional pre-learning. Here, fine tuning may correspond to a process of learning a new model based on an existing pretrained model. More specifically, fine-tuning may include an operation of removing a classifier existing in the original model and adding a classifier suitable for a new purpose in order to redefine the pretrained model according to a new purpose. Then, fine-tuning can be performed by applying various learning strategies to the modified model.

For example, the fine-tuning process can be performed in a way of learning the entire modified model, and in a way of learning only the remaining layers and classifiers while keeping a part of the convolutional base fixed. It can proceed, and the convolution base can proceed in a way to learn only the classifier in a fixed state. The transfer learning performing unit 260 may adjust a learning rate for fine tuning through a hyperparameter, and may perform fine tuning by variably applying the learning rate according to a purpose to be used.

The controller (not shown in FIG. 2) controls the overall operation of the additional pre-learning device 130, the emotion classification performer 210, the corpus generator 220, the tokenizer expander 230, and the optional masking processor 240, the additional pre-learning unit 250 and the transfer learning unit 260 may manage control flow or data flow.

3 is a flowchart illustrating an embodiment of a selective masking-based additional pre-learning process according to the present invention.

Referring to FIG. 3 , the additional pre-learning device 130 may perform emotion classification with the learning data of the learning data population for pre-training through the emotion classification performer 210 (step S310). ). The additional pre-learning device 130 may generate a valid word corpus for words in the training data population according to the result of sentiment classification through the corpus generator 220 (step S330).

In addition, the additional pre-learning device 130 may build an extended tokenizer by applying a valid word corpus to the tokenizer built through pre-learning through the tokenizer expansion unit 230 (step S350). The additional pre-learning device 130 may mask the learning data using the tokenizer extended through the optional masking processor 240 (step S370). The additional pre-learning device 130 may perform further pre-training using the learning data masked through the additional pre-learning unit 250 (step S390).

Hereinafter, a selective masking-based additional pre-learning method for improving classification accuracy according to the present invention will be described in detail with reference to FIGS. 4 to 19.

Referring to FIG. 4, the additional pre-learning method according to the present invention is based on phase 1 of distinguishing each word into a clue word and a neighboring word according to the level of contribution to the emotion classification task, and selective masking using the information of the word thus distinguished. It can consist of Phase 2, which proceeds with additional pre-learning. More specifically, Phase 1 receives a sentence with a sentiment tag attached as an input, performs sentiment classification through an attention-based LSTM classifier, and classifies each word according to the attention weight of each word in the sentence acquired in this process. It can be divided into clue words and peripheral words from the viewpoint of emotion classification. Phase 2 builds an extended tokenizer by adding the words acquired in Phase 1 to the pre-learned BERT tokenizer, and can perform tokenizing on the input sentence by utilizing it. Subsequently, in Phase 2, additional pre-learning can be performed through the pre-learned BERT's MLM, and in this process, selective masking can be performed by utilizing the information of the previously identified clue words and surrounding words. Through this process, it is possible to finally derive a sentence embedding specialized for sentence classification, that is, a sentence embedding that faithfully reflects the emotional information in the sentence, and details of each process will be described.

Referring to FIG. 5, the additional pre-learning device 130 performs Phase 1 of FIG. 4, that is, the process of extracting the attention weight for each word in the sentence in the emotion classification process using the attention-based LSTM (Step 1), and the extraction It is possible to perform a process (step 2) of constructing a corpus according to emotion information by distinguishing the words. In addition, the additional pre-learning device 130 may use sentences to which a positive or negative tag is attached as review data including emotions. In FIG. 5, a sentence such as 'I couldn't properly reproduce the tension of the original work' can be attached with a negative tag, and 'the highest peak of youth films. Sentences such as ‘self-portraits of wandering and depressed days’ can be tagged as positive.

In order to perform selective masking, it may first be necessary to determine which words in a sentence contribute significantly to sentiment classification. For this purpose, the additional pre-learning device 130 may perform emotion classification by utilizing Attention-based LSTM, which is one of attention-based language models. In this case, the attention-based LSTM may calculate a unique attention weight for each word in the input sentence in the classification process, and the corresponding weight may correspond to the degree of contribution of each word to the determination of the emotional tag. A general learning process for attention-based LSTM can be expressed as shown in FIG.

Referring to FIG. 6, the attention-based LSTM may be largely composed of an LSTM layer and an attention layer. First, the LSTM layer may use the embedding vector value for each token as an input of the LSTM, pass it through the hidden layer inside the LSTM, and then generate a vector in a hidden state. The attention layer may calculate an attention score by applying an attention function to a vector in a hidden state, and then obtain an attention distribution by applying a softmax function to the calculated attention score. At this time, each value of the attention distribution may correspond to an attention weight, and a sentence vector expressing the context is finally extracted through a weighted sum between the attention weight and the vectors in the hidden state that have passed through the LSTM. can Thereafter, in the classification layer, the attention weight may be updated by performing learning to predict the sentiment tag from the sentence vector. Sentimental sentences are classified through attention-based LSTM, and an example of visualizing a part of the results of extracting attention weights for each word in the process can be expressed as shown in FIG. 7 .

Referring to FIG. 7 , an attention weight value of each word may be derived through attention-based LSTM learning, and may be classified according to a preset threshold. Since the sentences of FIG. 7 are all classified as negative, the attention weight of each word may correspond to the degree of contribution of the corresponding word to classify a given sentence as negative. For example, words such as 'wasted' and 'worst' in the first sentence and 'wasted' and 'boring' in the second sentence may correspond to words that significantly affect the prediction of the negative tag.

In this way, words that play an important role in determining the emotion of a sentence and words that do not play a role can be distinguished through the attention weight generated in the classification process through the attention-based LSTM. In the additional pre-learning method according to the present invention, words with a high attention weight in emotional sentences can be classified as 'clue words' with a high contribution when classifying the corresponding sentence, and words with a low attention weight can be classified as 'surrounding words' with a low contribution to classification. there is. In addition, words with an intermediate level of contribution can be designated as 'Excluded Terms' and excluded from the analysis. The additional pre-learning method according to the present invention may perform selective masking using the role information of the divided words.

However, since the attention-based LSTM generates attention weights for each word for each sentence, the role designation of words can be done in units of sentences rather than in the entire document. That is, the same word may be classified as a clue word in one sentence and as a neighboring word in another sentence. In FIG. 7 , the word 'waste' can be classified as a clue word in the first sentence and as a peripheral word in the second sentence. In addition, in the additional pre-learning method according to the present invention, a different value may be designated for each sentence as a threshold value for distinguishing the role of each word. More specifically, for each sentence, a word having an attention weight higher than the upper 1-Sigma of the average may be selected as a clue word, and a word having an attention weight lower than the average may be selected as a neighboring word. In addition, words that are not included in both the clue word and the neighboring words, that is, words whose attention weight is higher than the average and lower than the upper 1-Sigma of the average, may be classified as excluded words. In FIG. 7 , weight thresholds (1-sigma) for being classified as clue words may be set differently, such as 0.192 for the first sentence and 0.277 for the second sentence.

Afterwards, the clues and neighboring words classified by the threshold can be used for two purposes: one for extending BERT's tokenizer and one for performing selective masking. For the first use, since the main purpose is the expansion of the lexicon, it can be used for the expansion of the BERT tokenizer after integrating all the clue words and all the surrounding words to remove redundancies. Meanwhile, in the case of the second use, since word role information is used for selective masking performed for each sentence, overlapping words between sentences can be stored as they are without removing them. For example, (a) of FIG. 8 may indicate a set of clue words used for selective masking, and (b) of FIG. 8 may indicate a set of neighboring words. In addition, (c) of FIG. 8, which is a corpus from which duplicates are removed after integrating them, can be used to expand the BERT tokenizer. For example, in the case of 'boring', it can be selected as a clue word in both the second and fourth sentences in FIG. 8(a), and can appear only once after removing duplicates in FIG. 8(c).

The additional pre-learning device 130 adds the clue/surrounding word corpus built in the previous step to the pre-learned tokenizer of BERT to segment sentiment sentences (step 3), and divides the segmented sentiment sentences into clue words and surrounding words. The process of performing selective masking using the role information of and the process of performing additional pre-learning through the pre-learned BERT (step 4) can be performed. BERT is a model built in both directions by borrowing only the encoder part of a transformer, and pre-learning can be achieved through generally 12 transformer layers and a large text corpus. The pre-learning process of BERT can be expressed as shown in FIG.

Referring to FIG. 9 , a general pre-learning process of BERT may be largely composed of an embedding layer, a transformer layer, and a training layer. Specifically, the embedding layer converts large-scale text data into vectors, and results of performing token embedding, segment embedding, and position embedding for a single word. You can construct a vector by adding them all. In the process of passing through the hidden layer of the transformer, the constructed vectors can learn weights for vector representation from various viewpoints through 12 transformer encoders and multi-head attention in each encoder. After that, in the learning layer, the weights of all layers can be updated so that BERT can express the context better by simultaneously performing learning using MLM and NSP, which are BERT's pre-learning methods.

In order to additionally learn emotional sentences by applying selective masking to the pre-learned BERT, an operation of first segmenting and converting the text into an input form of the pre-trained BERT embedding layer may be performed. Most of the pre-trained BERTs have their own tokenizer that segments text into tokens, and the tokenizer is based on the Byte Pair Encoding (BPE) algorithm that learns by splitting a single word into multiple subwords. text can be segmented. However, if the pre-learned tokenizer of BERT is applied as it is, there is a risk that the clue words derived from steps 1 and 2 and surrounding words will be segmented into sub-words and lose their original meaning. Therefore, in order to prevent this, a process of extending the BERT tokenizer by adding a corpus composed of clues/surrounding words to the pretrained BERT tokenizer (step 3) may be required. 10 may show an example of a result of tokenizing with the existing corpus of clues/surrounding words added and tokenizing without it.

10(a) may correspond to an embodiment of a result of applying the BERT tokenizer as it is without expanding it, and the clue word of the original sentence and the surrounding words such as 'detailed' and 'concluding' are divided into sub-words. It can be seen that the meaning of is damaged. On the other hand, (b) of FIG. 10 may correspond to an example of the result of segmentation through the extended BERT tokenizer through corpus addition, and it can be confirmed that clues/surrounding words are maintained intact without loss of meaning. Therefore, the additional pretraining device 130 may construct and use an extended BERT tokenizer by adding these words to the existing BERT tokenizer in order for the BERT tokenizer to fully recognize the clue words and surrounding words.

Next, the additional pre-learning device 130 may perform selective masking (step 4) only on peripheral words that have less influence on sentiment tag prediction by utilizing word role information. Existing random masking methods may have limitations in that it is difficult for other words to sufficiently learn the emotion of a sentence when a word that has a significant influence on the emotion judgment of a sentence is used for masking. In contrast, the additional pre-learning device 130 controls clue words not to be used for masking, thereby ensuring that other words can sufficiently learn the emotion of the sentence through the clue words. Furthermore, the additional pre-learning device 130 may maximize the emotional learning effect of other words by forcing only surrounding words to be used for masking. More specifically, the additional pre-learning device 130 receives the segmented emotional sentence as an input of the pre-learned BERT embedding layer, and then covers only the words surrounding each sentence with [MASK] tokens. Performs a modified MLM can do. An embodiment of performing selective masking on neighboring words may be illustrated as shown in FIG. 11 .

Referring to FIG. 11, selective masking ensures an opportunity for other words to learn positive nuances from clue words by masking the peripheral word 'direction' and maintaining the clue words 'detailed' and 'inspiring'. can Similarly, in the next learning, another peripheral word 'conclusion' may be masked, and even in this case, clue words 'detailed' and 'impressive' may still be maintained. That is, the additional pre-learning device 130 may fundamentally block the possibility that the clue words 'detailed' and 'impressive' are masked.

The additional pre-learning device 130 performs MLM learning using the pre-learned BERT, but can apply a method of selectively selecting the target to cover the [MASK] token according to the role of the word rather than an arbitrary method. The additional pre-learning device 130 ensures that clue words rich in emotional information are always maintained without being removed in the masking process, thereby maximizing the influence of clue words on the inference of surrounding words, and ultimately providing emotional information. It is possible to derive a sentence embedding result in which is faithfully expressed.

Hereinafter, the result of performing an experiment by applying the additional pre-learning method according to the present invention to actual data and the performance evaluation result will be described in more detail.

As the experimental data, NSMC (Naver Sentiment Movie Corpus), a public Korean movie review data set, can be used. As shown in FIG. 5, NSMC is open data consisting of pairs of tags for comments and emotions for movies, and 200,000 data is composed of 1:1 ratio. Here, after attention-based sentiment classification is performed by building about 16,000 cases for training and validation among the entire data, additional pre-learning is performed through the additional pre-learning method according to the present invention. can The experimental environment can be built with Python 3.7, and KoBERT, which is implemented based on Pytorch, can be used as the pre-learning model.

First of all, the emotion classification of about 16,000 emotional comments is performed using the attention-based emotion classification model, and through this, the result of extracting the attention weight can be obtained. More specifically, here, the LSTM layer vectorizes tokens in a sentence using bidirectional LSTM, and the attention layer may calculate an attention weight and extract a sentence vector through Bahdanau Attention. Then, classification learning can be performed by building fully connected layers on the extracted sentence vectors. 12 shows a model structure used for emotion classification.

Thereafter, an attention weight updated through classification learning is extracted, and a threshold value is calculated for each sentence to construct a clue word list and a neighboring word list, respectively. More specifically, for clue words, if the average of the attention weights of constituent words is obtained for each sentence, and words having a value higher than 1-sigma above the average are classified as clue words, and neighboring words are lower than the average of the attention weights It can be distinguished by surrounding words. Next, to extend the BERT tokenizer, we can construct a corpus of 26,896 valid words by integrating the clue word list and the neighboring word list, and then removing redundancies. FIG. 13 may correspond to an example of an attention weight result extracted through classification learning, and FIG. 14 is a construction of a clue word list, a surrounding word list, and a valid word corpus. This may correspond to an embodiment of the result.

Next, the additional pre-learning device 130 builds an extended BERT tokenizer by adding the constructed effective word corpus to the pre-learned BERT, and then performs additional pre-learning based on selective masking using clue/surrounding word list information. can be done Here, the KcBERT-Base model can be used for learning Korean. KcBERT was pre-trained using large-scale Korean news comment data, and about 15.5 GB of text data and more than 89 million sentences were used. The tokenizer used for learning uses the BPE-based Word Piece tokenizer, and a total of 30,000 words are built. 50,672 extended tokenizers can be built after finally deduplication by adding valid word corpus built to the corresponding tokenizer. 15 shows an example of a result of segmenting a sentiment sentence using the extended BERT tokenizer.

Next, the additional pre-learning device 130 may perform additional pre-learning after selectively masking the segmented emotional sentence by using a list of neighboring words. That is, clue words are excluded from masking, and selective masking may be performed in a manner in which up to two words are selected from the neighboring word list and replaced with [MASK] tokens. Thereafter, additional pre-learning may proceed using MLM. At this time, the weight of KcBERT-Base can be used as an initialization value, and the hyperparameters used for learning can be set to 12 transformer blocks, 768 dimensions of the hidden layer, and 12 attention heads. The maximum length of emotional comments can be set to 128, and it can be confirmed through FIG. 16 that the embedding vector value of KcBERT-Base has changed after additional pre-learning.

Here, we introduce the results of analyzing the performance of the proposed methodology by comparing the additional pre-trained BERT with selective masking and three models built for performance verification. The overall process of the performance evaluation experiment is shown in FIG. 17 .

17(A) may show a flow of classifying emotional comments by applying the additional pre-learning method according to the present invention. Specifically, the additional pre-learning method applies masking only to surrounding words, performs fine-tuning with the weight of the additionally learned BERT, and then extracts sentence vectors for emotional comments to perform sentiment classification learning. Models (B), (C), and (D) of FIG. 17 are models built for comparison with the present invention. (D) the model can use only the pre-trained BERT without performing additional pre-training. On the other hand, model (C) is a model that can be implemented in the process of selective masking according to the present invention, and may correspond to a model in which masking is applied only to clue words as opposed to applying masking only to surrounding words in the present invention.

To perform classification learning on emotional comments, 15,932 emotional comments were divided into 12,745 for training and 3,187 for verification, and the positive/negative ratio can be equally set to 1:1 within each data set. In addition, the performance of the model can be evaluated through 5,000 test data. A deep neural network was used as a classification model for performance comparison, and classification accuracy was measured for each of the four models including the additional pre-learning method according to the present invention to compare the performance of the model. can do. The classification accuracy comparison results of each model can be found in FIGS. 18 and 19 .

FIG. 19 may correspond to a comparison graph showing loss according to emotion classification learning in units of epochs for the four models introduced in FIG. 17 . The data for verification was used as the measured value, and it can be seen that all four models have the lowest loss value in the second epoch. Therefore, each model adopts the model of the corresponding epoch to perform inference on the evaluation data, and the classification accuracy can be shown as shown in FIG. 19.

19 is a result of measuring the classification accuracy for the four models, and it can be confirmed that the (A) model according to the present invention shows higher classification accuracy than other comparison models for both the verification data and the evaluation data. In particular, the (C) model in which masking was performed only on clue words showed the lowest classification accuracy. can

As a result, the model derived through the selective masking method according to the present invention, that is, the method of performing additional dictionary learning by distinguishing peripheral words from clue words and performing masking on only the peripheral words, is different from the existing method, that is, additional dictionary learning. It can show superior performance in terms of classification accuracy compared to the model derived by the method that does not perform and the method that performs additional pre-learning through random masking.

The additional pre-learning method according to the present invention may propose a selective masking method using emotion information in emotional sentences, that is, a method of inferring a sentence vector specialized for emotion classification by masking peripheral words having a low contribution to emotion classification. In addition, the additional pre-learning method according to the present invention may also propose a method of measuring the emotional contribution of words in emotional sentences by using the attention weight to select neighboring words. The additional pre-learning method according to the present invention may be performed with the goal of improving classification accuracy, but may also contribute to performance improvement of other natural language processing tasks through application of transformation.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: additional pre-learning system
110: user terminal 130: additional pre-learning device
150: database
210: emotion classification unit 220: corpus generator
230: tokenizer extension unit 240: optional masking unit
250: Additional pre-learning unit 260: Transfer learning unit

Claims (16)

performing emotion classification with learning data of a learning data population for pre-training;
generating a valid word corpus of words of the learning data population according to a result of the sentiment classification;
constructing an extended tokenizer by applying the valid word corpus to the tokenizer built through the prior learning;
masking the training data using the extended tokenizer; and
Optional masking-based additional pre-training method comprising: performing additional pre-training using the masked training data.
The method of claim 1, wherein the performing of the emotion classification comprises:
and performing the emotion classification by using sentences to which emotion tags are attached as the training data.
The method of claim 1, wherein the performing of the emotion classification comprises:
A selective masking-based additional dictionary learning method comprising extracting the attention weights of the words through an attention-based language model.
The method of claim 3, wherein the performing of the emotion classification comprises:
Classifying the words into a clue term, a surrounding term, and an excluded term by comparing the attention weight with a preset threshold value Optional masking-based additional dictionary learning method.
The method of claim 4, wherein the performing of the emotion classification comprises:
Classifying words having an attention weight higher than the upper 1-Sigma of the average as the clue words, words having the attention weight lower than the average as the neighboring words, and classifying the remaining words as the excluded words. Optional masking-based additional pre-learning method with .
The method of claim 4, wherein the performing of the emotion classification comprises:
A selective masking-based additional pre-learning method comprising the step of independently setting the threshold for each of the sentences.
5. The method of claim 4, wherein generating a corpus of valid words comprises:
and constructing the effective word corpus with words classified as the clue words and the neighboring words.
The method of claim 4, wherein masking the learning data
segmenting each of the learning data into word units using the extended tokenizer; and
and selectively masking any one of the words classified as the neighboring words for each of the learning data.
The method of claim 1, wherein the performing of the additional pre-learning
and applying the masked training data to a pretrained BERT model to perform the additional pretraining.
According to claim 1,
Optional masking-based additional pre-learning method further comprising: fine-tuning the learning model built through the pre-learning and the additional pre-learning.
an emotion classification performer performing emotion classification with learning data of a learning data population for pre-training;
a corpus generating unit generating a valid word corpus of words of the learning data population according to a result of the sentiment classification;
a tokenizer extension unit constructing an extended tokenizer by applying the effective word corpus to the tokenizer built through the prior learning;
an optional masking processing unit for masking the learning data using the extended tokenizer; and
Optional masking-based additional pre-training device comprising: an additional pre-training unit that performs additional pre-training using the masked training data.
The method of claim 11, wherein the emotion classification performing unit
A selective masking-based additional dictionary learning device, characterized in that for extracting the attention weights of the words through an attention-based language model.
13. The method of claim 12, wherein the emotion classification performing unit
Optional masking-based additional dictionary learning device, characterized in that the words are classified into a clue term, a surrounding term, and an excluded term by comparing the attention weight with a preset threshold.
The method of claim 13, wherein the corpus generator
Optional masking-based additional dictionary learning device, characterized in that for configuring the effective word corpus with words classified as the clue word and the neighboring word.
The method of claim 13, wherein the masking processing unit
Selective masking-based addition characterized in that each of the training data is segmented into word units using the extended tokenizer and any one of the words classified as the neighboring words is selectively masked for each of the training data Pre-learning device.
According to claim 11,
Optional masking-based additional pre-learning device further comprising: a transfer learning performer for fine-tuning the learning model built through the pre-learning and the additional pre-learning.

Images