KR20230058951A - Apparatus and method for classifying text - Google Patents

Abstract

A text classification apparatus and method are provided. A text classification apparatus according to an embodiment determines whether an input sentence to be classified is a positive sentence or a negative sentence based on a language model learned by training data including a plurality of sentences labeled as positive or negative. a first classifier that does; and a second classifier that reclassifies at least some of the target sentences classified as positive or negative by the first classifier into neutral sentences based on a trustscore of a classification result of the first classifier. .

Description

Apparatus and method for text classification {APPARATUS AND METHOD FOR CLASSIFYING TEXT}

The disclosed embodiments relate to techniques for classifying the polarity of text.

Sentiment analysis is a problem that classifies the polarity of a given sentence as positive or negative. Since most of the datasets for sentiment analysis are labeled positive or negative, it is common for models learned through this to be classified as positive or negative. However, it is difficult for the trained model to judge the polarity of neutral sentences that do not belong to either positive or negative sentences. As described above, the problem of classifying classes not used in learning is required in the inference process is called open set recognition.

In order to solve Open Set Recognition, it is important to express the probability distribution of classes to be classified well. However, in conventional technologies, the accuracy of the model increases as the structure of the model increases, but the confidence in the classes predicted by the model often decreases, so there is a limit to directly applying this to Open Set Recognition.

In addition, a classifier based on a neural network generally classifies a class by using a value obtained by applying a softmax function to a model output value as a probability distribution. However, it is not suitable for solving the Open Set Recognition problem, which requires prediction of new unlearned classes.

Republic of Korea Patent Registration No. 10-2216768 (2021.02.09)

The disclosed embodiments are intended to provide a technical means for classifying all positive, negative, and neutral sentences using only a dataset consisting of only positive and negative sentences.

According to an exemplary embodiment, a first step for determining whether an input sentence to be classified is a positive sentence or a negative sentence based on a language model learned by training data including a plurality of sentences labeled as positive or negative. 1 classifier; and a second classifier that reclassifies at least some of the target sentences classified as positive or negative by the first classifier into neutral sentences based on a trustscore of a classification result of the first classifier. , a text classification device is provided.

The first classifier may perform learning on the training data using an objective function set to include normalized noise in the training data.

The objective function may include a preset noise function, and the noise function may be set such that a function value is maximized when a distance between the noise-included learning data and the original learning data is within a preset range.

The first classifier uses a hidden state vector generated by inputting the [CLS] token included in the target sentence into a pre-trained language model, so that the target sentence for classification is It is possible to determine whether the sentence is positive or negative.

The second classifier determines the distance between the target sentence on the manifold in which the training data is embedded and the class predicted by the first classifier, and the distance between the target sentence and the nearest class excluding the predicted class. The confidence score can be calculated based on .

The second classifier generates a plurality of hidden state vectors by inputting the [CLS] token included in each of the training data to the language model, and extracts the rest of the hidden state vectors from the hidden state vectors except for a certain ratio with a low distribution density into the language model. You can create a set for each class by embedding on a fold.

The second classifier may reduce dimensions of the plurality of hidden state vectors before embedding the generated plurality of hidden state vectors on the manifold.

The second classifier may reclassify the target sentence as a neutral sentence when the confidence score is equal to or less than a preset first threshold value.

When the distance between the target sentence and each class on the manifold in which the training data is embedded is equal to or greater than a preset second threshold value, the target sentence may be reclassified as a neutral sentence.

According to another exemplary embodiment, as a method performed on a computer, based on a language model learned by training data including a plurality of sentences labeled as positive or negative, whether an input sentence to be classified is positive or negative. a first classification step of determining whether the sentence is a sentence; and a second classification step of re-classifying at least some of the target sentences classified as positive or negative by the first classifier into neutral sentences based on a trustscore of a classification result of the first classifier. A text classification method is provided.

In the first classification step, learning on the training data may be performed using an objective function set to include normalized noise in the training data.

The objective function may include a preset noise function, and the noise function may be set such that a function value is maximized when a distance between the noise-included learning data and the original learning data is within a preset range.

In the first classification step, the classification target sentence is generated by using a hidden state vector generated by inputting the [CLS] token included in the target sentence to a pre-trained language model. It is possible to determine whether the sentence is positive or negative.

In the second classification step, the distance between the target sentence on the manifold in which the learning data is embedded and the class predicted by the first classifier, and the distance between the target sentence and the closest class excluding the predicted class Based on the distance, the confidence score may be calculated.

In the second classification step, a plurality of hidden state vectors are generated by inputting the [CLS] token included in each of the learning data into the language model, and the rest of the hidden state vectors excluding a certain ratio having a low distribution density are described above. A set for each class can be created by embedding on a manifold.

The second classification step may further include reducing dimensions of the plurality of hidden state vectors before embedding the generated plurality of hidden state vectors on the manifold.

In the second classification step, when the confidence score is equal to or less than a preset first threshold value, the target sentence may be reclassified as a neutral sentence.

In the second classification step, when distances between the target sentence and each class on the manifold in which the training data is embedded are all equal to or greater than a preset second threshold value, the target sentence may be reclassified as a neutral sentence.

According to the disclosed embodiments, a sentiment classification model capable of classifying even neutral sentences can be configured using only a dataset consisting of only positive and negative sentences. Accordingly, it is possible to reduce the time and cost required to build learning data.

Also, according to the disclosed embodiments, reliability of a PLM-based classification model may be increased by applying adversarial training to a pre-trained language model (PLM).

1 is a block diagram for explaining a text classification apparatus according to an exemplary embodiment;
2 is a block diagram for explaining a detailed configuration of a first classifier according to an exemplary embodiment;
3 is an exemplary diagram for explaining a process of reclassifying at least a part of a target sentence into a neutral sentence by using a confidence score in a second classifier according to an embodiment;
Figure 4 (a) is an example of a low-dimensional manifold when the classification problem is defined as a closed-set classification (closed-set classification)
Figure 4 (b) is an example of a low-dimensional manifold when the classification problem is defined as an open-set recognition problem.
5 is a flowchart illustrating a re-classification process of a second classifier according to an exemplary embodiment;
6 is a flowchart for explaining a text classification method according to an exemplary embodiment;
7 is a block diagram for illustrating and describing a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The detailed descriptions that follow are provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification. Terminology used in the detailed description is only for describing the embodiments of the present invention and should in no way be limiting. Unless expressly used otherwise, singular forms of expression include plural forms. In this description, expressions such as "comprising" or "comprising" are intended to indicate any characteristic, number, step, operation, element, portion or combination thereof, one or more other than those described. It should not be construed to exclude the existence or possibility of any other feature, number, step, operation, element, part or combination thereof.

1 is a block diagram illustrating a text classification apparatus 100 according to an exemplary embodiment. The text classification apparatus 100 according to an embodiment learns training data labeled as positive or negative, and classifies an input sentence into one of three classes of positive/negative/neutral using the learned model, a so-called open text classification apparatus 100. It is a device to solve the problem of set recognition (Open Set Recognition). As shown, the text classification apparatus 100 according to an embodiment includes a first classifier 102 and a second classifier 104 .

The first classifier 102 determines whether the input sentence to be classified is a positive sentence or a negative sentence. The first classifier 102 determines whether the target sentence belongs to any class among positive and negative by using a model learned from training data including a plurality of sentences labeled as positive or negative.

In one embodiment, the first classifier 102 may use a classifier based on a language model in the present invention. For example, the first classifier 102 may use a pre-trained language model (PLM) based on a transformer structure. PLM refers to a model that learns the overall structure and contextual text representation of a language to be learned using a large-scale unlabeled text corpus. PLM can be fine-tuned to execute a specific task, and since linguistic knowledge has already been learned, it has the advantage of being able to achieve high performance in that task with only a small amount of training data.

2 is a block diagram for explaining a detailed configuration of the first classifier 102 according to an exemplary embodiment. As shown, the first classifier 102 according to one embodiment includes a language model 202 and a classifier 204 .

The language model 202 is a pretrained language model (PLM), and may be a model learned by training data including a plurality of sentences labeled as positive or negative. The language model 202 receives a plurality of input tokens 206 generated from sentences to be classified. At this time, the input token 206 may include a [CLS] token indicating the beginning of a sentence and a [SEP] token for distinguishing a sentence from another sentence. Inside the language model 202, a hidden states vector 210 corresponding to each token embedding 208 is generated. The classifier (204, classifier) converts the vector of the hidden state (212) of the [CLS] token generated by the language model (202) into a two-dimensional probability vector (214, logit), and inputs it to the argmax function so that the sentence is positive or classified as negative sentences.

In an embodiment, the first classifier 102 may perform learning on the training data using a loss function set to include normalized noise in the training data.

As described above, the PLM-based classifier classifies sentences as positive or negative through fine-tuning to perform a classification task on a pretrained model. However, the amount of training data used for fine tuning is significantly smaller than the amount of data used for pre-training of the language model. This is because pre-trained language models are very large and complex. For this reason, the language model 202 included in the first classifier 102 has a limitation of overfitting the training data used in the fine tuning step. An overfitting model is vulnerable to reasoning on data that is not included in the training data, and its generalization ability is poor, reacting sensitively to even the slightest difference from the training data, which can lead to classification errors.

In order to solve this problem, in the disclosed embodiment, learning is performed on learning data using an adversarial training method. Adversarial learning refers to creating so-called adversarial data and training the model along with existing data to attack the model and in the process of defending the model, it is more robust than the existing model. It is a learning method that makes it robust against the method of incapacitating a deep learning model by giving In the disclosed embodiment, by learning the first classifier 102 through adversarial learning, not only the accuracy of positive/negative classification can be improved, but also neutral sentence detection performance based on the corresponding model can be improved.

이고, n개의 입력 데이터가

(x_i는 입력 데이터의 i번째 임베딩, y_i는 해당 입력 데이터의 레이블)로 표현될 때, 목적함수는 다음의 수학식 1과 같다.In one embodiment, the first classifier 102 performs learning by adding normalized noise to an embedding for an input token in a learning step (fine tuning) using positive/negative training data. To this end, the first classifier 102 uses an objective function (loss function) including a preset noise function. Specifically, the model

, and n input data

When expressed as (x _i is the ith embedding of the input data, y _i is the label of the corresponding input data), the objective function is as shown in Equation 1 below.

[Equation 1]

*

)를 추가적으로 포함한다. 상기 노이즈 함수는, 상기 노이즈가 포함된 학습 데이터와 원 학습 데이터와의 거리가 기 설정된 범위(ε) 이내인 경우 함수값이 최대가 되도록 설정될 수 있다. 이와 같은 목적 함수를 사용할 경우 결과적으로 학습 데이터에 정규화된 노이즈가 추가된 것과 같은 효과를 줄 수 있다. 이렇게 만들어진 학습 데이터는 원 데이터와는 차이가 있지만, 학습된 모델의 결과가 크게 달라지지는 않는다. 이와 같이 학습 데이터에 정규화된 노이즈를 추가할 경우 모델의 과적합을 방지함으로써 적은 양의 데이터로 파인 튜닝 되는 경우에도 더 나은 일반화 성능을 가진 강건한 모델을 만들 수 있다. 또한 강건한 모델의 경우 학습 데이터에 포함되지 않은 문장들에 대한 추론 성능이 그렇지 않은 모델보다 높으므로, 학습 데이터에 포함되지 않은 중립 데이터를 분류해내고자 하는 과제에서도 큰 성능 개선을 가져온다.Normally, in fine tuning, an objective function is used to compare the model result with the correct answer label. In the disclosed embodiment, the second term here is the noise function (

) is additionally included. The noise function may be set to have a maximum function value when a distance between the noise-included training data and the original training data is within a preset range ε. When such an objective function is used, as a result, it can give the same effect as adding normalized noise to the training data. Although the training data created in this way is different from the original data, the result of the trained model does not change significantly. In this way, when normalized noise is added to the training data, it is possible to create a robust model with better generalization performance even when fine-tuned with a small amount of data by preventing overfitting of the model. In addition, in the case of a robust model, inference performance on sentences not included in the training data is higher than that of a model that is not included in the training data, resulting in a large performance improvement in the task of classifying neutral data not included in the training data.

Next, the second classifier 104 reclassifies at least a portion of the target sentences classified as positive or negative by the first classifier 102 into neutral sentences.

In sentiment analysis, there are cases in which sentences cannot be classified as positive or negative. For example, a sentence may not contain an evaluation of an object, and conflicting evaluations may coexist within a sentence. The second classifier 104 defines sentences that cannot be classified as positive or negative as neutral sentences, and uses the first classifier 102 trained only with data labeled only as positive or negative to class not included in the training data. is configured to detect neutral sentences.

The second classifier 104 converts at least a part of the target sentence classified as positive or negative by the first classifier 102 into a neutral sentence based on a trustscore for the classification result of the first classifier 102. It is configured to be reclassified as

3 is an exemplary diagram for explaining a process of reclassifying at least a part of a target sentence into a neutral sentence by using a confidence score in the second classifier 104 according to an embodiment. In the disclosed embodiments, the confidence score is a value for determining whether the positive or negative classification result of the first classifier 102 is reliable based on the distribution of training data. In order to calculate the confidence score, it is assumed that a low-dimensional manifold (302, low-dimensional manifold embedded with training data) in which training data used to learn the first classifier 102 is embedded. The second classifier 104 generates a plurality of hidden state vectors by inputting the [CLS] token included in each of the training data into the language model 202, and the rest of the hidden state vectors except for a certain percentage of the hidden state vectors having a low distribution density A set for each class is created by embedding on the manifold.

In one embodiment, the second classifier 104 may reduce the dimensionality of the plurality of hidden state vectors before embedding them on the manifold 302 . In general, in the case of a PLM-based language model, the dimension of the hidden state vector is output as very large, such as 256 or 768. Therefore, if this is used as it is, the amount of calculation may be excessively large, so the second classifier 104 can reduce the dimension of the vector by using a method such as principal component analysis (PCA).

The second classifier 104 configures an α-high-density-set with the rest of the hidden state vectors having a reduced dimension, excluding a certain proportion having a low distribution density. The α-high-density-set may be defined as a set of training data excluding data of α% having the lowest distribution density among training data as follows.

이고, f를

상의 연속확률밀도라고 하면,

and f

If the continuous probability density of the phase is

은

로 정의된다. α-high-density-set of f

silver

is defined as

이다. here

am.

4(a) is an exemplary diagram of a low-dimensional manifold when the classification problem is defined as closed-set classification. When the problem is defined in this way, all data is classified only into the positive and negative classes included in the training data, and out-of-domain data (neutral sentences) are classified as in-domain It is misclassified as if it were data.

Figure 4 (b) shows an example of a manifold when the classification problem is defined as an open-set recognition problem. In this case, in order to classify label data not included in the training data as out-of-domain data on the manifold, a set for each class is configured on the manifold and a region not belonging to any class is created. In order to classify out-of-domain data, it is necessary to determine whether the corresponding data is included in the class set of training data on the manifold of (b). In the disclosed embodiments, this may be determined using a confidence score and a distance between the classes. In the disclosed embodiments, in order to improve the out-of-domain data classification performance by more robustly expressing the set for each class on the corresponding manifold, α-high-density-excluding training data that can be judged as outliers Only the set is embedded into the manifold.

The second classifier 104 may calculate a confidence score using the α-high-density-set of the training data calculated as described above. In the manifold 302 , remaining data (α-high-density-set) from which data that may be determined as outliers are removed from the learning data is embedded. The confidence score is the distance between the target sentence to be classified on the manifold 302 in which the training data is embedded and the class classified (predicted) by the first classifier 102 and the closest class excluding the target sentence and the predicted class. is calculated based on the distance of Specifically, the second classifier 104 divides the distance between the target sentence and the class classified (predicted) by the first classifier 102 by the distance between the target sentence and the nearest class excluding the predicted class. The trust score can be defined as It can be considered that the classification result of the first classifier 102 is more reliable as this value is larger.

5 is a flowchart 500 illustrating a reclassification process of the second classifier 104 according to an exemplary embodiment. In the illustrated flowchart, the method or process is divided into a plurality of steps, but at least some of the steps are performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, or shown. One or more steps not yet performed may be added and performed.

In step 502, the second classifier 104 reduces the dimensionality of the hidden state vector generated when the [CLS] token included in the sentence to be classified is input to the language model 202 of the first classifier 102. As described above, the second classifier 104 may reduce the dimension of the corresponding vector using a method such as principal component analysis (PCA).

In step 504, the second classifier 104 calculates the distance between the sentence to be classified and all classes (positive and negative) on the low-dimensional manifold 302. Since a method for calculating the distance per class on a manifold is well known in the art, a detailed description thereof will be omitted. For example, the second classifier 104 may set the maximum distance between data belonging to each class and the sentence to be classified as the distance for each class, or set the distance to the central point of each class as the distance for each class.

At step 506, the second classifier 104 calculates a confidence score. As described above, the second classifier 104 determines the distance between the target sentence on the manifold in which the learning data is embedded and the class predicted by the first classifier 102, and the target sentence and the predicted class. The confidence score may be calculated based on the distance to the nearest class excluding .

In step 508, the second classifier 104 determines whether the confidence score is greater than a preset first threshold value. If, as a result of the determination in step 508, the confidence score is equal to or less than the preset first threshold value, in step 510, the second classifier 104 reclassifies the target sentence as a neutral sentence.

If, as a result of the determination in step 508, the confidence score is greater than the first threshold value, in step 510, the second classifier 508 calculates the distance calculated in step 504, that is, the manifold in which the training data is embedded. It is determined whether all distances between the target sentence and each class on the image are equal to or greater than a preset second threshold value.

If, as a result of the determination in step 510, the distances to each class are equal to or greater than the second threshold, the second classifier 104 reclassifies the target sentence as a neutral sentence. However, if at least one of them is less than the second threshold, the second classifier 104 maintains the classification result of the first classifier 102 in step 514 .

6 is a flowchart illustrating a text classification method 600 according to an exemplary embodiment. The illustrated method may be performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, such as the text classification apparatus 100 . In the illustrated flowchart, the method or process is divided into a plurality of steps, but at least some of the steps are performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, or shown. One or more steps not yet performed may be added and performed.

In step 602, the first classifier 102 of the text classification apparatus 100 determines whether the input sentence to be classified is a positive sentence or a negative sentence. In one embodiment, the first classifier 102 is trained by learning data including a plurality of sentences labeled as positive or negative, and based on this, it can determine whether the target sentence is positive/negative.

In step 604, the second classifier 104 of the text classification apparatus 100 classifies the text as positive or negative based on the trustscore of the classification result of the first classifier 102. At least a part of the target sentence is reclassified as a neutral sentence.

Table 1 below is a table comparing the performance of four methods for classifying neutrals using only the positive-negative classification model. The training dataset used to train the model is positive/negative labeled data that includes product evaluations, and the test data for model evaluation is data such as customer center utterance data and internet comments. Data that has been collected, refined, and inspected. The test data contains neutral, positive, and negative sentences in a ratio of 2:1:1. The neutral classification criterion threshold was set as the Q1 value of the entire training data for each criterion. Experiments (A) to (H) all used the same language model, and after pooling the results of the language model, a structure with a single-layer Fully Connected Neural Network was used to predict the distribution of correct answers.

(go) (me) (all) (la) accuracy 0.668 0.687 0.716 0.820

(A) and (B) classify neutral data using only positive/negative classifiers. In (A), when the softmax value of the probability vector (logits) resulting from the classifier is smaller than the critical value, ( B) classifies a case where the sigmoid value of each probability vector (logits) for positive/negative labels is smaller than a threshold value as neutral. (C) and (D) obtain an α-high-density-set of training data, calculate the distribution of the data, obtain a trustscore, and classify neutral based on this. The accuracy of (c) and (d) for classifying neutrals using confidence scores is higher than that of (a) and (b), which means that the result of the language model-based classifier reflects the reliability of the actual result. show that it is not In (D), noise was added to the learning data by the method of Equation 1 described above so that the classifier more robustly learned the distribution of the learning data, and it can be seen that the accuracy is greatly increased compared to (C). In other words, it can be confirmed that the overfitting of the model is prevented and the performance is improved when appropriate noise is added to the training data.

The text classification method according to the disclosed embodiments can be applied to various classification tasks in addition to sentiment classification to create a classifier capable of detecting out-of-domain data not included in training data. That is, the disclosed embodiments can solve various types of Open Set Recognition problems as well as sentiment classification. As mentioned above, previous studies have used the softmax value, which is the result of the model, to detect out-of-domain data, or added insufficient out-of-domain data and used similar data as a new model to supplement it. , and used a method of learning with more data. However, as can be seen in Table 1 and Table 2 below, the method proposed in the disclosed embodiments has higher accuracy than the method using softmax and is easy to learn because it does not require out-of-domain data.

Table 2 is a result of applying the text classification method according to the disclosed embodiment to the question type classification model. The training data set used to train the model consists of only questions, and consists of six question types: {who, when, what, where, why, how}. The test was configured to include question data labeled the same as learning data and sentence data, not questions collected through various sources such as articles and messengers, in a similar ratio for each label.

OOD Classification Criteria with Perturbation without Perturbation (mind) X 0.8179 0.8115 (bar) Softmax 0.8603 0.8435 (buy) Sigmoid 0.8961 0.8615 (ah) Trustscore 0.9218 0.8744

In that task, if a sentence that is not a question is classified as any of the labels learned by the model (ie, a sentence that is not a question is judged to be a question), this may be judged as incorrect inference. The accuracy of (E) is the result of unconditionally processing the model's inference about the data, not the question, as an incorrect answer. (F) and (G) assumed non-question data as out-of-domain (OOD) data and detected non-question data based on the softmax value and sigmoid value, respectively. (H) is the result of preventing the model from inferring data that is not a question with a trustscore, and processing out-of-domain data that has not been classified by the model as the correct answer. As in Table 1, even in the corresponding task, when the out-of-domain data classification criterion was set as the confidence score, performance was improved compared to when the classification criterion was set as the softmax and sigmoid values.

In addition, it can be confirmed that the difference in performance according to the presence or absence of perturbation in (E), (F), and (G) is smaller than the difference in performance shown in (H), which is the same as in the disclosed embodiment. When the adversarial training method and the confidence score were used together, it can be explained that the confidence score using the distribution of the training data is greatly improved because the distribution of the training data is better generalized and learned.

7 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, computing device 12 may be text classification device 100 described above.

Computing device 12 includes at least one processor 14 , a computer readable storage medium 16 and a communication bus 18 . Processor 14 may cause computing device 12 to operate according to the above-mentioned example embodiments. For example, processor 14 may execute one or more programs stored on computer readable storage medium 16 . The one or more programs may include one or more computer-executable instructions, which when executed by processor 14 are configured to cause computing device 12 to perform operations in accordance with an illustrative embodiment. It can be.

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. Program 20 stored on computer readable storage medium 16 includes a set of instructions executable by processor 14 . In one embodiment, computer readable storage medium 16 includes memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that can be accessed by computing device 12 and store desired information, or any suitable combination thereof.

Communications bus 18 interconnects various other components of computing device 12, including processor 14 and computer-readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . An input/output interface 22 and a network communication interface 26 are connected to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output devices 24 include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or a photographing device. input devices, and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included inside the computing device 12 as a component constituting the computing device 12, or may be connected to the computing device 12 as a separate device distinct from the computing device 12. may be

Although the present invention has been described in detail through representative examples above, those skilled in the art can make various modifications to the above-described embodiments without departing from the scope of the present invention. will understand Therefore, the scope of the present invention should not be limited to the described embodiments and should not be defined, and should be defined by not only the claims to be described later, but also those equivalent to these claims.

100: text classification device
102: first classifier
104: second classifier
202: language model
204: classifier
206: input token
208: token embedding
210: hidden state vector
212: [CLS] token's hidden state vector
214: probability vector
302: low-dimensional manifold

Claims (18)

a first classifier that determines whether an input sentence to be classified is a positive sentence or a negative sentence based on a language model learned by training data including a plurality of sentences labeled as positive or negative; and
A second classifier for reclassifying at least a part of the target sentence classified as positive or negative by the first classifier into a neutral sentence based on a trustscore of a classification result of the first classifier, Text classification device. The method of claim 1,
The first classifier,
A text classification apparatus for performing learning on the training data using an objective function set to include normalized noise in the training data. The method of claim 2,
The objective function includes a preset noise function,
The noise function is set to have a maximum function value when the distance between the noise-containing training data and the original training data is within a preset range. The method of claim 1,
The first classifier,
Using a hidden state vector generated by inputting the [CLS] token included in the target sentence into a pre-trained language model, whether the sentence to be classified is a positive sentence or a negative sentence A text classification device that determines whether a text is recognized or not. The method of claim 4,
The second classifier,
The confidence score based on the distance between the target sentence on the manifold in which the training data is embedded and the class predicted by the first classifier, and the distance between the target sentence and the closest class excluding the predicted class. A text classification device that computes . The method of claim 5,
The second classifier,
A plurality of hidden state vectors are generated by inputting [CLS] tokens included in each of the training data into the language model, and the rest of the hidden state vectors except for a certain ratio having a low distribution density are embedded on the manifold to obtain each A text classification device that creates a set of classes. The method of claim 6,
The second classifier,
Before embedding the generated plurality of hidden state vectors on the manifold, a dimension of the plurality of hidden state vectors is reduced. The method of claim 5,
The second classifier,
and re-classifying the target sentence as a neutral sentence when the confidence score is equal to or less than a preset first threshold. The method of claim 8,
The second classifier,
Text classification apparatus for re-classifying the target sentence as a neutral sentence when all distances between the target sentence and each class on the manifold in which the learning data is embedded are equal to or greater than a preset second threshold value. As a method performed on a computer,
a first classification step of determining whether an input sentence to be classified is a positive sentence or a negative sentence, based on a language model learned from training data including a plurality of sentences labeled as positive or negative; and
And a second classification step of reclassifying at least a part of the target sentence classified as positive or negative by the first classifier into neutral sentences based on a trustscore of a classification result of the first classifier. , a text classification method. The method of claim 10,
In the first classification step,
A text classification method for performing learning on the training data using an objective function set to include normalized noise in the training data. The method of claim 11,
The objective function includes a preset noise function,
The noise function is set to have a maximum function value when the distance between the noise-containing training data and the original training data is within a preset range. The method of claim 10,
In the first classification step,
Using a hidden state vector generated by inputting the [CLS] token included in the target sentence into a pre-trained language model, whether the sentence to be classified is a positive sentence or a negative sentence A text classification method that determines whether or not it is recognized. The method of claim 13,
The second classification step,
The confidence score based on the distance between the target sentence on the manifold in which the training data is embedded and the class predicted by the first classifier, and the distance between the target sentence and the closest class excluding the predicted class. A text classification method that computes . The method of claim 14,
The second classification step,
A plurality of hidden state vectors are generated by inputting [CLS] tokens included in each of the training data into the language model, and the rest of the hidden state vectors except for a certain ratio having a low distribution density are embedded on the manifold to obtain each A text classification method that creates a set of classes. The method of claim 15
The second classification step is
The text classification method of claim 1, further comprising reducing dimensions of the plurality of hidden state vectors before embedding the generated plurality of hidden state vectors on the manifold. The method of claim 14,
The second classification step,
and re-classifying the target sentence as a neutral sentence when the confidence score is equal to or less than a preset first threshold. The method of claim 17
The second classification step,
The text classification method of reclassifying the target sentence as a neutral sentence when all distances between the target sentence and each class on the manifold in which the training data is embedded are equal to or greater than a preset second threshold value.

Images