KR20220091035A - Summary evaluation device, control method thereof and summary evaluation program - Google Patents

Abstract

The present invention relates to an apparatus, a control method and a computer program for evaluating a predictive summary for a source document to determine whether the predictive summary for a source document properly summarizes the contents of the source document. More specifically, the present invention provides a first comparison result for comparing the source document and the prediction summary, obtains a second comparison result for comparing the correct answer summary with the prediction summary, and combining the first and second comparison results Thus, it is characterized in that the evaluation result for the prediction summary is obtained.

Description

SUMMARY EVALUATION DEVICE, CONTROL METHOD THEREOF AND SUMMARY EVALUATION PROGRAM

The present invention relates to techniques for evaluating summaries of articles, documents, and the like. More specifically, the present invention relates to a technique for performing evaluation processing of summary automatically performed through machine learning.

In recent years, with the development of information technology, an automatic document summarization technique for automatically summarizing articles or documents using a computer has become widespread. As we move into the digital age centered on electronic documents, the amount of digitized information is explosively increasing, and the importance of such automatic document summarization technology is being emphasized as a way to solve the information overload problem.

As automatic document summarization techniques become important, the correct evaluation of summaries created using various automated document summarization techniques is also becoming important.

Automatic document summarization technology can be divided into extractive summarization and abstract summarization. Extraction summary is a method of composing a summary by extracting important key sentences from within the document as it is, and the summary is composed of words appearing in the document. Abstract summary is a method in which the machine learning model directly generates a summary sentence that can reflect the contents of the document.

ROUGE (Recall Oriented Understudy of Gisting Evaluation) is commonly used as a method for automatically evaluating summary texts. The ROUGE method is a method in which an expert creates an answer summary and compares how similar the automatically generated summary is to the correct answer summary. In the ROUGE method, the performance is evaluated based on the n-gram overlap between the summary sentence automatically generated by the model and the correct answer summary sentence.

Since the existing evaluation method as described above is an n-gram overlap-based evaluation method, there is a problem that the evaluation reliability for the abstract summary method is low because it cannot reflect semantic information.

Accordingly, there is a need for research on a technology that can evaluate the summary by reflecting the semantic information on the summary.

SUMMARY OF THE INVENTION An object of the present invention is to provide a summary evaluation apparatus, a control method, and a program that have high evaluation reliability and are optimized for Korean.

Another object to be solved by the present invention is to provide an apparatus, method and program capable of evaluating a summary text through a comparison in consideration of semantic information between a source document and an automatically generated summary text.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

According to an aspect of the present invention to solve the above or other problems, the method comprising: obtaining a first comparison result of comparing a source document and a prediction summary; obtaining a second comparison result of comparing the correct answer summary and the prediction summary; and obtaining an evaluation result for the prediction summary by combining the first and second comparison results.

obtaining a first sentence embedding vector based on the source document; and obtaining a second sentence embedding vector based on the prediction summary, wherein the first comparison result may be a first degree of similarity between the first and second sentence embedding vectors.

The method may further include obtaining a third sentence embedding vector based on the correct answer summary, wherein the second comparison result may be a second degree of similarity between the second and third sentence embedding vectors.

The first to third sentence embedding vectors may be embedding vectors including the meaning of the sentence.

The first to third sentence embedding vectors may be embedding vectors obtained based on a Sentence Bidirectional Encoder Representations (SBERT) model.

The first and second similarities may be cosine similarity.

According to another aspect of the present invention to solve the above or other problems, a memory for storing instructions; and a processor configured to execute the stored instructions, wherein the processor is configured to: obtain a first comparison result of comparing a source document and a prediction summary, and obtain a second comparison result of comparing the correct answer summary with the prediction summary; and combining the first and second comparison results to obtain an evaluation result for the prediction summary sentence.

The processor obtains a first sentence embedding vector based on the source document, obtains a second sentence embedding vector based on the prediction summary, and the first comparison result is a second sentence between the first and second sentence embedding vectors. 1 may be similarity.

The processor may obtain a third sentence embedding vector based on the correct answer summary, and the second comparison result may be a second degree of similarity between the second and third sentence embedding vectors.

The first to third sentence embedding vectors may be embedding vectors including the meaning of the sentence.

The first to third sentence embedding vectors may be embedding vectors obtained based on a Sentence Bidirectional Encoder Representations (SBERT) model.

The first and second similarities may be cosine similarity.

The effect of the summary evaluation apparatus according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, there is an advantage in that it is possible to perform evaluation with high reliability on the summary generated by the automatic document summarization technique.

In addition, according to at least one of the embodiments of the present invention, there is an advantage that the evaluation of the summary can be provided through semantic comparison between the source document and the abstract.

Further scope of applicability of the present invention will become apparent from the following detailed description. However, it should be understood that the detailed description and specific embodiments such as preferred embodiments of the present invention are given by way of illustration only, since various changes and modifications within the spirit and scope of the present invention may be clearly understood by those skilled in the art.

1 is a block diagram of a summary evaluation apparatus 100 according to an embodiment of the present invention.
2 is a control flowchart of the summary evaluation apparatus 100 according to an embodiment of the present invention.
3 is a detailed block diagram of the first and second comparison units 121 and 122 according to an embodiment of the present invention.
4 is a diagram illustrating a configuration of a summary evaluation apparatus 100 according to an exemplary embodiment.
5 is a diagram illustrating experimental results for various summary models in order to verify an evaluation index according to an embodiment of the present invention.
6 is a diagram illustrating a comparison result between an evaluation index (RDASS) and an existing evaluation index (ROUGE) according to an embodiment of the present invention.
7 shows data comparing the efficacy of language models (P-BERT, FWA-BERT) with the application of other language models (MUSE) according to an embodiment of the present invention.
8 is a table illustrating a correlation between an evaluation index and another evaluation index according to an embodiment of the present invention.
9 and 10 are qualitative analysis results of an experiment to prove the efficacy according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

1 is a block diagram of a summary evaluation apparatus 100 according to an embodiment of the present invention.

The summary evaluation apparatus 100 according to an embodiment of the present invention may be configured to include a first comparison unit 121 , a second comparison unit 122 , and an evaluation result acquisition unit 123 .

The summary evaluation apparatus 100 is an apparatus for evaluating the prediction summary sentence 102 , and a prediction summary sentence based on the source document 101 , the prediction summary sentence 102 , and the correct answer summary sentence 103 input to the summary evaluation apparatus 100 . (102) judges how well the contents of the source document 101 are summarized and outputs the evaluation result 102. In this case, the evaluation result 102 may be a result of quantifying how well the prediction summary sentence 102 summarizes the contents of the source document 101, for example, it may be a value close to 1 expressed as a number between 0 and 1 It is a result of better reflecting the contents of the source document 101, and conversely, a value closer to 0 may be a result of reflecting incorrectly.

2 is a control flowchart of the summary evaluation apparatus 100 according to an embodiment of the present invention. 3 is a detailed block diagram of the first and second comparison units 121 and 122 according to an embodiment of the present invention. Hereinafter, it will be described with reference to FIGS. 2 and 3 together.

를 획득(S201-1)한다. 이는, 문장 간의 의미 정보를 고려하기 위하여, 소스 문서(101)를 고정된 차원의 벡터로 변환하는 작업이다. 이때 고정된 차원의 벡터는 문장의 의미 정보를 반영할 수 있으며, 문장 임베딩 벡터라 부른다.When the source document 101 is input, the first sentence embedding vector calculating unit 301-1 is configured to calculate the first sentence embedding vector.

to obtain (S201-1). This is an operation of transforming the source document 101 into a fixed-dimensional vector in order to consider semantic information between sentences. In this case, the vector of a fixed dimension may reflect the semantic information of the sentence, and is called a sentence embedding vector.

As a representative method of obtaining the sentence embedding vector as described above, there is a method of obtaining the word embedding value (vector) of each word constituting the sentence using GLOVE or BERT, and averaging these values.

In one embodiment of the present invention, in order to compare semantic information between sentences, it is proposed to utilize Sentence-BERT (hereinafter referred to as SBERT), which is a language model pre-learning a large-scale dataset segmented by a BPE (Byte Pair Encoding) algorithm. The BPE algorithm is a sub-word unit segmentation method used as a preprocessor in recent natural language processing algorithms such as NMT and BERT.

SBERT refers to a model designed to better extract the semantic meaning of sentences than BERT by modifying the BERT model structure and performing fine-tuning in the NLI (Natural language inference) dataset.

는, 아래와 같은 수학식 1 및 2로 획득할 수 있다.Sentence Embedding Vector Using SBERT

can be obtained by Equations 1 and 2 below.

w ₁ , ..., w _n represents each of the words constituting a sentence, [CLS] is a tag added to the front of the first sentence, and [SEP] is a tag that separates between sentences or is added to the end of a sentence to be.

e _cls , e ₁ , ..., e _n , e _sep are word embedding vectors for each of the [CLS], w ₁ , ..., w _n and [SEP] tags. For example, e ₁ is the word embedding vector for the word w ₁ . e ₁ may reflect semantic information of the word w ₁ as well as semantic information about other words ( w ₂ , ..., w _n ) included in the sentence.

를 획득할 수 있다.Thereafter, the first sentence embedding vector calculating unit 301-1 applies mean pooling to the word embedding vectors e ₁ , ..., e _n to obtain the source document 101 as shown in Equation 2 below. first sentence embedding vector

can be obtained.

Such an average pooling method is only an example, and it will be apparent that other pooling methods may be applied.

Here, j is a dimension index value of embedding, and n is the number of words constituting a sentence (source document 101).

를 획득(S201-2)할 수 있다. 그리고 제 3 문장 임베딩 벡터 산출부(301-3)는 정답 요약문(103)이 입력되면, 제 3 문장 임베딩 벡터

를 획득(S201-3)할 수 있다.Through the same process, when the prediction summary sentence 102 is input, the second sentence embedding vector calculating unit 301-2 calculates the second sentence embedding vector

can be obtained (S201-2). In addition, the third sentence embedding vector calculating unit 301-3 receives the third sentence embedding vector when the correct answer summary 103 is input.

can be obtained (S201-3).

및 제 2 문장 임베딩 벡터

에 기초하여 제 1 유사도를 산출(S202-1)할 수 있다. 제 1 유사도 및 이하에서 후술되는 제 2 유사도는, 비교 대상인 두 벡터가 벡터 공간 상에서 얼마나 가까운지 여부를 나타낸다.Then, the first similarity calculator 302-1 generates the first sentence embedding vector.

and a second sentence embedding vector

A first degree of similarity may be calculated ( S202 - 1 ) based on . The first degree of similarity and the second degree of similarity, which will be described below, indicate how close two vectors to be compared are in a vector space.

The first and second similarities according to an embodiment of the present invention are calculated as cosine similarity. The calculation of the first degree of similarity based on the cosine similarity may be performed by the following Equation (3).

는 제 1 유사도(303-1)를 의미할 것이다.here

will mean the first similarity 303 - 1 .

와 예측 요약문(102)의 의미 정보를 담고 있는 제 2 문장 임베딩 벡터

간의 유사도이기 때문이다.That is, the first degree of similarity may indicate whether the source document 101 and the prediction summary 102 are semantically similar. Because the first sentence embedding vector containing the semantic information of the source document 101

and the second sentence embedding vector containing semantic information of the prediction summary sentence 102

This is because the similarity between

와 제 3 문장 임베딩 벡터

에 기초하여 제 2 유사도를 산출(S202-2)할 수 있다.Similarly, the second similarity calculating unit 302-2 then performs the second sentence embedding vector

and 3rd sentence embedding vector

A second degree of similarity may be calculated on the basis of ( S202 - 2 ).

The calculation of the second degree of similarity based on the cosine similarity may be performed by the following Equation (4).

는 제 2 유사도(303-2)를 의미할 것이다.here

will mean the second degree of similarity 303 - 2 .

The calculated first and second similarities 303 - 1 and 303 - 2 are transmitted to the evaluation result obtaining unit 123 . The evaluation result obtaining unit 123 obtains a final evaluation result by combining the first and second similarities 303-1 and 303-2 (S203), and outputs the obtained evaluation result (S204).

The combination of the first and second similarities 303 - 1 and 303 - 2 performed by the evaluation result obtaining unit 123 may be performed as in Equation 5 below.

Here, RDASS is the final output result of the evaluation procedure performed according to an embodiment of the present invention, and is an evaluation index for the prediction summary.

That is, according to the evaluation index RDASS according to the embodiment of the present invention, in evaluating the prediction summary sentence 102, the prediction summary sentence 102 and the correct answer summary sentence 103 can be compared in consideration of the semantic information of each, as well as the prediction Since the summary sentence 102 and the semantic information of the source body 101 are evaluated in consideration of the evaluation, the advantage that higher evaluation reliability can be expected can be expected.

Furthermore, in the present invention, it is proposed to perform fine-tuning on the SBERT model.

SBERT may be further trained to be more suitable for the source document 101 and the correct answer summary 103 as a learnable model. That is, as in the present invention, we propose a fine-tuning method for SBERT that is optimized for the abstract summary model.

Most neural approaches for abstraction abstraction are based on encoder-decoder architectures. Formally, given the document D = [w ₁ , ..., w _k ], the goal is a summary from the hidden expression h _p = [h ₁ , ..., h _n ] y _p = [w ₁ , ... , w _n ]. The hidden expression h _p = [h ₁ , ..., h _n ] is the output vector of the decoder, the embedding vector for the prediction summary 102 .

In one embodiment of the present invention, a hidden representation of the decoder is utilized to fine-tune the SBERT.

In particular, in one embodiment of the present invention, it is proposed to use a triplet objective function to fine-tune the SBERT.

Given the anchor h _p , the positive reference expression v _pr , the negative expression v _nr , and the Euclidean distance d, the triple objective function J(p, r) for the prediction summary (102) and the correct answer summary (103) is can be defined as

Here, ε is a margin that guarantees that h _p is closer to v _pr than v _nr , and ε is set to 1 in an embodiment of the present invention. Similarly, the triple objective function J(p, d) between the source document 101 and the prediction summary 102 may be defined as in Equation 7 below.

That is, the micro-precision for SBERT consists in minimizing the triple objective functions J(p, r) and J(p, d) as shown in Equation 8 below.

The objective function J of SBERT in Equation 8 is optimized for abstract summary. In general, a negative log-likelihood (NLL) goal between the prediction summary 102 and the correct answer summary 103 is used for the abstract summary. This fine-tuned SBERT is hereinafter referred to as FWA-SBERT.

4 is a diagram illustrating a configuration of a summary evaluation apparatus 100 according to an exemplary embodiment.

Referring to FIG. 4 , the summary evaluation apparatus 100 includes a memory 192 and a processor 191 . Memory 192 stores one or more instructions executable by processor 191 . Processor 191 executes one or more instructions stored in memory 192 . The processor 191 may execute one or more operations described above with respect to FIGS. 1-3 by executing instructions. The processor 191 may evaluate the prediction summary 102 according to the instruction, and may output an evaluation result.

So far, embodiments of the present invention have been described. Hereinafter, the effects of the above-described embodiments will be described based on specific experimental results.

* Preparation of experiments

In the News tab of Daum, a Korean portal site, a model was trained and evaluated using a data set consisting of 10 topics including politics, economy, international, culture, and information technology. About 3 million news articles were extracted from this. The number of articles for learning, validation and testing were 2.98M, 0.01M, and 0.01M, respectively. We call this data set the "Daum/News data set".

Through the Daum/News data set, the contents of the article were fully understood and an appropriate evaluation was conducted. This data set contains articles from 143 newspapers, each with a different summary style, and an experiment was performed to confirm the effectiveness of the method proposed in the present invention using them.

In order to utilize SBERT, in the present invention, bert-base-uncased (BERT) was first pre-learned on a Korean data set consisting of 23 million sentences and 1.6 million documents including wikis, Sejong corpus, and web documents. Next, the SBERT was trained with the classification and regression goals of the NLI and semantic textual similarity (STS) benchmarks (STSb), respectively.

Since the NLI and STSb data sets are in English, the Korean NLI and STS data sets translated by Kakao Machine Translator were further utilized. An evaluation was performed on the STS benchmark test data set to present an 80.52 Spearman rank correlation result.

After that, the pre-trained SBERT model was fine-tuned with the abstract summary model to capture contextual information of the correct answer summary (103) and the source document (101) along with the prediction summary (102).

In order to demonstrate the effectiveness in the present invention, several raters were asked to evaluate relevance, consistency, and flexibility. Relevance indicates the degree of relevance of the document, consistency indicates the degree of realism, and flexibility indicates the degree of quality of the generated summary.

Given a source document 101, a predictive summary 102, and a correct answer summary 103, each rater evaluates the assessment in such a way that it scores on a scale of 1 to 5 for an assessment indicator (i.e., relevance, consistency, flexibility). was done

A total of six people conducted the evaluation, three with doctoral degrees and three with master's degrees majoring in computer science. Averaging over 200 sample summaries drawn from the Daum/News data set, the score was 3.8 for relevance, 3.6 for consistency, and 3.9 for flexibility.

In the following, the effect is demonstrated through the method of comparison with the ROUGE evaluation method.

5 is a diagram illustrating experimental results for various summary models in order to verify an evaluation index according to an embodiment of the present invention.

In the experimental result shown in FIG. 5 , it is based on a baseline method in which the correct answer summary sentence 103 (Reference Summary) is set as the upper limit criterion (1.00) and the results for the rest are compared.

In the experimental results shown, the Lead-1 model is a model that selects the first sentence from the source document 101 and sets it as the prediction summary 102, and the Lead-3 model selects the first to third sentences of the source document 101. It refers to a model that sets a sentence as a predictive summary sentence 102 .

Reporters tend to use implicit words when summarizing documents, so the value of s(p, d) (similarity between source document 101 and predictive summary 102) is relatively low compared to the upper limit criterion. However, since the score of s(p, r) is 1.00, the highest RDASS score can be found in the correct answer summary (103, Reference Summary).

It can be confirmed that in the case of Lead-1, s(p, r) shows higher performance than s(p, d), and in the case of Lead-3, s(p, d) shows higher performance than s(p, r). . The reason for this performance is that, since Lead-3 contains more sentences in the document, s(p, r), which is similar to the correct answer summary sentence 103, is low, but s(p, r), which is similar to the source document 101, is low. , d) increases.

In the case of the ROUGE performance of the upper limit criterion, it can be seen that the performance is relatively low compared to other studies performed on the English data set. The reason is that in the case of Korean, the same meaning is expressed differently due to the linguistic characteristics of an agglutinative language.

However, it can be seen that the RDASS score of the upper limit is similar to the score of the correct answer summary (103). This confirms that the proposed evaluation method can reflect and document the semantic meaning of the reference summary well.

In the case of 'BERTSUMABS', although it has a higher similarity with the correct answer summary sentence (103) than the upper limit criterion, the sentence is not extracted from the document as the lead baseline because it is based on a generative model. As a result, it shows a relatively low s(p,d) score.

6 is a diagram illustrating a comparison result between an evaluation index (RDASS) and an existing evaluation index (ROUGE) according to an embodiment of the present invention. The comparison was made by analyzing the correlation between each evaluation index and the results by the evaluators. That is, if the result according to the evaluation index is similar to the index by the evaluator, it can be recognized as a better evaluation index, so the correlation with the evaluation result by the evaluator is analyzed.

Figure 6 (a) is a comparison according to the Pearson (Pearson) correlation, Figure 6 (b) shows a comparison result according to the Kendall rank (Kendall rank) correlation.

Referring to FIG. 6 (a), the left side shows the Pearson correlation between the evaluation result by the evaluator and ROUGE, and the right side shows the Pearson correlation between the evaluation result by the rater and FWA-SBERT (the present invention). It can be seen that the FWA-SBERT is much higher than the ROUGE score.

Referring to FIG. 6 (b), the left side shows the evaluation result by the evaluator and the Kendall rank with ROUGE, and the right side shows the evaluation result by the evaluator and the Kendall rank with FWA-SBERT (the present invention). Similarly, in the Kendall ranking, it can be seen that the FWA-SBERT is much higher than the ROUGE score.

According to the graphs shown in FIGS. 6 (a) and (b), it can be confirmed that RDASS showed a higher correlation with the judgment of the evaluator than ROUGE.

Therefore, according to the experimental results, it is proved that the summary evaluation technique according to the embodiment of the present invention can be evaluated by reflecting semantic information more than ROUGE based on n-gram overlap.

Hereinafter, the effect of the present invention will be described through comparison with the case where other language models are applied.

7 shows data comparing the efficacy of language models (P-BERT, FWA-BERT) with the application of other language models (MUSE) according to an embodiment of the present invention.

The experiment is explained through the comparison of three language models: MUSE (Multilingual Universal Sentence Encoder), P-BERT (Pre-trained SBERT), and FWA-SBERT.

MUSE is a multilingual sentence encoder that uses multi-task learning to insert texts from 16 languages into a single semantic space. The model has been trained on over a billion question-answer pairs and is known to show competitive state-of-the-art results for semantics, double-text searches, and search queries.

P-BERT is a language model according to an embodiment of the present invention, and is a model using only pre-trained SBERT without fine tuning.

FWA-SBERT is a SBERT model with fine adjustments made according to an embodiment of the present invention.

Referring to the illustrated figure, it can be confirmed that P-SBERT shows a higher correlation with the evaluation result of the evaluator than MUSE. Furthermore, it can be confirmed that overall, when FWA-SBERT was used, it showed the highest correlation with the judge's judgment.

That is, in FIG. 7, it can be confirmed that the RDASS, an evaluation index according to an embodiment of the present invention, is closer to the evaluation of the evaluator than when other evaluation indicators are used, and in particular, it is proved that higher efficacy can be obtained by fine adjustment can do.

8 is a table illustrating a correlation between an evaluation index and another evaluation index according to an embodiment of the present invention.

Referring to FIG. 8 , it can be confirmed that a high correlation exists between the ROUGE metrics. For example, a high correlation of 0.84 between ROUGE-1 and ROUGE-2 and a high correlation of 0.99 between ROUGE-1 and ROUGE-L are confirmed.

,

및 RDASS) 사이에는 상대적으로 낮은 상관 관계가 있음을 확인할 수 있다. 즉, 본 발명의 실시예에 따른 평가 지표들이 ROUGE가 할 수 없었던 의미정보를 반영하였기 때문이라고 해석할 수 있을 것이다.However, the ROUGE metric and the evaluation index according to the present invention (

,

and RDASS), it can be seen that there is a relatively low correlation. That is, it can be interpreted that the evaluation indicators according to the embodiment of the present invention reflect semantic information that ROUGE could not do.

9 and 10 are qualitative analysis results of an experiment to prove the efficacy according to an embodiment of the present invention.

Referring to FIG. 9 , the source document 101 , the prediction summary sentence 102 , the correct answer summary sentence 103 , the RDASS result 701 , the ROUGE result 702 , and the evaluation by the evaluator 703 are output. It can be seen that the summary of the correct answer (103), "Messi's 30th birthday is with his wife and son", and the prediction summary (102) "Messi's 30th birthday, spend closely with his family" contain substantially the same content. have. Therefore, it can be confirmed that the evaluation result 703 by the evaluator is high.

However, it can be confirmed that the ROUGE evaluation result 702 is not very high. This is because the expressions 'family' and 'wife and son' were perceived differently because semantic information was not considered.

It can be seen that the RDASS result 701 according to the embodiment of the present invention in consideration of semantic information is a high score (0.81), unlike the ROUGE evaluation result 702 . Unlike ROUGE, this can prove that semantic information was sufficiently considered and evaluated.

Similarly, referring to FIG. 10, the answer summary (103) "Samsung Electronics launches qled tv in Brazil, the largest market in Latin America" and the prediction summary 102 "Samsung Electronics launches a new 'gled tv' in Brazil" are practically almost You can see that they contain the same content. As a result, it can be confirmed that the evaluation result 703 by the evaluator is high.

However, it can be confirmed that the ROUGE evaluation result 702 is not very high. This is because the expressions 'launch' and 'launch of new products' were perceived differently because semantic information was not considered.

It can be seen that the RDASS result 701 according to the embodiment of the present invention in consideration of semantic information has a high score (0.71), unlike the ROUGE evaluation result 702 . Unlike ROUGE, this can prove that semantic information was sufficiently considered and evaluated.

According to the above experimental results, it can be confirmed that the ROUGE evaluation index, which is widely used when evaluating Korean summaries, does not reflect semantic information. Korean is a language with various expressions, and prediction summaries of various expressions having the same meaning as the correct answer summary sentences may exist. Therefore, high accuracy cannot be expected just by using the ROUGE metric.

Therefore, in the embodiment of the present invention, a new evaluation index, RDASS, is proposed to solve this problem. It was confirmed that RDASS can reflect a deep semantic relationship between the prediction summary and the source document. Through extensive evaluation, it was demonstrated that the correlation with human judgment is higher in the evaluation index (RDASS) according to the embodiment of the present invention than the ROUGE evaluation result.

Although the embodiments of the apparatus, control method, and computer program for evaluating the prediction summary according to the present invention have been described above, these are described as at least one embodiment, and thereby the technical spirit of the present invention and its configuration and operation are not limited. is not, and the scope of the technical idea of the present invention is not limited / limited by the drawings or the description with reference to the drawings. In addition, the concepts and embodiments of the present invention presented in the present invention can be used by those of ordinary skill in the art as a basis for modifying or designing other structures in order to perform the same purpose of the present invention. , an equivalent structure modified or changed by a person of ordinary skill in the art to which the present invention belongs is bound by the technical scope of the present invention described in the claims, and does not depart from the spirit or scope of the invention described in the claims Various changes, substitutions and changes are possible within the limits.

Claims (13)

In the control method of the summary evaluation device,
obtaining a first comparison result of comparing the source document and the prediction summary;
obtaining a second comparison result of comparing the correct answer summary and the prediction summary; and
Combining the first and second comparison results to obtain an evaluation result for the prediction summary,
Control method of summary evaluation device. The method of claim 1,
obtaining a first sentence embedding vector based on the source document; and
Further comprising the step of obtaining a second sentence embedding vector based on the prediction summary,
The first comparison result is a first degree of similarity between the first and second sentence embedding vectors,
Control method of summary evaluation device. 3. The method of claim 2,
Further comprising the step of obtaining a third sentence embedding vector based on the correct answer summary,
The second comparison result is a second degree of similarity between the second and third sentence embedding vectors,
Control method of summary evaluation device. The method of claim 3, wherein the first to third sentence embedding vectors are
Characterized in that it is an embedding vector containing the meaning of the sentence,
Control method of summary evaluation device. The method of claim 4, wherein the first to third sentence embedding vectors are:
Characterized in that it is an embedding vector obtained based on a SBERT (Sentence Bidirectional Encoder Representations) model,
Control method of summary evaluation device. The method of claim 3, wherein the first and second similarities are:
Characterized in that the cosine similarity (cosine similarity),
Control method of summary evaluation device. In the summary evaluation device,
a memory storing instructions; and
a processor configured to execute the stored instructions, the processor comprising:
obtain a first comparison result comparing the source document and the forecast summary;
obtaining a second comparison result of comparing the correct answer summary and the prediction summary;
Combining the first and second comparison results to obtain an evaluation result for the prediction summary,
Summary evaluation device. 8. The method of claim 7, wherein the processor
obtaining a first sentence embedding vector based on the source document,
obtaining a second sentence embedding vector based on the prediction summary;
The first comparison result is a first degree of similarity between the first and second sentence embedding vectors,
Summary evaluation device. 9. The method of claim 8, wherein the processor is
obtaining a third sentence embedding vector based on the correct answer summary,
The second comparison result is a second degree of similarity between the second and third sentence embedding vectors,
Summary evaluation device. 10. The method of claim 9, wherein the first to third sentence embedding vectors,
Characterized in that it is an embedding vector containing the meaning of the sentence,
Summary evaluation device. 11. The method of claim 10, wherein the first to third sentence embedding vectors,
Characterized in that it is an embedding vector obtained based on a SBERT (Sentence Bidirectional Encoder Representations) model,
Summary evaluation device. 10. The method of claim 9, wherein the first and the second degree of similarity,
Characterized in that the cosine similarity (cosine similarity),
Summary evaluation device. A computer program stored in a medium for executing the method of any one of claims 1 to 6 in combination with hardware.

Images