KR101882906B1 - Device and method to generate abstractive summaries from large multi-paragraph texts, recording medium for performing the method - Google Patents

Abstract

An apparatus for generating an abstract summary of a plurality of paragraph texts includes an input unit for automatically classifying a document into paragraphs and transmitting the same; Converts a paragraph transferred from the input unit into an internal representation vector, and generates a representation of the transformed vector through a regression neural network including multiple timescales gated recurrent units (GRUs) having multiple time constants An encoding processing unit; A decoding processing unit for decoding the internal representation received from the encoding processing unit through a regression neural network including the MTGRU and generating sentences using language modeling; And an output unit for collecting the summary output of each paragraph and outputting the final abstract summary. Accordingly, by generating an abstract expression, it is possible to generate a summary close to a man-made summary.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for generating an abstract summary of a plurality of paragraphs of text, and a recording medium for performing the method. BACKGROUND ART < RTI ID = 0.0 >

The present invention relates to an apparatus and method for generating an abstract summary of a multi-paragraph text, and a recording medium for performing the method. More particularly, the present invention relates to an automatic summary system for paragraphs of text having a plurality of paragraphs, And abstract abstractions such as those written by people other than simple word extraction.

Summarization has been extensively studied over the last few decades. In summary, summary methods can be categorized into an extraction approach and an abstract approach, depending on the type of calculation task. While abstracted summaries are a matter of choice, abstract summaries require a deeper understanding of the semantics and discourse of the text and the process of creating a new text. In the past, focus has been focused on extraction summaries, and abstract summaries still have a number of challenges that need to be addressed.

Recently, sequence-to-sequence (SEQ2SEQ) repetitive neural networks (RNNs) have been widely applied in many tasks. Such an RNN encoder-decoder (Cho et al., 2014; Bahdanau et al., 2014) combines an expression learning encoder and a language modeling decoder to perform a mapping between two sequences.

Similarly, recent research suggested casting a summary as a mapping problem between the input sequence and the summary sequence. Rush et al. (2015); Nallapati et al. (2016) showed that the RNN encoder-decoder is very good at summarizing short text. This seq2seq approach provides a complete data-centric solution for both semantic and discourse understanding and text generation.

While seq2seq provides a promising way to abstract abstracts, it is not easy to extrapolate the methodology to other tasks, such as a summary of scientific documents, and there are a number of practical and theoretical problems:

1) It is difficult to simply learn the RNN encoder-decoder for the entire article. Because the science article is too long compared to the current memory capacity of the GPU, it can not process the whole through the RNN. 2) Moving from one or two sentences to multiple sentences or paragraphs introduces additional levels of combinability and rich discourse structure. In this case, there is a problem of how to improve the RNN encoder-decoder to be able to capture better. 3) Deep learning is highly dependent on good quality, large data sets. It is difficult to collect the source summary data pairs, and the data sets are insufficient outside the news wire domain.

In summary, a number of deep learning models have been proposed recently due to the development of deep learning based artificial intelligence technology, but still some difficult problems remain. One of them is the abstractive summary in the field of natural language processing. The existing summary system has a limit to expressiveness by using the method of extracting by extracting the words existing in the original text.

KR 10-2013-0116908A KR 10-0785927 B1

 Minsoo Kim, Moirangthem Dennis Singh, and Minho Lee, "Towards Abstraction from Extraction: Multiple Timescale Gated Recurrent Unit for Summarization", Association for Computational Linguistics (2016).

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and it is an object of the present invention to provide an apparatus for generating an abstract summary of plural paragraph texts.

Another object of the present invention is to provide a method for generating an abstract summary of a multi-paragraph text.

It is still another object of the present invention to provide a recording medium on which a computer program for recording an abstract summary of a plurality of paragraph texts is recorded.

According to an embodiment of the present invention, there is provided an apparatus for generating an abstract summary of a plurality of paragraph texts, the apparatus comprising: an input unit for automatically classifying a document into paragraphs; Converts a paragraph transferred from the input unit into an internal representation vector, and generates a representation of the transformed vector through a regression neural network including multiple timescales gated recurrent units (GRUs) having multiple time constants An encoding processing unit; A decoding processing unit for decoding the internal representation received from the encoding processing unit through a regression neural network including the MTGRU and generating sentences using language modeling; And an output unit for collecting the summary output of each paragraph and outputting the final abstract summary.

In an embodiment of the present invention, the encoding processing unit may include: an inherent vectorization unit that converts a paragraph transferred from the input unit into an internal representation vector; And MTGRU, and may include a coding neural network that generates a representation of the transformed vector based on the deep learning technique and transmits the representation to the decoding processing unit.

In an embodiment of the present invention, the decoding processing unit includes a decoding neural network that includes a regression neural network composed of multiple layers of MTGRUs and decodes the internal representation received from the encoding processing unit; And a natural language generation unit for generating sentences to be expressed in a natural language from the decrypted internal representation received from the decryption neural network according to learning contents using language modeling.

In an embodiment of the present invention, the regression neural network may use a sequence-to-sequence model using a temporal hierarchical model with MTGRU as a base unit.

In the embodiment of the present invention, the MTGRU has a time constant set for each layer, and the larger the time constant, the slower the context unit can be constructed.

In an embodiment of the present invention, the time constant may increase as the layer rises.

In an embodiment of the present invention, the regression neural network may have multiple temporal-based compositionality.

In an embodiment of the invention, the input may extract introductions from a LaTeX source file.

According to another aspect of the present invention, there is provided a method for generating an abstract summary of a plurality of paragraph texts, the method comprising: automatically classifying a document into paragraphs; Converting the paragraph into an internal representation vector; Generating a transformed vector as an internal representation based on a deep learning technique through a multi-layered regression neural network of Multiple Timescales Gated Recurrent Unit (GRU) with multiple time constants; Decoding the internal representation through a multi-layer regression neural network of MTGRU; Generating sentences expressed in natural language from the decoded internal expression according to learning contents using language modeling; And collecting the summary output of each paragraph and outputting a final abstract summary.

In an embodiment of the present invention, the regression neural network may use a sequence-to-sequence model using a temporal hierarchical model with MTGRU as a base unit.

In the embodiment of the present invention, the MTGRU has a time constant set for each layer, and the larger the time constant, the slower the context unit can be constructed.

In an embodiment of the present invention, the time constant may increase as the layer rises.

In an embodiment of the present invention, the regression neural network may have multiple temporal-based compositionality.

In an embodiment of the present invention, the method of generating an abstract summary of the multi-paragraph text may further comprise extracting Introductions from the LaTeX source file.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for performing a method for generating an abstract summary of a plurality of paragraphs of text.

According to such an apparatus and method for generating an abstract summary of a multi-paragraph text, an abstract summary can be obtained by generating words necessary for summary rather than combining words existing in a paragraph from previously learned contents. Existing systems are awkward and distant from human-generated summaries because they only bring out the same effects as extraction and compression. The present invention can generate an abstract summary based on the understanding and understanding of long text by introducing an internal representation process based on the deep learning technology.

The present invention applies abstract learning to deep learning-based artificial intelligence to enable abstract summarization without adding additional storage or structural complexity. Further, the present invention can also summarize a long text composed of a plurality of paragraphs as an abstract summary rather than a simple extraction.

1 is a block diagram of an apparatus for generating an abstract summary of a multi-paragraph text according to an embodiment of the present invention.
2 is a diagram showing a gate repetition unit (GRU).
FIG. 3 is a diagram illustrating a multiple timescale gated recurrent unit (GRU) having multiple time constants proposed in the present invention.
Figure 4 shows the structure of the summary model of the present invention.
5 is a graph showing a learning speed comparison between GRU and MTGRU.
6 is a diagram showing an example of a summary generated using the MTGRU.
7 is a diagram showing an output summary and an example of an extracted target.
8 is a graph comparing learning performance between multiple time constants.
9 is a flowchart of a method for generating an abstract summary of a multi-paragraph text according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of an apparatus for generating an abstract summary of a multi-paragraph text according to an embodiment of the present invention.

An apparatus for generating abstract abstract texts (hereinafter abbreviated as apparatuses) of a plurality of paragraphs of text according to the present invention is an automatic abstract system for each paragraph of text having a plurality of paragraphs. It can understand deepening meaning, This is possible.

In order to realize such a function, a self-coded network (Auto Encoder) is used as a basic component of a regression neural network using a newly proposed MTGRU (Multiple Timescales Gated Recurrent Unit).

Referring to FIG. 1, an apparatus 10 according to the present invention includes an input unit 100, an encoding processing unit 300, a decoding processing unit 500, and an output unit 700.

A device (10) according to the present invention creates abstract abstracts from text consisting of a plurality of paragraphs. The apparatus 10 according to the present invention implements a sequence to sequence (hereinafter referred to as a seq2seq) model using a temporal hierarchical model having MTGRU (Multiple Timescale Gated Recurrent Unit) having multiple time constants as a basic element.

 The apparatus 10 of the present invention can be installed and executed by a software application for generating abstract abstract texts of a plurality of paragraphs of text and the input unit 100, the encoding processing unit 300, the decryption processing unit 500, The configuration of the output 700 may be controlled by software for generating an abstract summary of the multi-paragraph text that is executed in the device 10. [

The device 10 may be a separate terminal or some module of the terminal. The configuration of the input unit 100, the encoding processing unit 300, the decoding processing unit 500, and the output unit 700 may be an integrated module or may be composed of one or more modules. However, conversely, each configuration may be a separate module.

The device 10 may be mobile or stationary. The device 10 may be in the form of a server or an engine and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS) a wireless device, a handheld device, and the like.

The device 10 may execute or produce various software based on an operating system (OS), i.e., a system. The operating system is a system program for allowing software to use the hardware of a device. The operating system includes a mobile computer operating system such as Android OS, iOS, Windows Mobile OS, Sea OS, Symbian OS, Blackberry OS, MAC, AIX, and HP-UX.

The input unit 100 automatically classifies the document into paragraphs and transmits the same to the encoding processing unit 300. For example, the input unit 100 can create a new data set of computer science (CS) articles at ArXiv.org, and extract introductions from LaTeX source files and segment them into paragraphs.

The encoding processing unit 300 converts a paragraph transferred from the input unit 100 into an internal representation vector and converts the vector converted through the regression neural network including multiple timescales gated recurrent units (GRUs) having multiple time constants And can transmit the decoded result to the decryption processing unit 500. [

To this end, the encoding processing unit 300 includes an inherent vectorization unit 310 for converting a paragraph transferred from the input unit 100 into an internal representation vector, and a regression neural network composed of multiple layers of the MTGRU. And a coding neural network 330 for generating an internal representation of the transformed vector and delivering the transformed vector to the decoding processing unit.

In the present invention, an internal representation grammar is generated, and then an abstract expression is generated by applying a sentence generation technique to generate a summary similar to a human-created summary. The present invention can also summarize a long text composed of a plurality of paragraphs as an abstract summary rather than a simple extraction.

The present invention presents a first intermediate step for an end-to-end abstract summary of a scientific article. The goal of the present invention is to extend the seq2seq-based summary to larger text, a more complex summary task.

In order to solve the above problem, the present invention proposes a paragraph summarization system which is learned through the sentence system of the paragraph key. The term frequency-inverse document frequency (TF-IDF) (Luhn, 1958; Jones, 1972) is used to extract key sentences in each paragraph.

We also propose a new model, GRU (multiple timescale gated recurrent unit) with multiple time constants. The MTGRU adds a temporal layer component that can handle the compositionality of multiple layers. This is a concept similar to the temporal hierarchy found in the human brain and is implemented by modulating different layers of multiple RNNs to different time scales (Yamashita and Tani, 2008).

The model proposed by the present invention understands the meaning of the multi-sentence source text and knows what is important to it, which is an essential first step to an abstract summary.

In one embodiment, the present invention creates a new data set of computer science (CS) articles at ArXiv.org and extracts an introduction from a LaTeX source file. Introduction is broken down into paragraphs, each paragraph being a natural unit of discourse.

Finally, link the summary generated for each paragraph to create a non-expert summary of the article introduction, and compare the results with the actual summary. The model of the present invention shows that it is possible to summarize several sentences as the most important part of invisible data, and seq2seq mapping supports a larger summary view.

The present invention demonstrates that the MTGRU model meets some of the key requirements of the abstract summary system. Also, MTGRU can significantly reduce learning time compared to existing RNN encoder-decoder.

Hereinafter, the background related to the model of the present invention will be described, and the application of the architecture and summary newly presented in the present invention will be described in detail.

2 is a diagram showing a gate repetition unit (GRU).

The principle of compositionality defines the meaning conveyed by language expressions as a function of the syntactic combination of constituent units. That is, the meaning of a sentence is determined by how words are combined with each other. In multi-sentence text, the possibility of combining at the level of the sentence (the way the sentences are combined together) is an additional function that gives meaning to the whole text.

When dealing with large texts, to fully understand the meaning of the text, we must consider the possibility of combining the sentences and the paragraph level. The approach that has been studied in recent literature is to build a dedicated architecture in a hierarchical format to determine the level of future combinability. Li et al. (2015) and Nallapati et al. (2016) grasps the possibility of combining at different levels of text units to improve performance.

However, such a structural change to the RNN encoder-decoder has the drawback of increasing both learning time and memory usage. Thus, the present invention proposes an alternative to an architecture that can improve performance without this overhead. The present invention is inspired by the neuroscience that generates a temporal stratum naturally occurring in functional differentiation in the human brain (Meunier et al., 2010; Botvinick, 2007).

It is well known that neurons can organize themselves hierarchically with different adaptation rates to stimuli. A typical example of this phenomenon is an auditory system in which syllable level information is integrated into word level information in a longer time window in a short time window. Previous studies have applied this concept to RNNs in motion tracking (Paine and Tani, 2004) and speech recognition (Heinrich et al., 2012).

FIG. 3 is a diagram illustrating a multiple timescale gated recurrent unit (GRU) having multiple time constants proposed in the present invention.

The GRU (multiple timescale gated recurrent unit) model with multiple timescale proposed by the present invention applies the temporal layer concept to the problem of seq2seq text summarization in the RNN encoder-decoder framework. Previous studies such as Multiple Timescale Recurrent Neural Network (MTRNN) (Yamashita and Tani, 2008) used a temporal hierarchy for motion prediction.

However, MTRNN has difficulty in capturing the same problems that exist in RNNs, such as long-term dependencies, and the gradient problem tends to disappear (Hochreiter et al., 2001). The short-term long-term memory network (Hochreiter et al., 2001) uses a complex gating architecture to help learn about long-term dependence, and is more efficient than RNN in tasks with long time dependencies such as machine translation (Sutskever et al., 2014) And showed much better performance.

The Gated Recurrent Unit (Cho et al., 2014), which has proven to be comparable to LSTM (Chung et al., 2014), has a similar complex gating architecture but has less memory. The standard GRU architecture is shown in FIG.

Since the seq2seq summary contains potentially many long-distance time dependencies, the model of the present invention applies a temporal hierarchy to the GRU. The present invention adds another constant gating unit that applies a time constant to the end of the GRU to essentially adjust the mix of past and present hidden states. The reset gate r _t , the update gate z _t, and the candidate activation u _t are calculated similar to those of the original GRU, as shown in Equation 1 below.

[Equation 1]

The time constant τ is added to the activation h _t of the MTGRU as shown in Equation 2 below.

&Quot; (2) "

is used to control the time constant of each GRU cell. A large τ denotes a slow cell output, but creates a cell that focuses on the slow function of the dynamic sequence input. The MTGRU model proposed in the present invention is shown in FIG. The existing GRU is a special case of MTGRU when τ = 1, and there is no attempt to construct a hierarchy with different time constants.

Equation 3 below shows a learning algorithm derived for MTGRU according to the defined forward process and back propagation through time rules.

&Quot; (3) "

는 시간 t-1에서 셀 출력의 오차이고,

는 셀 출력의 현재 기울기이다. 서로 다른 시상수는 각 계층 마다 설정된다. 큰 τ는 느린 컨텍스트 단위를 의미하고, τ=1은 디폴트 또는 입력 시상수를 정의한다.

Is the error of the cell output at time t-1,

Is the current slope of the cell output. Different time constants are set for each layer. A large τ denotes a slow context unit, and τ = 1 defines a default or input time constant.

Based on the hypothesis that later classes should learn features that operate at slower time constants, we set a larger τ as we move up the hierarchy.

At this point, the problem is whether the word sequence analyzed by the RNN possesses information operating over a different temporal hierarchy, such as in the case of a continuous audio signal received by the human auditory system.

The present invention assumes that the hypotheses used by the RNN and the word level, clause level and sentence level combination possibilities are strong candidates. In this regard, the multiple time-constant modification facilitates the learning of functions that operate at increasingly slower time constants corresponding to subsequent levels of the constituent hierarchy by explicitly guiding each layer of the neural network.

In order to apply the newly proposed multiple time-constant model to the summary in the present invention, a new data set of academic data is constructed. As an example, the arXiv preprint server CS. Collect LaTeX source files of articles in the {CL, CV, LG, NE} domain to extract intros and abstracts. Breaks an introduction into paragraphs, and links each paragraph to the most important sentence into a target summary. These target summaries are generated using widely adopted TF-IDF scoring.

Figure 4 shows the structure of the summary model of the present invention.

Referring to FIG. 4, the constructed data set includes rich combinability and long text sequences, which increases the complexity of the summary problem. The temporal layer function has the greatest effect when a complex configuration hierarchy is present in the input data. Therefore, the concept of multiple time constants is described by Rush et al. Will play a greater role in the context of the present invention when compared to previous summarization work, such as (2015).

This model using MTGRU is learned using these paragraphs and their targets. The generated summary of each introductory is evaluated using the abstract of the collected articles. The abstract was chosen as a gold summary because it makes the abstract a good baseline because it usually contains important discourse structures such as goals, related works, methods and results.

The decryption processing unit 500 decrypts the internal representation received from the encoding processing unit 300 through the regression neural network including the MTGRU, and generates sentences using language modeling.

For this, the decoding processing unit 500 includes a regression neural network composed of multiple layers of the MTGRU, and includes a decoding neural network 510 for decoding the internal representation received from the encoding processing unit 300, and a learning content using language modeling And a natural language generation unit 530 for generating sentences expressed in a natural language from the decoded internal representation received from the decryption neural network.

The output unit 700 collects the summary output of each paragraph from the natural language generation unit 530 and outputs a final abstract summary.

The present invention has the ability to summarize long texts consisting of a plurality of paragraphs by having multiple compositionalities based on time constants. The abstract summary generated in accordance with the present invention is similar to a human-generated summary, unlike the abstracts generated by the existing extractive summary system, and has a natural grammar using the deep-run technique.

Hereinafter, in order to verify the effectiveness of the method proposed in the present invention, a comparison was made with a conventional RNN encoder-decoder in terms of learning speed and performance.

For the experiment, we studied two seq2seq models, the first model using the existing GRU in the RNN encoder decoder and the second model using the proposed MTGRU in the present invention. Both models are learned using the same hyperparameter settings in an optimal configuration for existing hardware functions.

Sutskever et al. (2014), the input is divided into several buckets. The GRU and MTSGRU models consist of four layers and 1792 hidden units, as shown in Table 1 below.

[Table 1]

Because the model uses longer input and target sequence sizes, the number of hidden unit sizes and hierarchies is limited. 512 built-in sizes were used for both networks. The time constant τ of each layer is set to 1, 1.25, 1.5, and 1.7, respectively. The model is learned for 110,000 text summary pairs. The original text is a paragraph extracted from an introduction to an academic paper, and the target is the most prominent sentence extracted from the paragraph using the TF-IDF score.

5 is a graph showing a learning speed comparison between GRU and MTGRU.

To compare the learning speed of the model, Figure 5 shows a plot of the learning curve until successive perplexity reaches 9.5. Both models are learned using the Nvidia Ge-Force GTX Titan X GPU, which takes approximately 4 and 3 days. During testing, greedy decoding is used to generate the most probable output from the original intro.

For the evaluation, we adopt Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics (Lin, 2004) proposed by Lin and Hovy (2003). ROUGE is a recall-centric measure that scores on a system summary that has been proven to be highly correlated with human assessment. Measure the n-gram recovery between the candidate summary and the gold summary.

The ROUGE score is given by Li et al. (2015). ROUGE-1, ROUGE-2, and ROUGE-L are used to report model performance. For performance evaluation, the learning perplexity of both GRU and MTGRU is learned in step 74750, which is shown in Table 2 for both models.

[Table 2]

At this stage we have chosen the early stopping point, so we get the lowest test perplexity of the GRU model. The ROUGE scores calculated using these learned networks are shown in Table 3 and Table 4 for the GRU and MTGRU models, respectively.

[Table 3]

[Table 4]

A sample summary generated by the MTGRU model of the present invention is shown in FIG.

6 is a diagram showing an example of a summary generated using the MTGRU. 7 is a diagram showing an output summary and an example of an extracted target. 8 is a graph comparing learning performance between multiple time constants.

Referring to FIGS. 5-7, the ROUGE score obtained for the summary model using GRU and MTGRU shows that the multiple time-constant concept improves the performance of existing seq2seq models without a highly complex architecture layer.

Another big advantage is the learning speed improvement of one epoch. Moreover, Figure 6 shows that the model successfully generalized the difficult task of summarizing large paragraphs into a one-line summary.

Depending on the parameter of the time constant τ, it is explained as follows (Yamashita and Tani, 2008). As we move up the hierarchy so that the upper layer has slower context units, τ gradually increases.

Also, as shown in FIG. 8, the multiple setting of? Is experimented and the learning performance is compared. The τ of MTRGU-2 and MTRGU-3 are set to {1, 1.42, 2, 2.5} and {1,1,1.25,1.25}, respectively.

MTGRU-1 is the final model adopted in the experiments described above. MTGRU-2 has a relatively slow context layer, and MTGRU-3 has two fast context layers and two slow context layers. As can be seen from the comparison, the learning performance of the MTRGU-1 is superior to the other two, which can be the basis for setting the time constant.

The results of the experiments provide evidence that organizational processes such as functional differentiation occur in RNNs in language tasks. MTGRU can learn faster than an existing GRU by one epoch. It is expected that the MTRGU will facilitate a sort of functional differentiation process already occurring by routing the hierarchy to multiple time constants in the RNN, otherwise this temporal hierarchy will occur more gradually.

Referring to FIG. 7, there is shown a comparison of the generated summary of the input paragraphs for the extracted summary. As can be seen from the example, the model according to the present invention successfully extracts key information from multiple sentences and reproduces it as a one line summary. The system was learned only for the extraction summary, but the generalization ability of the model makes it possible to abstract the entire paragraph. The purpose of the seq2seq is to maximize the joint probability of the conditioned sequence of targets in the source sequence.

When the summary model is learned for a source-extracted salient sentence pair, it can be seen that the purpose consists of two sub-purposes. One is to correctly perform salvation research (criticality extraction) to identify the most prominent content, and the other is to generate the correct sequence of sentence targets. Indeed, during learning, we observed that the first sub-objective optimization was achieved before the second sub-goal. The second sub-goal is achieved only when the learning set and fitness occur.

The ability to generalize a model is due to the fact that the model is expected to learn several prominent points per given paragraph input (not just one significant section corresponding to a single sentence), as many learning examples show. This explains how the results from Fig. 7 are obtained from the model.

The work according to the invention is expected to have a meaningful impact on the seq2seq abstraction summary. First, the results of the present invention confirm that an encoder-decoder model can be learned to perform saliency identification without reference to external corpus at test time. Rush et al. (2015); Nallapati et al. (2016), and has become apparent in the present invention due to the selection of data consisting of short-striking sentence pairs.

Second, the stochastic language model can solve the task of generating new words in the setting of core criteria of abstract summary. Bengio et al. (2003) showed that original probabilistic language models can achieve much more generalization of similar words. This is due to the fact that the probability function is a smooth function of the word-embedding vector.

Since similar words have similar embedding vectors, a small change in feature leads to a small change in the predicted probability. This makes it the most useful solution for abstract summaries of the RNN language model needed to generate new sentences.

For example, the first summary in Figure 6 shows that the model of the invention creates the word "explored " that does not exist in the document. Moreover, the results of the present invention suggest that if the given abstract target is present, the same model can learn a totally abstract summary system.

As described above, in the present invention, after generating an internal expression grammar, a sentence generation technique is applied, and then an abstract expression is generated to generate a summary similar to a human-created summary. The present invention can also summarize a long text composed of a plurality of paragraphs as an abstract summary rather than a simple extraction.

9 is a flowchart of a method for generating an abstract summary of a multi-paragraph text according to an embodiment of the present invention.

The method of generating an abstract summary of a multi-paragraph text according to the present embodiment may proceed in substantially the same configuration as the apparatus 10 of FIG. Therefore, the same constituent elements as those of the apparatus 10 of FIG. 1 are denoted by the same reference numerals, and repeated description is omitted. In addition, the method of generating an abstract summary of a plurality of paragraph texts according to the present embodiment can be executed by software (application) for performing generation of an abstract summary of a plurality of paragraph texts.

Referring to FIG. 9, an abstract summary text generation method of a multi-paragraph text according to the present embodiment automatically classifies a document into paragraphs (step S10). For example, you can create a new set of data for Computer Science (CS) articles at ArXiv.org, extract introductions from LaTeX source files, and separate them into paragraphs.

The document is automatically classified into paragraphs, and then the paragraph is converted into an internal representation vector (step S20). In the present invention, after generating an internal expression grammar, a sentence generation technique is applied, and then an abstract expression is generated to generate a summary similar to a human-created summary. The present invention can also summarize a long text composed of a plurality of paragraphs as an abstract summary rather than a simple extraction.

The transformed vector is generated as an internal representation based on the deep learning technique through a multi-layered regression neural network of multiple timescales gated recurrent units (GRUs) with multiple time constants (step S30). The regressive neural network has multiple instantiations based compositionality.

The regression neural network uses a sequence-to-sequence model using a temporal hierarchical model with MTGRU as a basic unit. The MTGRU has a time constant set for each layer, and the larger the time constant, the slower the context unit. For example, the time constant may increase as the layer rises.

In addition, the internal representation is decoded through a regression neural network composed of multiple layers of MTGRU (step 40). Thereafter, sentences expressed in natural language are generated from the decoded internal expressions according to the learning contents using the language modeling (step 50).

As a result, the summary output of each paragraph is collected and the final abstract summary is output (step 60).

The present invention has the ability to summarize long texts consisting of a plurality of paragraphs by having multiple compositionalities based on time constants. The abstract summary generated in accordance with the present invention is similar to a human-generated summary, unlike the abstracts generated by the existing extractive summary system, and has a natural grammar using the deep-run technique.

As such, the method of generating an abstract summary of a plurality of paragraph texts may be implemented in an application or may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be ones that are specially designed and configured for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims. You will understand.

The device proposed in the present invention can be used as a core infrastructure technology in all areas where a summary is needed today. For example, Google's search system can be an example. The main application system will be a system that automatically summarizes important contents in long sentences and summarizes them. In particular, as the big data market expands, it will be able to have a great competitive edge by providing such functions. In addition, core source technology that can be used in fields such as IT and HCI (Human Computer Interface). It can be applied to natural language processing (NLP) -based application system, search engine, smart home conversation agent, and the like.

10: abstract abstract text generator for multiple paragraph text
100: Input unit
300:
310: Inner vectorization unit
330: Coded Neural Network
500: Decryption processor
510: Decoded neural network
530: Natural language generating unit
700: Output section

Claims (15)

An input unit for automatically delivering a document to paragraphs;
Converts a paragraph transferred from the input unit into an internal representation vector, and generates a representation of the transformed vector through a regression neural network including multiple timescales gated recurrent units (GRUs) having multiple time constants An encoding processing unit;
A decoding processing unit for decoding the internal representation received from the encoding processing unit through a regression neural network including the MTGRU and generating sentences using language modeling; And
And an output unit for collecting the summary output of each paragraph and outputting a final abstract summary.
The apparatus according to claim 1,
An internal vectorization unit for converting a paragraph transferred from the input unit into an internal representation vector; And
Generating an abstract summary of a plurality of paragraphs of text including a regression neural network consisting of multiple layers of MTGRU and generating a representation of the transformed vector based on a deep learning technique and delivering the representation to the decoding processor; Device.
The decoding apparatus according to claim 2,
A decoding neural network that includes a regression neural network composed of multiple layers of MTGRU and decodes the internal representation received from the encoding processing unit; And
And a natural language generation unit for generating sentences to be expressed in a natural language from the decrypted internal expression received from the decryption neural network according to learning contents using language modeling.
The method of claim 3,
Wherein said regression neural network uses a sequence-to-sequence model using a temporal hierarchical model with MTGRU as a base unit.
5. The method of claim 4,
Wherein the MTGRU has a time constant set for each layer, and the larger the time constant, the slower the context unit.
6. The method of claim 5,
Wherein the time constant increases as the hierarchy increases.
The method according to claim 1,
Wherein the regression neural network has multiple temporal-based compositionality.
The method according to claim 1,
Wherein the input unit extracts introductions from a LaTeX source file.
Automatically classifying the document into paragraphs;
Converting the paragraph into an internal representation vector;
Generating a transformed vector as an internal representation based on a deep learning technique through a multi-layered regression neural network of Multiple Timescales Gated Recurrent Unit (GRU) with multiple time constants;
Decoding the internal representation through a multi-layer regression neural network of MTGRU;
Generating sentences expressed in natural language from the decoded internal expression according to learning contents using language modeling; And
Collecting the summary output of each paragraph and outputting a final abstract summary. ≪ Desc / Clms Page number 20 >
10. The method of claim 9,
Wherein the recursive neural network uses a sequence-to-sequence model using a temporal hierarchical model with MTGRU as a base unit.
11. The method of claim 10,
Wherein the MTGRU has a time constant set for each layer, and the larger the time constant, the slower the context unit.
12. The method of claim 11,
Wherein the time constant is increased as the hierarchy is increased.
10. The method of claim 9,
Wherein the regressive neural network has multiple temporal-based compositionality, wherein the recursive neural network has multiple temporal-based compositionalities.
10. The method of claim 9,
Further comprising extracting an Introductions from a LaTeX source file. ≪ RTI ID = 0.0 > 8. < / RTI >
A computer-readable recording medium on which a computer program for performing a method of generating an abstract summary of a plurality of paragraphs of texts according to any one of claims 9 to 14 is recorded.

Images