CN116775855A - Automatic TextRank Chinese abstract generation method based on Bi-LSTM - Google Patents

Abstract

The invention relates to the field of computer technology, and specifically relates to a Bi-LSTM-based TextRank Chinese abstract automatic generation method. The word vector converted by the Word2vec model is used as input information, and the Bi-LSTM model is used for further processing, and the output information is used as each sentence of the text. sentence vectors and used to calculate the similarity between sentences. By using the similarity between sentences as the edge weight and each sentence as a node, a TextRank graph structure is constructed, the TextRank value of each sentence is calculated as the weight of each sentence, and sorted according to the size of the weight, and finally the candidate summary sentences are extracted to form the final summary. . The present invention integrates the Word2vec+TextRank automatic summary model through Bi‑LSTM and proposes a new fusion model W2v‑BiL‑TR, which improves the quality of summary extraction results.

Description

TextRank Chinese summary automatic generation method based on Bi-LSTM

Technical field

The invention relates to the field of computer technology, and in particular to a method for automatically generating TextRank Chinese abstracts based on Bi-LSTM.

Background technique

In order to handle the current complex and diverse Internet information and allow Internet users to quickly obtain effective and streamlined information from large and complex Internet texts, automatic text summarization technology is proposed. This technology can better streamline a large amount of complex and redundant information, allowing users to quickly obtain the information they want. Current automatic summary generation technology is mainly divided into two categories: extractive summary technology and generative summary technology. The present invention is mainly implemented based on the improvement of extractive summary technology. At present, domestic and foreign scholars have made many contributions to text extraction summarization technology:

Extractive summary technology mainly calculates and sorts the weight of each sentence in the text. The final summary result is mainly composed of the top-ranked summary candidate sentences. It is more readable and the summary result can be better expressed. Theme information of the original text. In 2004, Mihalcea proposed the TextRank algorithm based on the PageRank algorithm, in which the graph structure in the PageRank algorithm was quoted. This algorithm mainly calculates the weight of each sentence by calculating the similarity between sentences. In 2013, Tomas Mikolv and others proposed the Word2vec model. This model can extract word vectors from text, and rely on the fusion of word vectors to obtain the sentence vectors of each sentence. The sentence vectors contain certain text semantic features, and combined with the TextRank algorithm, can improve the quality of summary results. In order to have a deeper representation of the semantic features of text, some scholars try to combine neural network structures to process text. Cheng et al. encoded text in different forms of sentences and words, and used an encoding layer-decoding layer to calculate the features of sentences and words respectively. In 2020, Luo Feixiong used the BERT model to further process the text and accurately and deeply express the semantic information of the text. The processed text is used in the TextRank algorithm, and the summary results generated are of relatively high quality.

Through research on current automatic summary technology, the current mainstream extractive summary technology is not in-depth at extracting text semantic features. The current mainstream TextRank extractive summary technology that integrates Word2vec mainly uses the Word2vec model to process text and convert text into words. Vectors are used to construct text sentence vectors. Although this method is simple, the Word2vec model structure is relatively simple and cannot process long texts well, so it does not fully represent the overall semantic features of the full text.

Contents of the invention

The purpose of this invention is to provide a Bi-LSTM-based TextRank Chinese summary automatic generation method, aiming to solve the technical problem that the existing mainstream extractive summary technology does not extract text semantic features in depth, using the Bi-LSTM model, The word vector converted by the Word2vec model is used as input information, and after further processing, a higher quality summary result is generated.

In order to achieve the above objectives, the present invention provides a Bi-LSTM-based TextRank Chinese summary automatic generation method, which includes the following steps:

Step 1: Preprocess the text, then segment it into a one-dimensional sentence list S, where the length of the list represents the number of sentences in the text, and S[i] is defined as the i-th sentence in the text;

Step 2: Keep the punctuation marks of the text, perform word segmentation processing on the clause list, and obtain a two-dimensional word list W, where W[i][j] is defined as the j-th word of the i-th sentence in the text;

Step 3: Use the Word2vec model to process the two-dimensional word list W in step 2, extract the text word vector, and obtain the two-dimensional word vector table WV of the text, where WV[i][j] corresponds to the i-th sentence of the text. Word vectors of j words;

Step 4: Use the obtained two-dimensional word vector table WV as input information, use the Bi-LSTM model for processing, and use the obtained output state as a sentence vector to generate a sentence vector table SV, where SV[i] represents the i-th sentence vector of a sentence;

Step 5: Calculate the sentence vector obtained in step 4, obtain the similarity between sentences, and form a two-dimensional matrix X;

Step 6: Use the TextRank model to construct a graph structure based on the two-dimensional matrix , and used for sentence scoring;

Step 7: According to the ranking of the weight of each sentence, select the top sentence as the candidate summary sentence as the final extractive summary result.

Preferably, in the process of using the Bi-LSTM model for processing, the word vector converted by the Word2vec model is used as input information. After input at different times, the input gate, hidden gate and output gate of the two forward and reverse LSTM structures are used. processing, specifically filtering the input information through the input gate and passing it to the current cell state; at the same time, the forgetting gate filters the cell state at the previous moment, combines the retained information with the current cell state, and processes it through the output gate to obtain The hidden state information in different directions is spliced together as the final hidden state output result of the Bi-LSTM model.

Preferably, the Bi-LSTM model's specific processing of text information at time t includes the following steps:

Process and analyze the input data information and calculate candidate cell states

Based on the input information and the information at the previous moment, the input gate and forget gate at the current moment are updated and calculated;

Calculate the cell state C _t at the current moment through the bidirectional input gate, forget gate and hidden gate status information at the current moment;

Based on the previously obtained state, calculate the output gate state and obtain the output h _st of the one-way LSTM. Finally, by splicing the output of the forward LSTM and the reverse LSTM, the final Bi-LSTM output h _t is obtained.

Preferably, the candidate cell state The calculation formula is as follows:

where W _c represents the weight, h _t-1 represents the output vector of the previous time step, and b _c is the bias.

The calculation formula of the input gate at the current moment is as follows:

i _t =σ(W _t ·[h _t-1 ,x _t ]+b _i )

Where W _t represents the weight of the output of the previous time and the input of the current time to the input gate i _t of the current time, σ is the activation function, _{and bi} is the bias.

The calculation formula of the forgetting gate at the current moment is as follows:

f _t =σ(W _f ·[h _t-1 ,x _t ]+b _f )

Where W _f represents the weight of the output of the previous time and the input of the current moment to the forgetting gate f _t at the current moment, σ is the activation function, and b _f is the bias.

The calculation formula of the memory cell state C _t at the current moment is as follows:

Through the forgetting gate, the probability of forgetting the memory cell state at the previous moment is obtained, and the remaining information of the cell state at the previous moment is obtained. The information of the candidate cell state is selectively input by the input gate. The current memory cell state information is obtained by combining the two.

The calculation formula of the output gate at the current moment is as follows:

o _t =σ(W _o ·[h _t-1 ,x _t ]+b _o )

Among them, W _o represents the weight of the output of the previous time and the input of the current moment to the output gate o _t at the current moment, σ is the activation function, _and bo is the bias.

Through the output gate processing of the cell state at the current moment, the output information at the current moment is obtained. The calculation formula is as follows:

h _st =o _t *tanh(C _t )

Among them, h _st represents the output of single-directional LSTM, and tanh is the activation function. By splicing the output of LSTM in two directions, the complete Bi-LSTM output information is obtained, namely:

Preferably, the word vector obtained by processing the text with Word2vec is used as the input sequence, and the word vector of the sentence _Si in the text is converted into a fixed-dimensional vector representation through the embedding layer of Bi-LSTM. Then the input vector is encoded in the bidirectional LSTM, and the current input information x _t is processed through the input gate i _t according to time steps. Then, the cell state C _t-1 of the previous moment is probabilistically forgotten through the forget gate f _t and the useful information is retained. The combination of the two obtains the current memory cell state C _t . The cell status at the current moment is processed through the output gate o _t combined with the tanh function to obtain one-way output information h _st . By splicing the forward and reverse one-way output information, the complete Bi-LSTM output information h _t is obtained. The final Bi-LSTM output h _t is used to represent the context information of the text.

The present invention provides a method for automatically generating TextRank Chinese abstracts based on Bi-LSTM. The word vector converted by the Word2vec model is used as input information, and the Bi-LSTM model is used for further processing, and the output information is used as the sentence vector of each sentence of the text, and used To calculate the similarity between sentences. By using the similarity between sentences as the edge weight and each sentence as a node, a TextRank graph structure is constructed, the TextRank value of each sentence is calculated as the weight of each sentence, and sorted according to the size of the weight, and finally the candidate summary sentences are extracted to form the final summary. . The present invention integrates the Word2vec+TextRank automatic summary model through Bi-LSTM and proposes a new fusion model W2v-BiL-TR, which improves the quality of summary extraction results.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow chart of a Bi-LSTM-based TextRank Chinese summary automatic generation method of the present invention.

Figure 2 is a schematic structural diagram of the fusion model W2v-BiL-TR of the TextRank Chinese abstract automatic generation method based on Bi-LSTM of the present invention.

Figure 3 is a Bi-LSTM feature extraction flow chart of a Bi-LSTM-based TextRank Chinese summary automatic generation method of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and are not to be construed as limiting the present invention.

Please refer to Figure 1. The present invention provides a TextRank Chinese summary automatic generation method based on Bi-LSTM, which includes the following steps:

S1: Preprocess the text, then segment it into a one-dimensional sentence list S, where the length of the list represents the number of sentences in the text, and S[i] is defined as the i-th sentence in the text;

S2: Preserve the punctuation marks of the text, perform word segmentation processing on the clause list, and obtain a two-dimensional word list W, in which W[i][j] is defined as the j-th word of the i-th sentence in the text;

S3: Use the Word2vec model to process the two-dimensional word list W in step S2, extract the text word vector, and obtain the two-dimensional word vector table WV of the text, where WV[i][j] corresponds to the i-th sentence j of the text word vector of a word;

S4: Use the obtained two-dimensional word vector table WV as input information, use the Bi-LSTM model for processing, and use the obtained output state as a sentence vector to generate a sentence vector table SV, where SV[i] represents the i-th Sentence vector of a sentence;

S5: Calculate the sentence vector obtained in step S4, obtain the similarity between sentences, and form a two-dimensional matrix X;

S6: Use the TextRank model to construct a graph structure based on the two-dimensional matrix and used for sentence scoring;

S7: According to the ranking of the weight of each sentence, select the top sentence as the candidate summary sentence as the final extractive summary result.

Specifically, the present invention integrates the Bi-LSTM model to further process the text information, and the processed information is used as the sentence vector of each sentence to calculate the similarity between each sentence. TextRank is used to construct a graph structure, in which the graph nodes are sentences, and the edge weights between nodes are represented by the similarity between sentences. The weight of each sentence is calculated through the graph structure for sentence sorting, and finally the summary result is generated. This invention is mainly based on the Word2vec model, Bi-LSTM model and TextRank algorithm, and proposes a new fusion model W2v-BiL-TR, whose structure is shown in Figure 2.

Further, the following is explained in conjunction with the specific implementation process. In the process of using the Bi-LSTM model for processing, after the text is processed by the Bi-LSTM model, the information obtained can be used as the sentence vector information of each sentence of the text.

The specific workflow diagram of Bi-LSTM processing text information is shown in Figure 3. The Bi-LSTM model processes text information. When extracting features from the text, the word vector converted by the Word2vec model is mainly used as input information, as shown in Figure 1 { x ₁ ,x ₂ ,...x _n }. After inputting at different times, through the processing of the input gate, hidden gate and output gate of the two forward and reverse LSTM structures, two hidden state information are obtained and spliced, as shown in {h ₁ , h ₂ , in Figure 1. ..h _n }, as the final hidden state output result of the Bi-LSTM model.

The specific steps for the Bi-LSTM model to process text information at time t are as follows:

Process and analyze the input data information and calculate candidate cell states Calculated as follows:

where W _c represents the weight, h _t-1 represents the output vector of the previous time step, and b _c is the bias.

Based on the input information and the information at the previous moment, the input gate and forget gate at the current moment are updated and calculated. The calculation formula is as follows:

i _t =σ(W _t ·[h _t-1 ,x _t ]+b _i ) (2)

Where W _t represents the weight of the output of the previous time and the input of the current time to the input gate i _t of the current time, σ is the activation function, _{and bi} is the bias.

f _t =σ(W _f ·[h _t-1 ,x _t ]+b _f ) (3)

Where W _f represents the weight of the output of the previous time and the input of the current moment to the forgetting gate f _t at the current moment, σ is the activation function, and b _f is the bias.

Through the bidirectional input gate, forget gate and hidden gate status information at the current moment, the cell state C _t at the current moment is calculated. The calculation formula is as follows:

Through the forgetting gate, the probability of forgetting the memory cell state at the previous moment is obtained, and the remaining information of the cell state at the previous moment is obtained. The information of the candidate cell state is selectively input by the input gate. The current memory cell state information is obtained by combining the two.

Finally, based on the previously obtained state, the output gate state is calculated and the output h _st of the one-way LSTM can be obtained. Finally, the final output h _t of Bi-LSTM is obtained by splicing the output of the forward LSTM and the reverse LSTM. Calculated as follows:

o _t =σ(W _o ·[h _t-1 ,x _t ]+b _o ) (5)

Among them, W _o represents the weight of the output of the previous time and the input of the current moment to the output gate o _t at the current moment, σ is the activation function, _and bo is the bias.

h _st =o _t *tanh(C _t ) (6)

Among them, h _st represents the output of single-directional LSTM, and tanh is the activation function.

By splicing the output of LSTM in two directions, the complete Bi-LSTM output information h _t is obtained.

The final h _t output by Bi-LSTM is used to represent the contextual information of the text. Due to the combination of the input information of forward and reverse LSTM, and due to the memory gate mechanism of Bi-LSTM, the dependence of long-distance sequences is captured, and the memory state can be changed at different times t, which can well solve long dependencies. Questions can also be used to dig deeper and extract the semantic features of the text.

After the text information is further processed through the Bi-LSTM model, the output information obtained is used as the sentence vector of each sentence. Finally, after calculating the similarity between sentences based on the sentence vector of the Bi-LSTM model, it is used to construct the TextRank graph structure, and the weight of each sentence is calculated to extract candidate summary sentences.

In order to analyze the quality of the summary results finally obtained by this technology, the present invention also proposes a specific embodiment, using Rouge-1, Rouge-2 and Rouge-L evaluation methods to respectively evaluate the traditional TextRank summary technology, TextRank summary technology integrated with Word2vec and The fusion model W2v-BiL-TR newly proposed by the present invention is evaluated. Rouge-1 and Rouge-2 both belong to Rouge-N, where N refers to the matching rate of the words and number of words in the generated summary and the standard summary. For example, Rouge-1 is calculated based on the matching of single words and single words in both. Rouge-L is calculated using the coverage criterion of the longest common subsequence in the generated summary and the standard summary. The specific evaluation process of the three evaluation standards is as follows:

(1) Convert the results of generated abstracts and standard abstracts into sequences of single words or words, double words or words;

(2) Calculate the number of occurrences of each single word or word, double word or word in the standard summary, and construct the longest common subsequence LCS based on the single word or word;

(3) Calculate the number of occurrences of each single word or word, double word or word in the generated summary, and construct the longest common subsequence LCS based on the single word or word;

(4) Calculate the P, R, and F ₁ values of Rouge-1, Rouge-2, and Rouge-L in sequence. The specific calculation formula is as follows:

Where n represents the number of divided word sequences that overlap between the generated summary and the standard summary. X and Y represent generated summary and standard summary respectively. u and v represent the number of word sequences appearing in the generated summary and the standard summary respectively. R represents the recall rate, P represents the precision rate, and the F ₁ value is the harmonic mean of the P value and the R value.

The following are the evaluation results of different evaluation standards:

Table 1 Rouge-1 values of extraction summary results of different technologies

As shown in Table 1, the summary results obtained after further processing the text by integrating the W2v-BiL-TR model are higher in all indicators in the Rouge-1 evaluation method. Therefore, it can be explained that after integrating the Bi-LSTM model, the semantic features of the text can be better represented, and the quality of the generated summary is also better.

Table 2 Rouge-2 values of extraction summary results of different technologies

Similarly, as shown in Table 2, the summary results obtained after further processing the text by integrating the W2v-BiL-TR model have higher indicators in the Rouge-2 evaluation method. It is further explained that after integrating the Bi-LSTM model, the semantic features of the text can be better represented, and the quality of the generated summary is also better.

Table 3 Rouge-L value of extraction summary results of different technologies

When each technology uses the longest common subsequence as the evaluation standard, that is, when using the Rouge-L evaluation method, the new technology still has the highest indicators in integrating the W2v-BiL-TR model. Therefore, it can be concluded that due to the addition of the Bi-LSTM model, the text word vectors extracted by the Word2vec model are further processed, and the sentence vectors obtained in this way have richer features and contain more semantic information than the sentence vectors simply fused using word vectors. . Therefore, when using the TextRank algorithm to calculate sentence weights, the effect will be better, and the quality of the final summary result will be better.

To sum up, the present invention uses the Bi-LSTM model and uses the word vector converted by the Word2vec model as input information. After further processing, the obtained output information is used as the sentence vector of each sentence. The similarity between sentences is calculated and used to construct TextRank. Graph structure. The resulting summary results are of higher quality.

What is disclosed above is only a preferred embodiment of the present invention. Of course, it cannot be used to limit the scope of the present invention. Those of ordinary skill in the art can understand all or part of the processes for implementing the above embodiments and according to the rights of the present invention. Equivalent changes to the requirements still fall within the scope of the invention.

Claims (5)

1. A text rank Chinese abstract automatic generation method based on Bi-LSTM is characterized by comprising the following steps:

step 1: preprocessing a text, then carrying out sentence dividing processing, and dividing the text into a one-dimensional sentence list S, wherein the list length represents the number of sentences in the text, and S [ i ] is defined as the ith sentence in the text;

step 2: preserving punctuation marks of the text, and carrying out word segmentation processing on the sentence list to obtain a two-dimensional word list W, wherein W [ i ] [ j ] is defined as a j-th word of an i-th sentence in the text;

step 3: processing the two-dimensional Word list W in the step 2 by adopting a Word2vec model, extracting text Word vectors to obtain a Word vector two-dimensional table WV of the text, wherein WV [ i ] [ j ] corresponds to the Word vector of the j-th Word of the i-th sentence of the text;

step 4: processing the obtained word vector two-dimensional table WV serving as input information by using a Bi-LSTM model, and generating a sentence vector table SV by taking the obtained output state as a sentence vector, wherein SV [ i ] represents the sentence vector of the ith sentence;

step 5: calculating sentence vectors obtained in the step 4, obtaining similarity among sentences, and forming a two-dimensional matrix X;

step 6: constructing a graph structure according to a two-dimensional matrix X by using a TextRank model, wherein nodes of the graph correspond to sentences of the text, edge weight values of the graph correspond to similarity among the sentences, and finally, calculating TextRank values of all the sentences as sentence weight values and using the TextRank values for scoring the sentences;

step 7: and selecting sentences with the front names as candidate abstract sentences according to the ranking degree of the weight values of the sentences, and taking the sentences as a final extraction abstract result.

2. The automatic generation method of TextRank Chinese abstract based on Bi-LSTM of claim 1, wherein,

in the process of processing by using the Bi-LSTM model, word vectors converted by the Word2vec model are used as input information, and after the Word vectors are input at different moments, the input information is screened by the input gate, the hidden gate and the output gate of the two forward and reverse LSTM structures, and the input information is transmitted to the current cell state; and meanwhile, the forgetting gate screens the cell state at the previous moment, combines the reserved information with the current cell state, obtains hidden state information in different directions through the processing of the output gate, and splices the hidden state information to be used as a final hidden state output result of the Bi-LSTM model.

3. The automatic Bi-LSTM based TextRank chinese summary generation method of claim 2, wherein,

the specific processing process of the Bi-LSTM model on the text information at the time t comprises the following steps:

processing and analyzing the input data information and calculating candidate cell states

Updating and calculating an input door and a forget door at the current moment according to the input information and the information at the last moment;

through the bidirectional input door, the forgetting door and the hidden door at the current momentThe state information of the hidden gate is used for calculating the state C of the cell at the current moment _t ；

From the previously obtained states, the output gate state is calculated and the output h of the unidirectional LSTM can be obtained _st Finally, the output h of the final Bi-LSTM is obtained by splicing the outputs of the forward LSTM and the reverse LSTM _t 。

4. The automatic generation method of TextRank Chinese abstract based on Bi-LSTM of claim 3 wherein,

the candidate cell stateThe calculation formula of (2) is as follows:

wherein W is _c Represents the weight, h _t-1 Output vector representing last time step, b _c Is biased;

input door i at current moment _t The calculation formula of (2) is as follows:

i _t ＝σ(W _t ·[h _t-1 ,x _t ]+b _i )

wherein W is _t Input gate i representing the input of the last time and the input of the current time to the current time _t Is the weight of (a), sigma is the activation function, b _i Is biased;

forgetting door f at current moment _t The calculation formula of (2) is as follows:

f _t ＝σ(W _f ·[h _t-1 ,x _t ]+b _f )

wherein W is _f Forget gate f indicating the input of the last time and the current time to the current time _t Is the weight of (a), sigma is the activation function, b _f Is biased;

memory cell state C at the present moment _t The calculation formula is as follows:

the probability of the memory cell state at the previous moment is forgotten through a forgetting door, so that information which is remained in the cell state at the previous moment is obtained, and the information is selected and input with the candidate cell state information through an input door, and the information and the candidate cell state information are combined to obtain the current memory cell state information;

output door o at current moment _t The calculation formula of (2) is as follows:

o _t ＝σ(W _o ·[h _t-1 ,x _t ]+b _o )

wherein W is _o Output gate o representing the output of the last time and the input of the current time to the current time _t Is the weight of (a), sigma is the activation function, b _o Is biased;

the output information of the current moment is obtained through the processing of the output gate on the cell state at the current moment, and the calculation formula is as follows:

h _st ＝o _t *tanh(C _t )

wherein h is _st Representing the output of LSTM in a single direction, wherein tanh is an activation function, and the complete Bi-LSTM output information is obtained by splicing the outputs of LSTM in two directions, namely:

5. the automatic generation method of TextRank Chinese abstract based on Bi-LSTM of claim 4 wherein,

output h of final Bi-LSTM _t For representing sentences S in text _i Is a sentence vector of (a).