CN117251562A - A text summary generation method based on fact consistency enhancement - Google Patents

Abstract

The present invention relates to the technical field of natural language processing. It discloses a text summary generation method based on enhanced fact consistency, which solves the problem that the existing technology ignores the different contributions of different importance levels between fact triplets to the final summary result, and improves the generation of text summaries. The credibility of the text summary. The present invention uses the Transformer architecture to build a sequence-to-sequence text summary generation model, and introduces a fact attention module between the feedforward network module and the cross-attention module of the decoder for attention vector sum based on each fact triplet. The cross attention module outputs the word vector of the generated word to calculate the influence of each fact triplet on the generated word, and uses this to update the word vector of the generated word; while the attention vector of the fact triplet is calculated in the fact triplet Based on the encoding vector, it is calculated through the self-attention mechanism.

Description

A text summary generation method based on factual consistency enhancement

Technical field

The invention relates to the technical field of natural language processing, and in particular to a text summary generation method based on enhanced fact consistency.

Background technique

With the widespread application of mobile Internet and the rapid popularization of smart terminal devices, people can create and publish content on the Internet, and can also easily obtain public information on the Internet. While making people's daily life and work more convenient, it also greatly It enriches people's spiritual life. However, online text information is often longer and more difficult to understand. Compared with short video information, it takes longer to read and has less focused focus. This makes it difficult to interest readers in today's fast-paced life. To a certain extent, It reduces the reader’s reading experience.

The generative text summarization model aims to convert text or text collections into short summaries containing key information. The emergence of this technology solves the problem of information overload. Generative summary models are usually based on the encoding and decoding framework to implement sequence-to-sequence tasks, and use the powerful mapping capabilities of the Transformer-based pre-training model and the network architecture to generate short and highly fluent summaries. However, the way the generative summary model outputs words based on probability may still lead to the generation of words or phrases that are inconsistent with the meaning expressed in the original text, leading to factual errors in the summary results, affecting the credibility of the summary results, thus limiting the promotion and use of the generative summary model. implementation, further reducing the research and application value of the generative summary model.

Therefore, how to solve the problem of factual errors that are prone to occur in generative summary models and effectively enhance the factual consistency of summary results is one of the hot topics in this field.

Factual consistency enhancement aims to reduce the probability of factual errors in generative summary results. Research on factual consistency of automatic text summarization technology first started in 2018. Ziqiang Cao et al. found that there were about 30% errors or unverifiable facts in the summary results generated by mainstream generative summary models. Later, Ziqiang Cao et al. proposed the FTSum model, which uses a new encoder to encode fact triples and splice them with the sentence encoding of the original document, so that the model can notice the influence of fact triples, but it ignores them. It solves the problem that different importance levels among fact triples contribute differently to the final summary result.

Gunel B and others proposed to construct a knowledge graph from wiki encyclopedia data and introduce the entity-level knowledge into the encoding and decoding framework to guide the model to generate correct facts. However, they ignored that the traditional encoding and decoding framework easily outputs information different from the original text during the decoding process. Vocabulary that does not match the facts will lead to factual errors.

Contents of the invention

The technical problem to be solved by the present invention is to propose a text summary generation method based on enhanced fact consistency, solve the problem that the existing technology ignores the different contributions of different importance levels between fact triplets to the final summary result, and improve the generated text The credibility of the abstract.

The technical solution adopted by the present invention to solve the above technical problems is:

A text summary generation method based on fact consistency enhancement, using the Transformer architecture to build a sequence-to-sequence text summary generation model. The decoder of the Transformer architecture includes a self-attention module, a cross-attention module and a feed-forward network connected in sequence. module, introducing a fact attention module between the feedforward network module and the cross-attention module of the decoder;

Define the fact triplet and express it as F _i =<s _i ,ri _, o _i >, where s is the subject word, r is the relational modifier, o is the object word, and the subscript i is the fact triplet. serial number, and each word in any fact triplet comes from the same sentence in the original text; the fact attention module uses the attention vector of each fact triplet in the original text, and the decoder cross The third word vector sequence output by the attention module is used as input. Based on the cross-attention mechanism, the influence coefficients of all fact triples on each generated word are obtained, and the influence coefficients of all fact triples on each generated word are calculated for the third word vector sequence. The word vector sequence is updated to obtain a fourth word vector sequence, which is used as the input of the decoder feedforward network module;

The attention vector of the fact triplet is calculated as follows:

A1. Use natural language processing tools to process the original text, extract fact triples F _i , and construct a set of fact triples F; then, use the words contained in each fact triplet F _i to construct nodes and connections to construct edges. Map the set of fact triples F into a graph, and use the graph neural network to calculate the node vectors of each node in the graph; then, splice the node vectors of the nodes contained in each fact triplet to obtain each fact triplet. encoding features in, and/> Respectively represent the node vectors of the nodes corresponding to si _, r _i and o _i contained in the fact triplet F _i ;

A2. Input the coding features of the fact triples into the recurrent neural network, and obtain the vector representation z _i of the semantic information of the fused fact triples before and after each fact triple;

A3. Use the vector representation z _i of each fact triplet and obtain the attention vector of each fact triplet based on the self-attention mechanism.

Further, in step A2, the recurrent neural network is a Bi-LSTM network; based on the encoding characteristics of the fact triplet, the Bi-LSTM network is used to obtain the forward hidden layer vector of each fact triplet. and backward hidden layer vector/> Then, convert the forward hidden layer vector/> and backward hidden layer vector/> Perform vector splicing to obtain the vector representation of the semantic information of each fact triplet before and after the fusion of each fact triplet/>

Further, in step A1, the graph neural network is used to calculate the node vectors of each node in the graph, including:

First, use the pre-trained model to initialize the node vectors of each node in the graph;

Then, use the GCN network to update the node vectors of each node in the graph according to the following formula:

Among them, ReLU represents the activation function, H ^l and H ^l+1 represent the outputs of the l and l+1 layers of the GCN network respectively, and A represents the adjacency matrix of the graph. represents the degree matrix, and W ^l represents the weight matrix of the lth layer of the GCN network.

Further, step A1 also includes:

First, use the decision tree model of the natural language processing tool to classify the relationship between each fact triplet F _i based on the relationship type it contains; then, based on the relationship classification results, obtain the relationship between the fact triplet F _i The relationship triplet between them constructs the relationship triplet set R;

Then, the coding features and relationship vectors of the fact triples contained in each relationship triplet are spliced to obtain the coding features of the relationship triplet R _j = [h _1j , cr _j , h _2j ], h _1j and h _2j Respectively represent the encoding characteristics of the fact triples contained in the j-th relationship triplet, cr _j represents the relationship vector obtained by the relationship contained in the j-th relationship triplet based on the pre-training model;

After that, the coding features of each relationship triplet and the coding features of each fact triplet are used to perform cross-attention calculations. Based on the calculation results, the coding features of each fact triplet are updated, and the updated fact triplet is The encoding feature h′ _i is used as the input of step A2.

Further, the coding features of each relationship triplet and the coding features of each fact triplet are used to perform cross-attention calculations, and the coding features of each fact triplet are updated based on the calculation results, including:

α _ij =h _i *R _j

Among them, α _ij expresses the correlation between the i-th fact triplet and the j-th relationship triplet, β _ij expresses the correlation weight between the i-th fact triplet and the j-th relationship triplet, and R represents the relationship. Collection of triples.

Further, in step A3, the vector representation z _i of each fact triple is used to obtain the attention vector of each fact triple based on the self-attention mechanism; the self-attention mechanism is multi-head self-attention, which calculates The process includes:

A31. Based on the transformation matrices W _k1 , W _k2 and W _k3 of each attention head, convert the vector representation z _i of the fact triplet into a Query value Key value/> and Value/>

A32, based on and/> Calculate the attention of each attention head:

Among them, _dk is dimensions;

A33. Aggregate the attention calculation results of each attention head to obtain the attention vector Z _i :

Among them, K is the number of attention heads;

A34. Use linear transformation to change the dimension of the attention vector Z _i to the dimension of the vector representation z _i , and obtain the attention vector r _i of the fact triplet;

r _i =Z _i *W ⁰

Among them, W ⁰ is the linear transformation matrix.

Further, at each time step, the decoding process of the decoder is:

B1. Based on each generated word obtained before the current time step t, use the pre-training model to obtain its word embedding, and use the word embedding of each generated word to form the first word vector sequence of the generated words in the order of generation;

B2. Input the first word vector sequence into the self-attention module of the decoder; first, update the first word vector sequence based on the self-attention mechanism; then, combine the input first word vector sequence with the updated first word vector sequence. The word vector sequence is residually connected and normalized to obtain the second word vector sequence of the generated word;

B3. Input the second word vector sequence into the cross-attention module of the decoder; first, the encoder with Transformer architecture constructs Key and Value values based on the context vector generated by the original text, and outputs the attention distribution of each word in the original text. a ^t , construct the Query value based on the second word vector sequence, and update the second word vector sequence combined with the attention distribution a ^t ; then, perform residual processing on the input second word vector sequence and the updated second word vector sequence. Difference connection and normalization to obtain the third word vector sequence of the generated words;

B4. Input the third word vector sequence into the fact attention module of the decoder; first, based on the attention vector and the third word vector sequence of each fact triple, use the cross attention mechanism to obtain all fact triple pairs The influence coefficient of each generated word; then, the input third word vector sequence and the influence coefficient of all fact triples on each generated word are connected by residuals according to the word correspondence, and normalized to obtain the generated word's third Four word vector sequence;

B5. Input the fourth word vector sequence into the feedforward network module of the decoder; first, use the feedforward network to calculate according to the formula p _t =max(0,ln _t W ₁ +b ₁ )W ₂ +b ₂ ; Then, the output p _t of the feedforward network layer is residually connected to the fourth word vector sequence ln _t and normalized to obtain the output of the decoder Among them, W ₁ , W ₂ , b ₁ and b ₂ are all learnable parameters of the feedforward network.

Furthermore, based on the attention vector and third word vector sequence of each fact triplet, the cross-attention mechanism is used to obtain the influence coefficient of all fact triplets on each generated word, specifically including:

in, represents the word vector of the m-th generated word in the third word vector sequence; α _im represents the correlation between the m-th generated word and the i-th fact triple, β _im represents the m-th generated word and the i-th fact triple The correlation weight of the tuple, u _m represents the influence coefficient of all fact triples on the m-th generated word, and F represents the set of fact triples.

Further, the text summary generation model also includes a pointer network;

At each time step, the processing process of the pointer network includes:

First, according to the following formula, use the linear layer to convert the output of the decoder at the current time step t Feature space mapped to vocabulary:

Among them, W _vocab and b _vocab represent the learnable parameter matrix corresponding to the word list, and the word list is the word list of the pre-trained model;

Then, based on l _t , calculate the vocabulary distribution of the current time step t and pointer probability/>

Among them, w _gen and b _gen represent learnable parameters; Represents the probability that the generated word at the current time step is generated from the word list,/> Represents the output probability distribution of each word in the word list as a generated word at the current time step;

Afterwards, based on and/> Calculate the final probability distribution P _t (w) for the current time step t:

Among them, P _t (w) represents the probability that word w in the expanded vocabulary is a generated word at the current time step t. The expanded vocabulary includes all words contained in the word list and the original text, and N represents the number of words in the original text. n represents the sequence number of the word in the original text, The attention distribution of each word vector of the original text generated by the cross attention module a The attention of the nth word in ^t ,/> Represents the attention sum of word w in the original text.

Further, the text summary generation model also includes a beam search algorithm. At each time step, the processing process of the beam search algorithm includes:

First, use the final probability distribution P _t (w) output by the pointer network at the current time step t to screen the P words with the highest probability to construct candidate words;

Then, make a judgment. If the candidate words contain words contained in the original text, select the word contained in the original text with the highest output probability as the generated word at the current time step; otherwise, use P candidate words and the current time step The generated words that have been generated before constitute P candidate summaries, and the combination probability of each candidate summary is calculated according to the following formula, and the candidate word corresponding to the candidate summary with the highest combination probability is selected as the generated word at the current time step:

P _Y =argmax(log(Π _t p(y _t |y ₁ ,y ₂ ,…,y _t-1 )))

Among them, Y represents the candidate summary, and p represents the output probability of the corresponding word.

The beneficial effects of the present invention are:

The present invention uses the Transformer architecture to build a sequence-to-sequence text summary generation model, and improves the decoder of the Transformer architecture, introducing a fact attention module between the feedforward network module and the cross-attention module of the decoder. The module calculates the influence of each fact triplet on the generated word based on the attention vector of each fact triplet and the word vector of the generated word output by the cross-attention module in the decoder, and updates the word vector of the generated word based on this; The calculation of the attention vector of the fact triplet is calculated through the self-attention mechanism based on the encoding vector of the fact triplet. Therefore, it is possible to avoid ignoring the problem of different importance levels among fact triplets contributing to the final summary result, and to improve the factual consistency of text summary generation.

Furthermore, considering the long text situation, there are also complex relationships between fact triples, and this complex relationship may also lead to different fact triples contributing differently to the final result. Therefore, the influence of complex relationships between different fact triples is also incorporated into the calculation process of the fact triple encoding vector.

Furthermore, the present invention also introduces a pointer network, which maps the output vector of the current time step decoder to the feature space of the word list through the pointer network, obtains the mapping vector representation, and calculates the generated words of the current time step from the word list based on the mapping vector representation. The probability generated in the word list and the output probability distribution of each word in the word list as a generated word at the current time step are then calculated, and then the probability of each word in the expanded word list composed of the word list and the original words is finally used as a generated word at the current time step; traditional decoding is avoided. In order to obtain the distribution probability of the current generated word, the process directly maps the previous decoding information to the word list, which may introduce factual errors in the generated summary due to words that do not belong to the original text but belong to the word list. That is, the present invention uses the introduction of pointer networks to try to avoid generating words that do not belong to the original text but belong to the word list, thereby reducing the probability of factual errors and thereby improving the factual consistency of text summary generation.

Furthermore, the present invention also introduces a beam search algorithm, which uses the beam search algorithm to use the probability of each word in the expanded word list output by the current time step pointer network to eventually become a generated word at the current time step, and selects multiple words with the highest probability to construct candidate words. , if the candidate word contains a word contained in the original text, then select the word contained in the original text and with the highest output probability as the generated word at the current time step; otherwise, use multiple candidate words and the generated words generated before the current time step. words to form multiple candidate summaries, and calculate the combination probability of each candidate summary, and select the candidate word corresponding to the candidate summary with the highest combination probability as the generated word at the current time step. That is, through the beam search algorithm, when the calculated candidate words do not contain words in the original text, the present invention uses the candidate words and the generated words before the current time step to construct candidate summaries, and calculates the combination probability of each candidate summary, Thus, the approximate global optimal solution in the relative sense is found at the phrase level, thereby improving the rationality of the semantic combination of generated words and generated words at the current time step, and thereby improving the factual consistency of text summary generation.

Description of drawings

Figure 1 is a principle framework diagram of text summary generation based on fact consistency enhancement in an embodiment of the present invention.

Detailed ways

The present invention aims to propose a text summary generation method based on enhanced fact consistency, solve the problem that the existing technology ignores the different importance between fact triples and contribute different contributions to the final summary result, and improve the credibility of the generated text summary. .

The text summary generation method of the present invention adopts the Transformer architecture to build a sequence-to-sequence text summary generation model. The Transformer architecture is an Encoder-Decoder framework. In the encoding part, first, the input original text is segmented to obtain the word sequence of the original text; then, through the pre-training model, the word vector of each word in the word sequence is obtained to form the original text. The word vector sequence; finally, the word vector sequence is input into the encoder of the Transformer architecture, and the context vector of the original text is obtained after encoding by the encoder. In the decoding part, based on the context vector obtained by encoding, the summary text is gradually generated, that is, the generated words are obtained one by one according to time steps, and one word is generated by decoding at each time step.

The key to the present invention is to improve the decoder of the Transformer architecture. Between the feedforward network module and the cross-attention module of the existing decoder, a fact attention module is introduced, that is, the decoder in the present invention, It includes self-attention module, cross-attention module, factual attention module and feed-forward network module connected in sequence. The fact attention module calculates the influence of each fact triplet on the generated word based on the attention vector of each fact triplet and the word vector of the generated word output by the cross attention module, and updates the word vector of the generated word based on this.

Specifically, in the method of the present invention, at each time step, the decoding process of the decoder is:

First, all the words generated before the current time step are used through the pre-training model to obtain the first word vector sequence of the generated words; then, the self-attention module is used to update the first word vector sequence and compare it with the input first word vector The sequence is residually connected and normalized to obtain the second word vector sequence of the generated word; then, based on the cross-attention module, the second word vector sequence is updated using the original text context vector obtained in the encoding part, and combined with the second word vector sequence The two word vector sequences are residually connected and normalized to obtain the third word vector sequence of the generated words; then, based on the attention vector and third word vector sequence of each fact triplet, the fact attention module is used to The cross-attention mechanism is used to obtain the influence coefficients of all fact triples on each generated word, and the residual connection and normalization are carried out corresponding to the third word vector sequence by word to obtain the fourth word vector of the generated word. Sequence, use the fourth word vector sequence as the input of the feedforward network to obtain the decoding output vector of the current time step.

The above decoding process obtains the distribution probability of the word at the current time step, which can be directly mapped to the word list to obtain the generated words; however, this method may introduce words that do not belong to the original text but belong to the word list to a certain extent in the generated summary. Factual errors may occur due to insufficient accuracy in calculating distribution probability. In order to effectively alleviate this problem, the present invention introduces a pointer network. Through the pointer network, the decoding output vector of the current time step is mapped to the feature space of the word table to obtain the mapping vector representation. According to the mapping vector representation, the generated word of the current time step is calculated from The probability generated in the word list and the output probability distribution of each word in the word list as a generated word at the current time step are then calculated, and then the probability distribution of each word in the expanded word list composed of all words contained in the word list and the original text is finally used as a generated word at the current time step. Probability.

Furthermore, the method of the present invention also introduces the beam search method, through the calculation of the probability of the combination of the current generated word and the generated word, to find the approximate global optimal solution in the relative sense at the phrase level, and improve the generated words at the current time step. The rationality of the semantic combination with the generated words, and try to avoid generating words that do not belong to the original text but belong to the word list, thereby further improving the factual consistency and credibility of text summary generation. The beam search algorithm uses the probability that each word in the expanded word list output by the current time step pointer network will eventually become a generated word at the current time step, and selects multiple words with the highest probability to construct candidate words. If the candidate words include words contained in the original text, Then select the word contained in the original text and with the highest output probability as the generated word at the current time step; otherwise, use multiple candidate words and the generated words generated before the current time step to form multiple candidate summaries, and calculate each candidate For the combination probability of the summary, select the candidate word corresponding to the candidate summary with the highest combination probability as the generated word at the current time step.

The method of the present invention will be further described below with reference to examples.

Example:

The text summary generation method based on enhanced fact consistency in this embodiment, its text summary generation model, as shown in Figure 1, is based on the Encoder-Decoder framework of the Transformer architecture. Compared with the existing technology, its encoder Encoder The structure remains unchanged, but in the decoder Decoder, a fact attention module is introduced. The improved decoder Decoder includes a self-attention module, a cross-attention module, a fact attention module and a feed-forward network module.

The following is a detailed explanation from three aspects: fact triple attention vector generation, encoding and decoding process, and model training.

1. Fact triple attention vector generation

The attention vector of the fact triplet is to introduce the weight of the fact triplet in the subsequent decoding process and calculate the different contributions of each fact triplet to the generated word, including the following process:

A1. Extract fact triples

In this step, the natural language processing tool is used to process the original text, extract the fact triplet F _i from the original text, and construct the fact triplet set F. In this embodiment, the natural language processing tool is the StanfordNLP toolkit.

Specifically, first, input the original text D into the StanfordNLP toolkit, perform word segmentation, and obtain the word sequence {x ₁ , x ₂ ,..., x _b ,..., x _N } of the original text, where x _n represents the word sequence in the text n words, n represents the sequence number of the words in the original text, and N represents the number of words in the original text.

Then, the StanfordNLP toolkit is used to output the fact triplet F _i extracted from the original text D through processes such as part-of-speech tagging, syntactic component analysis, coreference resolution, and relationship extraction, and construct a fact triplet set F. Set F = {F ₁ ,F ₂ ,…,Fi _, …,F _I }, fact triplet F _i =<s _i ,ri _, o _i >, where s is the subject word and r is the relation modification word, o is the object word, the subscript i is the sequence number of the fact triplet, I represents the number of fact triplet, and s _i , r _i and o _i in any fact triplet F _i all come from the same A sentence. For example, <Kobe, like, Coke> is a fact triple.

A2. Extract relationship triples

In this step, rhetorical structure theory is applied to extract complex relationships between fact triples based on fact triples. Furthermore, the decision tree model of the natural language processing tool is used to classify the relationship between each fact triplet F _i based on the relationship type it contains; then, based on the relationship classification result, the fact triplet F is obtained The relationship triplet between _i constructs the relationship triplet set R.

In this embodiment, the natural language processing tool is the StanfordNLP toolkit. Since the RST tree model in the rhetorical structure theory has been built into the StanfordCoreNLP toolkit, it can be called directly. Specifically, the original text D={x ₁ ,x ₂ ,…,x _n ,…,x _N } and the set of fact triples F = {F ₁ ,F ₂ ,…, _Fi ,…,F _I }, enter the StanfordCoreNLP toolkit. Use the StanfordCoreNLP toolkit to classify the relationships between each fact triplet F _i based on the relationship types contained in its RST tree model. Currently, there are 23 types of relationships included in the RST tree model; then, based on the relationship classification results, Obtain the relationship triplet R _j between the fact triplet F _i and construct the relationship triplet set R.

Set R = {R ₁ , R ₂ ..., R _j , ... R _J }, relationship triplet R _j = <F _1j , CR _j , F _2j >, where F _1j and F _2j respectively represent the jth relationship The fact triplet contained in the triplet, CR _j represents the relationship contained in the j-th relationship triplet, the subscript j is the sequence number of the relationship triplet, and J represents the number of relationship triplet. For example: the fact triplet <Xiao Ming, born in Sichuan> and <Xiao Ming, master, Sichuan dialect> are "background" relationships, and <Jack, like, Coke> and <Jack, suffer from, tooth decay> are "causality" relationships. .

A3. Vector encoding

In this step, the set of fact triples is mapped into a graph structure, and the graph convolution network is used to encode the nodes of the graph structure, and complex relationship information is integrated to obtain the encoding vector of each fact triple.

Specifically, the following processes are included:

A31. Map the set of fact triples into a graph

In order to use graph convolutional neural networks for encoding, the unstructured set of fact triples F needs to be converted into a structured graph. For each fact triple F _i =<s _i ,ri _, o _i >, create subject node s _i , relationship node _ri and object node o _i ; the subject node and relationship node are connected through edges, and the object node is connected with the relationship Nodes are also connected through edges, and edges do not represent any information, only that there is a connection between the two. That is, the words contained in each fact triple F _i are used to construct nodes and connections to construct edges, and the set of fact triples F is mapped into a graph.

A32, node initialization

This step uses the pre-trained Bert model to initialize the node vectors of each node in the graph.

A33, node update

This step uses the GCN network to update the node vectors of each node in the graph:

GCN network, for any node, it will collect the information of neighboring nodes around itself, and use this information to transform its own characteristic information. This process will be repeated many times. The specific calculation process is as follows:

Among them, ReLU represents the activation function, H ^l and H ^l+1 represent the outputs of the l and l+1 layers of the GCN network respectively, and A represents the adjacency matrix of the graph. represents the degree matrix, and W ^l represents the weight matrix of the lth layer of the GCN network.

After GCN network aggregation calculation, each node contains information about surrounding nodes and even nodes two or three hops away, effectively capturing associated information and including factual relationships to a certain extent. In this embodiment, the GCN network has two layers.

A34. Construct fact triple encoding features

In this step, the node vectors of the nodes contained in each fact triplet are spliced to obtain the coding characteristics of each fact triplet. Among them,/> and/> Respectively represent the node vectors of the nodes corresponding to s _i , r _i and o _i contained in the fact triplet F _i .

A35. Update fact triple encoding features

In order to effectively integrate the impact of complex relationships between fact triples on the information contained in different fact triples, this step first constructs the coding features of each relationship triplet, and then uses the attention mechanism to update the facts based on the coding features of the relationship triplet. Encoding features of triples.

Specifically, first, the coding features and relationship vectors of the fact triples contained in each relationship triplet are spliced to obtain the coding features of the relationship triplet R _j = [h _1j , cr _j , h _2j ], h _1j and h _2j respectively represent the encoding features of the fact triples contained in the j-th relationship triplet, and cr _j represents the relationship vector obtained by the relationship contained in the j-th relationship triplet based on the pre-training model.

Then, the coding features of each relationship triplet and the coding features of each fact triplet are used to perform cross-attention calculations, and the coding features of each fact triplet are updated based on the calculation results to obtain the updated fact triplet. The encoding feature h′ _i , the calculation process is as follows:

α _ij =h _i *R _j

Among them, α _ij expresses the correlation between the i-th fact triplet and the j-th relationship triplet, β _ij expresses the correlation weight between the i-th fact triplet and the j-th relationship triplet, and R represents the relationship. Collection of triples.

A4. Fusion of front and back semantics

For the fact triple coding features obtained previously, although it implies certain relationship structure information between fact triples, it only contains local structural information. In the case of long texts, the ability to capture long-distance fact relationships is insufficient. In order to solve the problem of long text For the distance dependence problem, in this step, the encoding features of the fact triples are input into the recurrent neural network to obtain the vector representation z _i of the semantic information of the fact triples before and after the fusion of each fact triplet.

Specifically, in this embodiment, the coding features h′ _i of the fact triples updated in step A35 are input into the Bi-LSTM network to obtain the semantic information of each fact triplet that is integrated with the previous and later fact triplets. The vector represents z _i . Furthermore, using the Bi-LSTM network, based on the forward LSTM, the forward hidden layer vector is obtained Based on backward LSTM, obtain the backward hidden layer vector/> Then, convert the forward hidden layer vector/> and backward hidden layer vector/> By performing vector splicing, the vector representation z _i of the semantic information of fact triples before and after fusion can be obtained. The formula is as follows:

A5. Self-attention calculation

In this step, the vector representation z _i of each fact triple is used, and the attention vector of each fact triple is obtained based on the self-attention mechanism. In order to improve the ability to capture different information and thereby improve the effect of attention calculation, the self-attention mechanism is multi-head self-attention, specifically as follows:

A51. Based on the transformation matrices W _k1 , W _k2 and W _k3 of each attention head, convert the vector representation z _i of the fact triplet into a Query value Key value/> and Value/>

A52, based on and/> Calculate the attention of each attention head:

Among them, _dk is dimensions;

A53. Aggregate the attention calculation results of each attention head to obtain the attention vector Z _i :

Among them, K is the number of attention heads;

A54. Use linear transformation to change the dimension of the attention vector Z _i to the dimension of the vector representation z _i , and obtain the attention vector r _i of the fact triplet;

r _i =Z _i *W ⁰

Among them, W ⁰ is the linear transformation matrix.

2. Coding process

For the encoding process, first, segment the original text to obtain the word sequence of the original text; then, by pre-training the Bert model, extract the word vectors of each word in the word sequence of the original text, and obtain the word vector sequence of the original text; after that, For the word vector sequence of the original text, the context vector of the original text is obtained through encoding by the encoder. Since the encoding process is consistent with the encoding process of the Transformer architecture in the prior art, it will not be described again here.

3. Decoding process

In the decoding process, the generated words are obtained one by one according to time steps, that is, one word is generated by decoding at each time step.

The decoding of each time step includes the following steps:

B1. Get the first word vector sequence of words generated before the current time step:

In this step, based on each generated word obtained before the current time step t, the pre-trained Bert model is used to obtain its word embedding, and the word embedding of each generated word is used to form the first word vector sequence of the generated words in the order of generation. .

B2. Perform self-attention calculation on the first word vector sequence to obtain the second word vector sequence of the generated words:

In this step, the first word vector sequence is input into the self-attention module of the decoder; first, the first word vector sequence is updated based on the self-attention mechanism; then, in order to alleviate the vanishing gradient problem of deep network training and speed up the network To converge, the input first word vector sequence and the updated first word vector sequence are residually connected and normalized to obtain the second word vector sequence of the generated word.

B3. Use the context vector of the original text to update the second word vector sequence based on the cross-attention mechanism to obtain the third word vector sequence of the generated word:

In this step, the second word vector sequence is input into the cross-attention module of the decoder; first, the encoder with Transformer architecture constructs Key and Value values based on the context vector generated by the original text (obtained in the encoding part), and outputs the original text Attention distribution a ^t of each word in , the Query value is constructed with the second word vector sequence, and combined with the attention distribution a ^t , the second word vector sequence is updated; then, the input second word vector sequence and the updated The second word vector sequence is residually connected and normalized to obtain the third word vector sequence of the generated word.

B4. Based on the attention vector and third word vector sequence of each fact triple, use the cross attention mechanism to calculate the influence coefficient of all fact triples on each generated word, and obtain the fourth word vector sequence of the generated word :

In this step, the third word vector sequence is input into the fact attention module of the decoder; first, based on the attention vector and the third word vector sequence of each fact triple, the cross attention mechanism is used to obtain all fact triples The influence coefficient of the group on each generated word; then, the input third word vector sequence and the influence coefficient of all fact triples on each generated word are connected by residuals corresponding to the words and normalized to obtain the generated words The fourth word vector sequence.

Among them, the cross-attention mechanism is used to calculate the influence coefficient of fact triples on each generated word, as follows:

First, calculate the correlation between each fact triple and each generated word:

Among them, α _im represents the correlation between the m-th generated word and the i-th fact triplet, Represents the word vector of the m-th generated word in the third word vector sequence;

Next, calculate the correlation weight between each generated word and each fact triple:

Among them, β _im represents the correlation weight between the m-th generated word and the i-th fact triple, and F represents the set of fact triples; then, calculate the influence coefficient of the fact triples on each generated word:

u _m represents the influence coefficient of all fact triples on the m-th generated word.

B5. Use the fourth word vector sequence as the input of the feedforward network to obtain the decoding output vector of the current time step:

In this step, input the fourth word vector sequence into the feedforward network module of the decoder; first, use the feedforward network to proceed according to the formula p _t =max(0,ln _t W ₁ +b ₁ )W ₂ +b ₂ Calculate; then, residual connection is made between the output p _t of the feedforward network layer and the fourth word vector sequence ln _t , and normalized to obtain the output of the decoder. Among them, W ₁ , W ₂ , b ₁ and b ₂ are all learnable parameters of the feedforward network.

B6. Based on the decoding output vector, use the pointer network to predict the words in the word list and the words in the original text as the probability of generating words at the current time step:

In order to obtain the distribution probability of the current generated word, the traditional decoding process directly maps the previous decoding information to the word list, which may introduce factual errors in the generated summary due to words that do not belong to the original text but belong to the word list. In order to minimize the resulting factual errors, in this step, a pointer network is introduced to try to avoid generating words that do not belong to the original text but belong to the word list, thereby reducing the probability of factual errors.

Specifically, first, according to the following formula, use the linear layer to convert the output of the decoder at the current time step t Feature space mapped to vocabulary:

Among them, W _vocab and b _vocab represent the learnable parameter matrix corresponding to the word list, and the word list is the word list of the pre-trained model;

Next, calculate the vocabulary distribution of the current time step t based on the mapping vector representation l _t and pointer probability/>

Among them, w _gen and b _gen represent learnable parameters; Represents the probability that the generated word at the current time step is generated from the word list,/> Represents the output probability distribution of each word in the word list as a generated word at the current time step;

Finally, based on and/> Calculate the final probability distribution P _t (w) for the current time step t:

Among them, P _t (w) represents the probability that word w in the expanded vocabulary is a generated word at the current time step t. The expanded vocabulary includes all words contained in the word list and the original text, and N represents the number of words in the original text. n represents the sequence number of the word in the original text, The attention distribution of each word vector of the original text generated by the cross attention module a The attention of the nth word in ^t ,/> Represents the attention sum of word w in the original text.

B7. Based on the predicted words in the word list and the words in the original text as the probability of generating words at the current time step, the beam search algorithm is used to obtain the generated words at the current time step:

In this step, since the words in the word list are foreign words to the original text, and the foreign words are more prone to factual errors than the words in the original text when generating the summary, therefore, the beam search algorithm is useful when it is really necessary to introduce foreign words. , calculate the combination probability between the foreign words to be introduced and the generated words, so as to judge the rationality of the words.

Specifically, the processing includes:

First, use the final probability distribution P _t (w) output by the pointer network at the current time step t to screen the P words with the highest probability to construct candidate words;

Then, make a judgment. If the candidate words contain words contained in the original text, select the word contained in the original text with the highest output probability as the generated word at the current time step; otherwise, use P candidate words and the current time step The previously generated generated words form M candidate summaries, and the combination probability of each candidate summary is calculated according to the following formula, and the candidate word corresponding to the candidate summary with the highest combination probability is selected as the generated word at the current time step:

P _Y =argmax(log(Π _t p(y _t |y ₁ ,y ₂ ,…,y _t-1 )))

Among them, Y represents the candidate summary, and p represents the output probability of the corresponding word.

The above steps B1-B7 are a time step generation process. One time step outputs one generated word. By executing the above generation process in a loop until the predetermined number of summary generated words is reached, the generated summary can be obtained.

4. Model training

The training process of the text summary generation model in this embodiment is as follows:

C1. From the training set, select a sample as the current training sample. The sample includes the original text and its real summary; specifically, common data sets in the field of text summarization can be used as basic data, such as the CNN/DailyMail data set as training set;

C2. Input the original text of the current training sample into the text summary generation model;

Use the encoder of the text summary generation model to obtain the context vector of the original text;

The text summary generation model generates generated words at each time step one by one according to steps B1 to B7. Finally, the generated words at each time step form a generated summary of the original text of the training sample, and the number of generated words is the same as the real summary. The number of words is consistent;

C3. Calculate the loss according to the following loss function, and update the text summary generation model based on the loss:

Among them, y _m represents the word vector of the m-th word in the real summary, Represents the word vector of the m-th generated word in the generated summary, and T represents the matrix transpose;

C4. Repeat steps C1 to C3 until the training of the text summary generation model is completed.

Finally, it should be noted that the above-mentioned embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those of ordinary skill in the art, several modifications, equivalent substitutions, improvements, etc. can be made without departing from the spirit of the present invention and the scope protected by the claims, which should all be included in the scope of the present invention. within the scope of protection.

Claims (10)

1. A text summary generation method based on enhanced fact consistency, using the Transformer architecture to build a sequence-to-sequence text summary generation model. The decoder of the Transformer architecture includes a self-attention module, a cross-attention module and a front-end module connected in sequence. Feed network module, which is characterized by: A fact attention module is introduced between the feedforward network module and the cross-attention module of the decoder; Define the fact triplet and express it as F _i =<s _i , r _i , o _i >, where s is the subject word, r is the relational modifier, o is the object word, and the subscript i is the fact triplet. serial number, and each word in any fact triplet comes from the same sentence in the original text; the fact attention module uses the attention vector of each fact triplet in the original text, and the decoder cross The third word vector sequence output by the attention module is used as input. Based on the cross-attention mechanism, the influence coefficients of all fact triples on each generated word are obtained, and the influence coefficients of all fact triples on each generated word are calculated for the third word vector sequence. The word vector sequence is updated to obtain a fourth word vector sequence, which is used as the input of the decoder feedforward network module; The attention vector of the fact triplet is calculated as follows: A1. Use natural language processing tools to process the original text, extract fact triples F _i , and construct a set of fact triples F; then, use the words contained in each fact triplet F _i to construct nodes and connections to construct edges. Map the set of fact triples F into a graph, and use the graph neural network to calculate the node vectors of each node in the graph; then, splice the node vectors of the nodes contained in each fact triplet to obtain each fact triplet. encoding features Among them,/> and/> Respectively represent the node vectors of the nodes corresponding to si _, r _i and o _i contained in the fact triplet F _i ; A2. Input the coding features of the fact triples into the recurrent neural network, and obtain the vector representation z _i of the semantic information of the fused fact triples before and after each fact triple; A3. Use the vector representation z _i of each fact triplet and obtain the attention vector of each fact triplet based on the self-attention mechanism. 2. A text summary generation method based on fact consistency enhancement as claimed in claim 1, characterized in that: In step A2, the recurrent neural network is a Bi-LSTM network; based on the coding characteristics of the fact triplet, the Bi-LSTM network is used to obtain the forward hidden layer vector of each fact triplet. and backward hidden layer vector/> Then, convert the forward hidden layer vector/> and backward hidden layer vector/> Perform vector splicing to obtain the vector representation of the semantic information of each fact triplet before and after the fusion of each fact triplet/> 3. A text summary generation method based on fact consistency enhancement as claimed in claim 1, characterized in that: In step A1, the graph neural network is used to calculate the node vectors of each node in the graph, including: First, use the pre-trained model to initialize the node vectors of each node in the graph; Then, use the GCN network to update the node vectors of each node in the graph according to the following formula: Among them, ReLU represents the activation function, H ^l and H ^l+1 represent the outputs of the l and l+1 layers of the GCN network respectively, and A represents the adjacency matrix of the graph. represents the degree matrix, and W ^t represents the weight matrix of the lth layer of the GCN network. 4. A text summary generation method based on fact consistency enhancement as claimed in claim 1, characterized in that: Step A1 also includes: First, use the decision tree model of the natural language processing tool to classify the relationship between each fact triplet F _i based on the relationship type it contains; then, based on the relationship classification results, obtain the relationship between the fact triplet F _i The relationship triplet between them constructs the relationship triplet set R; Then, the coding features and relationship vectors of the fact triples contained in each relationship triplet are spliced to obtain the coding features of the relationship triplet R _j = [h _1j , cr _j , h _2j ], h _1j and h _2j Respectively represent the encoding characteristics of the fact triples contained in the j-th relationship triplet, cr _j represents the relationship vector obtained by the relationship contained in the j-th relationship triplet based on the pre-training model; After that, the coding features of each relationship triplet and the coding features of each fact triplet are used to perform cross-attention calculations. Based on the calculation results, the coding features of each fact triplet are updated, and the updated fact triplet is The encoding feature h′ _i is used as the input of step A2. 5. A text summary generation method based on fact consistency enhancement as claimed in claim 4, characterized in that: The coding features of each relationship triplet and the coding features of each fact triplet are used to perform cross-attention calculations, and the coding features of each fact triplet are updated based on the calculation results, including: α _ij =h _i *R _j Among them, α _ij expresses the correlation between the i-th fact triplet and the j-th relationship triplet, β _ij expresses the correlation weight between the i-th fact triplet and the j-th relationship triplet, and R represents the relationship. Collection of triples. 6. A text summary generation method based on enhanced fact consistency according to any one of claims 1 to 5, characterized in that, in step A3, vector representation z _i of each fact triple is used, based on self-attention force mechanism to obtain the attention vector of each fact triplet; the self-attention mechanism is multi-head self-attention, and its calculation process includes: A31. Based on the transformation matrices W _k1 , W _k2 and W _k3 of each attention head, convert the vector representation z _i of the fact triplet into a Query value Key value/> and Value/> A32, based on and/> Calculate the attention of each attention head: Among them, d _k is dimensions; A33. Aggregate the attention calculation results of each attention head to obtain the attention vector Z _i : Among them, K is the number of attention heads; A34. Use linear transformation to change the dimension of the attention vector Z _i to the dimension of the vector representation z _i , and obtain the attention vector r _i of the fact triplet; r _i =Z _i *W ⁰ Among them, W ⁰ is the linear transformation matrix. 7. A text summary generation method based on fact consistency enhancement according to claim 1, characterized in that, at each time step, the decoding process of the decoder is: B1. Based on each generated word obtained before the current time step t, use the pre-training model to obtain its word embedding, and use the word embedding of each generated word to form the first word vector sequence of the generated words in the order of generation; B2. Input the first word vector sequence into the self-attention module of the decoder; first, update the first word vector sequence based on the self-attention mechanism; then, combine the input first word vector sequence with the updated first word vector sequence. The word vector sequence is residually connected and normalized to obtain the second word vector sequence of the generated word; B3. Input the second word vector sequence into the cross-attention module of the decoder; first, the encoder with Transformer architecture constructs Key and Value values based on the context vector generated by the original text, and outputs the attention distribution of each word in the original text. a ^t , construct the Query value based on the second word vector sequence, and update the second word vector sequence combined with the attention distribution a ^t ; then, perform residual processing on the input second word vector sequence and the updated second word vector sequence. Difference connection and normalization to obtain the third word vector sequence of the generated words; B4. Input the third word vector sequence into the fact attention module of the decoder; first, based on the attention vector and the third word vector sequence of each fact triplet, use the cross attention mechanism to obtain all fact triplet pairs The influence coefficient of each generated word; then, the input third word vector sequence and the influence coefficient of all fact triples on each generated word are connected by residuals according to word correspondence, and normalized to obtain the generated word's third Four word vector sequence; B5. Input the fourth word vector sequence into the feedforward network module of the decoder; first, use the feedforward network to calculate according to the formula p _t =max (0, ln _t W ₁ +b ₁ )W ₂ +b ₂ ; Then, the output p _t of the feedforward network layer is residually connected to the fourth word vector sequence ln _t and normalized to obtain the output of the decoder Among them, W ₁ , W ₂ , b ₁ and b ₂ are all learnable parameters of the feedforward network. 8. A text summary generation method based on fact consistency enhancement according to claim 1 or 7, characterized in that: Based on the attention vector and third word vector sequence of each fact triplet, the cross-attention mechanism is used to obtain the influence coefficients of all fact triplets on each generated word, including: in, represents the word vector of the m-th generated word in the third word vector sequence; α _im represents the correlation between the m-th generated word and the i-th fact triple, β _im represents the m-th generated word and the i-th fact triple The correlation weight of the tuple, u _m represents the influence coefficient of all fact triples on the m-th generated word, and F represents the set of fact triples. 9. A text summary generation method based on fact consistency enhancement as claimed in claim 7, characterized in that: The text summary generation model also includes a pointer network; At each time step, the processing process of the pointer network includes: First, according to the following formula, use the linear layer to convert the output of the decoder at the current time step t Feature space mapped to vocabulary: Among them, W _vocab and b _vocab represent the learnable parameter matrix corresponding to the word list, and the word list is the word list of the pre-trained model; Then, based on l _t , calculate the vocabulary distribution of the current time step t and pointer probability/> Among them, w _gen and b _gen represent learnable parameters; Represents the probability that the generated word at the current time step is generated from the word list,/> Represents the output probability distribution of each word in the word list as a generated word at the current time step; Afterwards, based on and/> Calculate the final probability distribution P _t (w) for the current time step t: Among them, P _t (w) represents the probability that word w in the expanded vocabulary is a generated word at the current time step t. The expanded vocabulary includes all words contained in the word list and the original text, and N represents the number of words in the original text. n represents the sequence number of the word in the original text, The attention distribution of each word vector of the original text generated by the cross attention module a The attention of the nth word in ^t ,/> Represents the attention sum of word w in the original text. 10. A text summary generation method based on fact consistency enhancement as claimed in claim 9, characterized in that: The text summary generation model also includes a beam search algorithm. At each time step, the processing process of the beam search algorithm includes: First, use the final probability distribution P _t (w) output by the pointer network at the current time step t to screen the P words with the highest probability to construct candidate words; Then, make a judgment. If the candidate words contain words contained in the original text, select the word contained in the original text with the highest output probability as the generated word at the current time step; otherwise, use P candidate words and the current time step The generated words that have been generated before constitute P candidate summaries, and the combination probability of each candidate summary is calculated according to the following formula, and the candidate word corresponding to the candidate summary with the highest combination probability is selected as the generated word at the current time step: P _Y =argmax(log(Π _t p(y _t |y ₁ , y ₂ ,..., y _t-1 ))) Among them, Y represents the candidate summary, and p represents the output probability of the corresponding word.