JP7414357B2 - Text processing methods, apparatus, devices and computer readable storage media - Google Patents

Description

TECHNICAL FIELD The present disclosure relates to the field of text processing, and specifically relates to text processing methods, apparatus, devices, and computer-readable storage media.

In the conventional text generation process, the output content of the text generation network is the result of learning training data. For example, in a scene that generates text such as a summary, many of the correct answers in the training data are concentrated in the previous few sentences in the text content, so the network trained using such training data also , tends to generate new text content about the previous sentence in the text content. Therefore, current text processing methods do not provide efficient means for summarizing and extracting text content.

The present disclosure provides text processing methods, apparatus, devices, and computer-readable storage media for efficiently extracting and generating summaries from text.

In one aspect of the present disclosure, a preprocessing unit configured to perform preprocessing on a source text to generate a plurality of word vectors for a plurality of words; a plurality of initial recommendation weight vectors; a sentence vector determining unit arranged to determine a plurality of sentence vectors based on word vectors of the plurality of sentence vectors; a recommendation probability determining unit arranged to adjust an initial recommendation weight vector of to determine a recommendation probability distribution for the plurality of words; and a recommendation probability determining unit configured to determine a word to be output based on the recommended probability distribution. A text processing device is provided, comprising an output unit disposed.

In some embodiments, the sentence vector determination unit processes the plurality of word vectors using an encoding neural network to determine a current encoded hidden state vector corresponding to each word vector, respectively; Based on each initial recommendation weight vector and the current encoded hidden state vector, the sentence vector corresponding to the initial recommendation weight vector is determined.

In some embodiments, the output unit utilizes a decoding neural network to determine a current decoded hidden state vector based on the current encoded hidden state vector, and determines a current decoded hidden state vector based on the current encoded hidden state vector. A current word probability distribution is determined using the vector and the current decoded hidden state vector, and a word to be output is determined based on the current word probability distribution and the recommended probability distribution. .

In some embodiments, the current word probability distribution includes a generation probability distribution and an attention probability distribution, and the output unit adjusts the attention probability distribution using the recommended probability distribution, and adjusts the attention probability distribution after adjusting the attention probability distribution. Determine the probability distribution, add the generation probability distribution and the adjusted attention probability distribution with weight to determine the output word probability distribution, and determine the word with the maximum probability in the output word probability distribution as the word to be output. It is arranged so that

In some embodiments, the current word probability distribution includes a production probability distribution and an attention probability distribution, and the output unit determines weights used for the production probability distribution, the attention probability distribution, and the recommendation probability distribution. Then, the output word probability distribution is determined based on the weights, and the word having the maximum probability of the output word probability distribution is determined as the word to be output.

In some embodiments, the recommendation probability determination unit further includes a relevance determination subunit, and for each sentence vector, the relevance determination subunit combines the sentence vector with other sentence vectors to form a combined sentence. By generating a vector and processing the combined sentence vector using an association matrix, the arrangement is made to determine the association between the sentence vector and the other sentence vector.

In some embodiments, the recommendation probability determination unit further includes an adjustment subunit, and the adjustment subunit is configured to determine the probability based on the association between the sentence vector and each of the other sentence vectors of the plurality of sentence vectors. Then, the recommendation coefficient of the sentence vector is determined, and for each of the initial recommendation weight vectors, the initial recommendation weight vector is adjusted using the recommendation coefficient of the sentence vector corresponding to the initial recommendation weight vector, and after the adjustment, is arranged to obtain a word probability vector of and determine a recommended probability distribution of the plurality of words based on the adjusted word probability vector.

In another aspect of the disclosure, performing preprocessing on a source text to generate a plurality of word vectors for a plurality of words; and based on a plurality of initial recommendation weight vectors and the plurality of word vectors; determining a plurality of sentence vectors and adjusting the plurality of initial recommendation weight vectors based on the relevance of each sentence vector to other sentence vectors among the plurality of sentence vectors; A text processing method is provided that includes determining a recommended probability distribution for a text, and determining words to be output based on the recommended probability distribution.

In some embodiments, determining sentence vectors based on the plurality of initial recommendation weight vectors and the plurality of word vectors includes processing the plurality of word vectors using an encoding neural network. , determine a current encoded hidden state vector corresponding to each word vector, and determine a sentence vector corresponding to the initial recommended weight vector based on each initial recommended weight vector and the current encoded hidden state vector. Including.

In some embodiments, determining the word to be output based on the recommended probability distribution includes determining the current decoded hidden state using a decoding neural network based on the current encoded hidden state vector. determining a current word probability distribution using the current encoded hidden state vector and the current decoded hidden state vector, based on the current word probability distribution and the recommended probability distribution; This includes determining the words to be output.

In some embodiments, the current word probability distribution includes a generation probability distribution and an attention probability distribution, wherein the word to be output is determined based on the current word probability distribution and the recommendation probability distribution. That is, the attention probability distribution is adjusted using the recommended probability distribution, the adjusted attention probability distribution is determined, and the output word probability distribution is determined by weighting and adding the generation probability distribution and the adjusted attention probability distribution. and determining the word with the maximum probability in the output word probability distribution as the word to be output.

In some embodiments, the current word probability distribution includes a generation probability distribution and an attention probability distribution, wherein the word to be output is determined based on the current word probability distribution and the recommendation probability distribution. That is, determining the weights used for the generation probability distribution, the attention probability distribution, and the recommendation probability distribution, determining the output word probability distribution based on the weights, and determining the maximum probability of the output word probability distribution. This includes determining a word as a word to be output.

In some embodiments, the association between each sentence vector and other sentence vectors of the plurality of sentence vectors is determined as follows. In other words, for each sentence vector, the sentence vector is combined with other sentence vectors to generate a combined sentence vector, and the combined sentence vector is processed using the association matrix. Determine the relationship between the sentence vector and the sentence vector.

In some embodiments, the plurality of initial recommendation weight vectors are adjusted based on the relevance of each sentence vector to other sentence vectors of the plurality of sentence vectors to make recommendations for the plurality of words. The recommended probability determining unit further includes an adjustment subunit, and the adjustment subunit determines the relationship between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors. Based on this, the recommendation coefficient of the sentence vector is determined, and for each of the initial recommendation weight vectors, the initial recommendation weight vector is adjusted using the recommendation coefficient of the sentence vector corresponding to the initial recommendation weight vector. obtaining a subsequent word probability vector and determining a recommended probability distribution for the plurality of words based on the adjusted word probability vector.

Still other aspects of the present disclosure include a processor and a memory in which computer readable program instructions are stored, the computer readable program instructions, when executed by the processor, performing text processing as described above. A text processing device is provided that performs the method.

In yet another aspect of the disclosure, a computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions, when executed by a computer, prompting the computer to write text as described above. A computer readable storage medium is provided for carrying out the processing method.

According to the text processing method, apparatus, device, and computer-readable storage medium according to the present disclosure, the text content is extracted by a text summary extraction method based on the relationship between each word in the text and a sentence consisting of each word. The content of the text can be better abstracted and summarized, and a text summary can be generated.

The above and other objects, features, and advantages of the present invention will become apparent from a more detailed description based on the embodiments of the present invention described below and the accompanying drawings. The drawings are used to provide a further understanding of the embodiments of the disclosure, constitute a part of the specification, and together with the embodiments of the disclosure serve to explain the disclosure and limit the disclosure. It's not a thing. Note that in the drawings, the same reference numerals indicate the same components or steps.
1 shows a schematic flowchart of a text processing method according to the present disclosure. FIG. 6 shows a schematic diagram for determining the relationship between each sentence vector and other sentence vectors among the plurality of sentence vectors, according to an embodiment of the present disclosure. FIG. 6 shows a schematic diagram of determining an output word probability distribution according to an embodiment of the present disclosure. FIG. 6 shows a schematic diagram of determining an output word probability distribution using a generation probability distribution and an adjusted attention probability distribution according to an embodiment of the present disclosure. FIG. 6 shows a schematic diagram of determining an output word probability distribution using a generation probability distribution, an attention probability distribution, and a recommendation probability distribution, according to an embodiment of the present disclosure. 1 shows a schematic block diagram of a text processing device, according to an embodiment of the present disclosure; FIG. 1 is a schematic diagram of a computing device, according to an embodiment of the present disclosure; FIG.

Hereinafter, the technical solutions in the embodiments of the present disclosure will be clearly and completely explained in conjunction with the accompanying drawings in the embodiments of the present disclosure. Of course, the described embodiments are only some, but not all, embodiments of the present disclosure. All other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without any creative efforts fall within the protection scope of the present disclosure.

Unless otherwise defined, technical or scientific terms used herein have their ordinary meanings as understood by one of ordinary skill in the art to which this invention belongs. As used herein, "first," "second," and similar terms do not imply any order, number, or importance, but are only used to distinguish between different components. It is. Similarly, similar words such as "comprising" or "comprising" mean that the elements or articles that occur before the word include the elements or articles that occur after the word and their equivalents; It does not exclude other elements or items. Similar terms such as "connected" or "adjacent to" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Top", "bottom", "left", "right", etc. are used to indicate relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship also changes. Subject to change.

FIG. 1 shows a schematic flowchart of a text processing method according to the present disclosure. As shown in FIG. 1, in step S102, preprocessing is performed on the source text to generate word vectors for the words.

When a text processing method is executed by a computer, the computer cannot directly process text data, so when processing the source text, it is necessary to convert the source text into numerical data. For example, the source text content may be one or more sentences. The preprocessing includes performing word division processing on each sentence so as to divide the sentence into a plurality of words, and converting each of the plurality of words into word vectors of predetermined dimensions. For example, this conversion can be performed using a word embedding method.

In step S104, a plurality of sentence vectors S are determined based on the plurality of initial recommended weight vectors and the plurality of word vectors.

In some embodiments, for each time step, an encoding neural network is used to process the plurality of word vectors generated in step S102 to determine the current encoding that corresponds to each word vector. The hidden state vector can be determined. In some implementations, the encoding neural network may be implemented as a long and short-term memory (lstm) network. It will be appreciated that the encoding neural network may also be implemented as any machine learning model capable of encoding word vectors.

With the word vector generated in step S102 as input, the encoding neural network generates the current encoded hidden state vector h ₁ , whose current time step corresponds to each of the word vectors x ₁ , x ₂ , x ₃ . h ₂ , h _{3 .} . . can be output. The number of encoded hidden state vectors and the number of word vectors may be the same or different. For example, if k word vectors are generated from the source text, the encoding neural network can generate k corresponding encoded hidden state vectors by processing the k word vectors. . k is an integer greater than 1.

Next, based on each initial recommendation weight vector and the current encoded hidden state vector, a sentence vector corresponding to the initial recommendation weight vector is determined.

In some examples, the initial recommendation weight vector W may be represented as a vector [W ₁ , W _{2 .} . . , w _k ]. Here, the number of elements of W is the same as the number of encoded hidden state vectors. Here, each element of the initial recommended weight vector W represents a weight coefficient of each encoded hidden state vector used when determining a sentence vector using the current encoded hidden state vector. Using these weighting factors, the encoded neural network can combine the encoded hidden state vectors corresponding to each input word vector to form a sentence vector containing information about each word vector. Note that the sentence vector referred to here may be an abstract sentence vector. The abstract sentence vector may not have a one-to-one correspondence with sentence information included in the input text. The sentence vector S may include information on some or all of the plurality of word vectors generated in S102.

In some implementations, the sentence vector S may be represented as a weighted average of the current encoded hidden state vectors h ₁ , h ₂ . . . h _k . For example, _the sentence vector S may be expressed as ^W *h, where W=[w ₁ , w _{2 .} . . , w _k ], h=[h ₁ , h ₂ . . Therefore, a predetermined number of sentence vectors S ₁ , S ₂ , . . . , S _n can be obtained using a predetermined number of initial recommended weight vectors W ₁ , W _{2 , .} . . , W _n that have been trained in advance. Here, n and m are integers greater than 1.

In step S106, the plurality of initial recommendation weight vectors are adjusted based on the relationship between each sentence vector and other sentence vectors among the plurality of sentence vectors to obtain a recommendation probability distribution for the plurality of words. Determine.

FIG. 2 shows a schematic diagram for determining the association between each sentence vector and other sentence vectors among the plurality of sentence vectors, according to an embodiment of the present disclosure. Although the principle of the present disclosure is described in FIG. 2 by taking five word vectors as an example, the scope of the present disclosure is not limited thereto. A processing method may also be implemented.

As shown in FIG. 2, x ₁ , x ₂ , x ₃ , x ₄ , x ₅ are word vectors generated from the source text and corresponding to words in the source text. Using the encoding neural network, encoded hidden state vectors h ₁ , h ₂ , h ₃ , h 4 , and h ₅ corresponding to x ₁ , x ₂ , x ₃ , _{x 4 ,} and x ₅ are generated, respectively.

FIG. 2 shows three initial recommended weight vectors W ₁ , W ₂ , W ₃ . Note that the present disclosure is not limited thereto, and the text processing method according to the present disclosure may be implemented using any other number of initial recommended weight vectors. As shown in FIG. 2, sentence vectors S ₁ , S ₂ , and S ₃ are determined using initial recommended weight vectors W ₁ , W ₂ , and W ₃ .

For each of the sentence vectors S ₁ , S ₂ , and S ₃ , the sentence vector is combined with other sentence vectors to generate a combined sentence vector. Here, the combined sentence vector includes information about at least two combined sentence vectors. The principle of the present disclosure will be explained below using an example of determining the relationship between two sentence vectors, but those skilled in the art will be able to determine the relationship between sentence vectors that are a combination of three or more sentence vectors. Even if it's confirmed.

For example, as shown in FIG. 2, the relationship between sentence vectors S ₁ and S ₂ is λ _1,2 , the relationship between sentence vectors S ₁ and S ₃ is λ _1,3 , and the relationship between sentence vectors S ₂ and S ₃ is The relevance λ _2,3 of can be calculated.

In some implementations, the sentence vector can be connected with other sentence vectors to obtain a higher dimensional combined sentence vector. For example, when the dimension of the sentence vector S is d, by connecting the sentence vectors S1 and S2, a combined sentence vector _S1,2 having the dimension 2d is obtained. However, d is an integer greater than 1.

Note that when calculating the relationship between S ₁ and S ₂ for S ₁ , the sentence vectors S ₁ and S ₂ are connected with S ₁ before and S ₂ after. When calculating the relationship between S ₂ and S ₁ for S ₂ , the sentence vectors S ₂ and S ₁ are connected with S ₂ before and S ₁ after. In this case, the combination sentence vector S _1,2 and the combination sentence vector S _2,1 are different.

In other implementations, vector-vector operations are performed on two sentence vectors (eg, addition, subtraction, vector product, etc.) to generate a combined sentence vector. In this case, the combination sentence vector S _1,2 and the combination sentence vector S _2,1 may be the same.

In fact, those skilled in the art can generate a combined sentence vector that combines the information of at least two sentence vectors in any manner.

Then, by processing the combined sentence vector using the relevance matrix, the relationship between the sentence vector and the other sentence vector can be determined. In some embodiments _, the association λ _1,2 between sentence vectors S ₁ and S ₂ may be expressed as λ=S _1,2 *Z. Here, S _1,2 represents a combination sentence vector of sentence vectors S ₁ and S ₂ , and Z represents a trained association matrix. The correlation coefficient λ _1,2 between S ₁ and S ₂ can be calculated using Z. In some embodiments, the relevance matrix Z can project the combined sentence vector S _1,2 into relevance coefficients as real numbers.

By the above method, it is possible to calculate the relationship between any two sentence vectors among the sentence vectors S ₁ , S _{2 .} . . , Sn.

For any one sentence vector described above, a recommendation coefficient for the sentence vector can be determined based on the relationship between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors. . In some implementations, the recommendation coefficient of the sentence vector may be expressed as the sum of the associations of the sentence vector with each of the other sentence vectors of the plurality of sentence vectors.

For example, the recommendation coefficient for sentence vector S ₁ is expressed as Σλ ₁ = λ _1,2 + λ _1,3 +...λ _1,m , and the recommendation coefficient for sentence vector S ₂ is expressed as Σλ ₂ = λ _2,1 + λ _{2 ,3} +...λ _2,m , and in this way, the recommendation coefficient of each sentence vector can be determined.

In other implementations, the recommendation coefficient of a sentence vector may be expressed as a weighted sum of associations between the sentence vector and each of the other sentence vectors of the plurality of sentence vectors. Weighted addition may be performed on the relationship between each sentence vector and other sentence vectors using a predetermined weighting coefficient.

The above recommendation coefficients can be used to adjust the initial recommendation weight vector to generate the corresponding sentence vector in order to obtain the adjusted word probability vector. For example, as shown in FIG. 2, initial recommendation weight vectors _{W 1} , W ₂ , W ₃ are processed using recommendation coefficients Σλ ₁ , Σλ ₂ and Σλ ₃ corresponding to sentence vectors S 1 , S ₂ and S ₃ can do.

As described above, the recommendation coefficient is determined based on the relationship between the sentence vector and other sentence vectors. Since it is necessary to summarize the content of the text in the process of generating a text summary, the higher the relationship with other sentence vectors, the more important the word vector information included in the sentence vector is in the text content. As a result, it is considered that there is a high possibility that the content will be the content of the text summary.

In some embodiments, the weighting factor for each word vector's encoded hidden state vector in the word probability vector is determined by multiplying the recommendation factor for each sentence vector by the word probability vector corresponding to that sentence vector. Can be adjusted. For example, the adjusted i-th word probability vector W _i ′ may be expressed as W _i ′=Σλ _i *W _i .

After adjusting the word probability vector of the sentence vector using the recommendation coefficient of each sentence vector, the recommended probability distribution of the plurality of words is performed using the adjusted plurality of word probability vectors W' obtained by the above method. may be determined.

In some embodiments, the recommended probability distribution P _V may be expressed as P _V =ΣW _i ', which is the sum of adjusted word probability vectors W' obtained by the method described above. In some implementations, the recommended probability distribution P _V may be represented as a weighted sum of adjusted word probability vectors W _{i ′} .

Referring to FIG. 1, in step S108, words to be output may be determined based on the recommendation probability distribution.

The recommended probability distribution output in step S106 can indicate the importance level in the source text of each word in the input source text, where the higher the probability in the recommended probability distribution, the higher the probability for the current time step. , consider that the word has high importance in the source text. Then, in some examples, the word with the highest probability within the recommended probability distribution may be determined as the word to be output at the current time step.

In some embodiments, the output word probability distribution may be determined by adjusting the word probability distribution generated by the current Generative Networks based on the recommended probabilities.

For each time step, a current decoded hidden state vector may be determined using a decoding neural network based on the current encoded hidden state vector. A current word probability distribution can be determined using the current encoded hidden state vector and the current decoded hidden state vector. Based on the current word probability distribution and the recommended probability distribution, determine the output word probability distribution for the current time step, and select the word corresponding to the word vector with the maximum probability from the output word probability distribution at the current time. The word can be selected as the word to be output to the step.

Here, the current word probability distribution may be an attention probability distribution. The attention probability distribution indicates a probability distribution that a word in the input text becomes a word in a text summary.

FIG. 3A shows a schematic diagram of determining an output word probability distribution, according to an embodiment of the present disclosure. As shown in FIG. 3A, by adjusting the attention probability distribution using the recommended probability distribution _PV , an adjusted attention probability distribution can be formed.

ここで、ｔは現在の時間ステップを示し、ａ^ｔは現在の時間ステップについての注意確率分布を示し、ｓｏｆｔｍａｘは正規化指数関数であり、ｅ^ｔは、式（２）により以下のように確定される。

ここで、ｖ^Ｔ、Ｗ_ｈ、Ｗ_Ｓ、ｂ_ａｔｔｎは、ポインター生成ネットワーク（Ｐｏｉｎｔｅｒ－Ｇｅｎｅｒａｔｏｒ Ｎｅｔｗｏｒｋｓ）にける学習パラメータであり、ｈ_ｉは現在の符号化隠れ状態ベクトルであり、ｓ_ｔは現在の復号化隠れ状態ベクトルである。 In one implementation, an attention probability distribution can be determined based on the encoded hidden state vector and the decoded hidden state vector for the current time step. For example, the above attention probability distribution can be determined using equation (1).

Here, t indicates the current time step, a ^t indicates the attention probability distribution for the current time step, softmax is the normalized exponential function, and e ^t is determined by equation (2) as follows. be done.

Here, v ^T , W _h , W _S , _battn are learning parameters in the Pointer-Generator Networks, h _i is the current encoded hidden state vector, and s _t is the current encoded hidden state vector. is the decoded hidden state vector.

In some embodiments, the recommended probability distribution is used to adjust the attention probability distribution, and the adjusted attention probability distribution is determined.

ここで、ｔは現在の時間ステップであり、ａ’^ｔは現在の時間ステップについての調整後の注意確率分布を示し、ｅ^ｔは式（２）により確定されたパラメータである。 For example, the adjusted attention probability distribution a' can be determined using equation (3).

Here, t is the current time step, a' ^t indicates the adjusted attention probability distribution for the current time step, and e ^t is the parameter determined by equation (2).

The adjusted attention probability distribution can be used to determine the probability distribution that a word in the input text becomes a word in a text summary. For example, the word with the highest probability from the input text is selected as the word to be output.

In some embodiments, the current word probability distribution further includes a production probability distribution P _vocab . The generated word probability distribution indicates a probability distribution that a word in the text entity dictionary becomes a word in a text summary.

FIG. 3B shows a schematic diagram of determining an output word probability distribution using a generation probability distribution and an adjusted attention probability distribution, according to an embodiment of the present disclosure.

ここで、Ｖ’、Ｖ、ｂ、ｂ’は、ポインター生成ネットワークにおける学習パラメータであり、ｈ_ｔ ^*は注意確率分布に基づいて確定されたコンテキストベクトルである。例えば、式（４）を利用して確定ｈ_ｔ ^*を確定することができる。

ここで、ａ_ｉ ^ｔは式（１）で確定された注意確率分布ａ^ｔにおけるｉ番目の元素であり、ｈ_ｉは現在のｉ番目の符号化隠れ状態ベクトルである。 In some embodiments, the above generation probability distribution can be determined based on the context vector and the decoded hidden state vector for the current time step. For example, the above generation probability distribution P _vocab can be further determined using equations (4) and (5).

Here, V', V, b, b' are learning parameters in the pointer generation network, and h _t ^* is a context vector determined based on the attention probability distribution. For example, the definite h _t ^* can be determined using equation (4).

Here, a _i ^t is the i-th element in the attention probability distribution a ^t determined by equation (1), and h _i is the current i-th encoded hidden state vector.

Then, by weighting and adding the generation probability distribution and the adjusted attention probability distribution, the output word probability distribution can be determined.

In some embodiments, a generated probability distribution is generated based on the encoded hidden state vector, the decoded hidden state vector, the attention probability distribution for the current time step, and the output of the decoding neural network at the previous time step. and the first weight P _gen of the adjusted attention probability distribution can be determined.

ここで、σは、活性化関数、例えばｓｉｇｍｏｉｄ関数を示し、ｗ_ｈ ^Ｔ、ｗ_ｓ ^Ｔ、ｗ_ｘ ^Ｔ及びｂ_ｐｔｒは訓練パラメータであり、ｈ_ｔ ^*は時間ステップｔに式（４）により確定したパラメータであり、ｓ_ｔは時間ステップｔでの復号化隠れ状態ベクトルであり、ｘ_ｔは時間ステップｔでの復号化ニューラルネットワークの入力、つまり、１つ前の時間ステップｔ－１での復号化ニューラルネットワークの出力である。式（６）により確定された第１の重みＰ_ｇｅｎはスカラーとして実現されてもよい。第１の重みＰ_ｇｅｎを利用して生成確率分布Ｐ_{ｖｏｃａｂ}と調整後の注意確率分布ａ’^ｔを重み平均して出力単語確率分布を取得することができる。 For example, a first weight P _gen for calculating a weighted sum for the generation probability distribution and the adjusted attention probability distribution may be expressed as Equation (6).

_Here ^{, σ} _denotes ^the _{activation function} ^, _e.g. , where s _t is the decoding hidden state vector at time step t, and x _t is the input of the decoding neural network at time step t, that is, the decoding value at the previous time step t-1. This is the output of the neural network. The first weight P _gen determined by equation (6) may be implemented as a scalar. The output word probability distribution can be obtained by weighted averaging of the generation probability distribution P _vocab and the adjusted attention probability distribution a' ^t using the first weight P _gen .

FIG. 3C shows a schematic diagram of determining an output word probability distribution using a generation probability distribution, an attention probability distribution, and a recommendation probability distribution, according to an embodiment of the present disclosure.

As shown in FIG. 3C, the output word probability distribution can be determined by weighting and adding the generation probability distribution, the attention probability distribution, and the recommendation probability distribution. In one implementation, based on the encoded hidden state vector, the decoded hidden state vector, the attention probability distribution, the recommendation probability distribution for the current time step, and the output of the decoding neural network at the previous time step, the A second weight P _gen2 for weighted addition of the generation probability distribution, the attention probability distribution, and the recommendation probability distribution can be determined.

ここで、σは活性化関数、例えばｓｉｇｍｏｉｄ関数を示し、ｗ_ｈ ^Ｔ、ｗ_ｓ ^Ｔ、ｗ_ｘ ^Ｔ、ｗ_Ｖ ^Ｔ及びｂ_ｐｔｒは訓練パラメータであり、ｈ_ｔ ^*は時間ステップｔに式（４）により確定されたパラメータであり、ｓ_ｔは時間ステップｔでの復号化隠れ状態ベクトルであり、ｘ_ｔは時間ステップｔでの復号化ニューラルネットワークの入力であり、つまり、１つ前の時間ステップｔ－１での復号化ニューラルネットワークの出力であり、Ｐ_Ｖは時間ステップｔでの推奨確率分布である。 A second weight P _gen2 for weighted addition of the generation probability distribution, the attention probability distribution, and the recommendation probability distribution can be determined using equation (7).

_Here ^, _{σ denotes} ^an _activation ^{function ,} _{e.g. _} ^_ ), s _t is the decoding hidden state vector at time step t, and x _t is the input of the decoding neural network at time step t, i.e., the previous time step is the output of the decoding neural network at t-1, and P _V is the recommendation probability distribution at time step t.

The weight P _gen2 determined by equation (7) is realized as a three-dimensional vector, where the elements in the three-dimensional vector are the generation probability distribution P _gen , the attention probability distribution a _t and the recommendation probability distribution P, respectively. Indicates the weighting coefficient of _V.

ここで、ｗ_ｔ ^*は時間ステップｔについての正解単語の時間ステップｔでの訓練の出力単語確率分布の確率値であり、Ｔは生成シーケンス全体にわたる合計時間ステップである。テキスト生成ネットワークの全体的な損失は、生成シーケンス全体にわたるすべての時間ステップでの損失値を統計することによって確定されることができる。 The training parameters of the model used in the above text processing are trained using a predetermined training data set. For example, input the training data into the text processing model described above and use the encoding neural network, the decoding neural network, and the initial recommended weight vectors to determine the association between the sentence vectors to generate the word vectors of the source text. By processing, an output word probability distribution trained as described above can be obtained. The training parameters in the above text processing model can be adjusted by calculating the probability loss of correct words in the trained output word probability distribution. Here, the loss function of the text generation network according to the present disclosure can be expressed as follows.

where w _t ^* is the probability value of the training output word probability distribution at time step t of the correct word for time step t, and T is the total time step over the entire generation sequence. The overall loss of a text generation network can be determined by statistics on the loss values at every time step over the entire generation sequence.

Training the parameters of the text processing model described above can be achieved by adjusting the training parameters of the text processing model so that the above loss is minimized.

According to the text processing method according to the present disclosure, for example, when generating text summary content, the words in the input text are The importance level of the content of the text can be determined, and the technical effect is that the content of the generated text can be determined based on the importance level of the word with respect to the text content. In the present disclosure, the principle has been explained using an example of generating a summary, but the content of the present disclosure is not limited thereto. The text processing method according to the present disclosure can also be applied to other application scenes such as text expansion, text rewriting, etc. without departing from the principles of the present disclosure.

FIG. 4 shows a schematic block diagram of a text processing device according to an embodiment of the present disclosure. As shown in FIG. 4, the text processing device 400 includes a preprocessing unit 410, a sentence vector determining unit 420, a recommendation probability determining unit 430, and an output unit 440.

Pre-processing unit 410 is arranged to perform pre-processing on the source text to generate word vectors for said plurality of words. For example, this preprocessing can be achieved by word embedding.

The sentence vector determination unit 420 is arranged to determine sentence vectors S based on the plurality of initial recommendation weight vectors and the plurality of word vectors.

In some embodiments, for each time step, an encoding neural network is used to process the plurality of word vectors generated by preprocessing unit 410 to determine the current encoded hidden state corresponding to each word vector. The vector can be determined.

Taking as input the word vectors generated by the preprocessing unit 410, the encoding neural network calculates the current encoded hidden state vector h ₁ corresponding to each word vector x ₁ , x ₂ , x _{3 .} . . at the current time step, respectively. , h ₂ , h ₃ . . . can be output. The number of encoded hidden state vectors and the number of word vectors may be the same or different. For example, when generating k word vectors based on a source text, the encoding neural network processes these k word vectors to generate corresponding k encoded hidden state vectors. k is an integer greater than 1.

Then, based on each initial recommendation weight vector and the current encoded hidden state vector, a sentence vector corresponding to the initial recommendation weight vector can be determined.

In some examples, the initial recommendation weight vector W may be represented as a vector [w ₁ , w _{2 .} . . , w _k ]. Here, the number of elements in W is the same as the number of encoded hidden state vectors. Here, each element in the initial recommended weight vector W indicates a weighting coefficient for each encoded hidden state vector when determining a sentence vector using the current encoded hidden state vector. By using these weighting coefficients and combining the encoded hidden state vector information of each word vector input from the encoded neural network input, a sentence vector containing each word vector information is formed. In some implementations, the sentence vector S may be represented as a weighted average of the current encoded hidden state vectors h ₁ , h ₂ . . . h _n . Then, a predetermined number of sentence vectors S ₁ , S ₂ , . . . , S _n are obtained using a predetermined number of initial recommended weight vectors W ₁ , W 2 , _{. .} . , W _n that have been trained in advance.

Recommendation probability processing unit 430 adjusts the plurality of initial recommendation weight vectors based on the relevance of each sentence vector to other sentence vectors among the plurality of sentence vectors. arranged to establish a recommended probability distribution.

As shown in FIG. 4, the recommendation probability processing unit 430 includes a relevance determination subunit 431 and an adjustment subunit 432.

The relevance determining subunit 431 is arranged to determine the relevance between sentence vectors. For example, each sentence vector can be combined with other sentence vectors to generate a combined sentence vector.

In some implementations, the sentence vector can be connected with other sentence vectors to obtain a higher dimensional combined sentence vector. For example, if the dimension of the sentence vector S is d, the sentence vectors S ₁ and S ₂ are connected to obtain a 2d-dimensional combined sentence vector S _1,2 . Here, d is an integer greater than 1.

In other implementations, vector-to-vector operations (eg, addition, subtraction, vector product, etc.) of two sentence vectors are performed to generate a combined sentence vector. In this case, the combination sentence vector S _1,2 and the combination sentence vector S _2,1 may be the same.

Next, by processing the combined sentence vector using a relationship matrix, the relationship between the sentence vector and the other sentence vector is determined. In some embodiments, the association λ _1,2 between sentence vectors S ₁ and S ₂ may be expressed as λ=S _1,2 *Z. Here, S _1,2 is a combination sentence vector of sentence vectors S ₁ and S ₂ , and Z indicates a trained association matrix. The correlation coefficient λ _1,2 between S ₁ and S ₂ can be calculated using Z. In some embodiments, the relevance matrix Z can project the combined sentence vector S _1,2 into relevance coefficients as real numbers.

By the above method, it is possible to calculate the relationship between any two sentence vectors among the sentence vectors S ₁ , S _{2 .} . . , S _n .

The adjustment subunit 432 determines a recommendation coefficient for the above-mentioned arbitrary sentence vector based on the relationship between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors. It is arranged like this. In some implementations, the recommendation coefficient of the sentence vector may be expressed as the sum of the associations of the sentence vector with each of the other sentence vectors of the plurality of sentence vectors.

In another embodiment, the recommendation coefficient of a sentence vector may be expressed as a weighted sum of the relationships between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors. The relationship between each sentence vector and other sentence vectors may be weighted and added using a predetermined weighting coefficient.

The recommendation coefficients can be used to adjust the initial recommendation weight vector to generate the corresponding sentence vector to obtain the adjusted word probability vector.

As described above, the recommendation coefficient is determined based on the relationship between the sentence vector and other sentence vectors. Since it is necessary to summarize the content of the text in the process of generating a text summary, the higher the relationship with other sentence vectors, the more important the word vector information included in the sentence vector is in the text content. As a result, it is considered that there is a high possibility that it will be the content of the text summary.

In some embodiments, adjustment subunit 432 determines the encoded hidden state of each word vector contained in the word probability vector by multiplying the word probability vector corresponding to the sentence vector by the recommendation coefficient of each sentence vector. Weighting factors for vectors can be adjusted. For example, the adjusted i-th word probability vector W _i ' may be expressed as W _i '=Σλ _i *W _i .

After adjusting the word probability vector of the sentence vector using the recommendation coefficient of each sentence vector, the adjustment subunit 432 adjusts the word probability vector of the sentence vector based on the adjusted word probability vector W′ obtained as described above. A word recommendation probability distribution can be determined.

In some embodiments, the recommended probability distribution P _V may be expressed as P _V =ΣW _i ', which is the sum of the adjusted word probability vectors W' obtained by the above method, i.e. Make use of it. In some implementations, the recommended probability distribution P _V may be represented as a weighted sum of adjusted word probability vectors W _{i ′} .

The output unit 440 is configured to determine the word to be output based on said recommendation probability distribution.

In some embodiments, the output word probability distribution may be determined by adjusting the word probability distribution generated by the network of current generation equations based on the recommended probabilities.

For each time step, a current decoded hidden state vector may be determined using a decoding neural network based on the current encoded hidden state vector. A current word probability distribution can be determined using the current encoded hidden state vector and the current decoded hidden state vector. Based on the current word probability distribution and the recommended probability distribution, determine the output word probability distribution for the current time step, and select the word corresponding to the word vector with the maximum probability from the output word probability distribution at the current time. The word can be selected as the word to be output to the step.

Here, the current word probability distribution may be an attention probability distribution ^at . The attention probability distribution indicates a probability distribution that a word in the input text becomes a word in a text summary. In one implementation, an attention probability distribution can be determined based on the encoded hidden state vector and the decoded hidden state vector for the current time step.

In some embodiments, by adjusting the attention probability distribution using the recommended probability distribution, the adjusted attention probability distribution a' ^t can be determined. The adjusted attention probability distribution can be used to determine the probability distribution that a word in the input text becomes a word in a text summary. For example, the word with the highest probability from the input text can be selected as the word to be output.

In some embodiments, the current word probability distribution further includes a production probability distribution P _vocab . The generated word probability distribution indicates a probability distribution that a word in the character entity dictionary becomes a word in a text summary. The above generation probability distribution can be determined based on the above context vector and the decoded hidden state vector for the current time step. Then, by weighting and adding the generation probability distribution and the adjusted attention probability distribution, the output word probability distribution can be determined.

In some embodiments, a generated probability distribution is generated based on the encoded hidden state vector, the decoded hidden state vector, the attention probability distribution for the current time step, and the output of the decoding neural network at the previous time step. and the first weight P _gen of the adjusted attention probability distribution can be determined.

In some embodiments, the output word probability distribution may be determined by weighted addition of the generation probability distribution, the attention probability distribution, and the recommendation probability distribution. In one implementation, based on the encoded hidden state vector, the decoded hidden state vector, the attention probability distribution, the recommendation probability distribution for the current time step, and the output of the decoding neural network at the previous time step, the A second weight P _gen2 for weighted addition of the generation probability distribution, the attention probability distribution, and the recommendation probability distribution can be determined. The second weight P _gen2 is realized as a three-dimensional vector, where the elements in the three-dimensional vector have weighting coefficients of the generation probability distribution P _gen , the attention probability distribution a _t , and the recommendation probability distribution P _V , respectively. show.

The training parameters used in the text processing device described above are trained using a predetermined training data set. For example, input the training data into the text processing device described above and use the encoding neural network, the decoding neural network, and the initial recommended weight vectors to determine the association between the sentence vectors to generate word vectors of the source text. By processing, an output word probability distribution trained as described above can be obtained. The training parameters in the above text processing model can be adjusted by calculating the probability loss of correct words in the trained output word probability distribution. Here, the loss function of the text generation network according to the present disclosure can be expressed by equation (8).

where w _t ^* is the probability value of the output word probability distribution after training at time step t of the correct word for time step t, and T is the total time step over the entire generation sequence. The overall loss of a text generation network can be determined by statistics on the loss values at every time step over the entire generation sequence.

Training the parameters of the text processing device described above can be achieved by adjusting the training parameters of the text processing device so that the above losses are minimized.

According to the text processing device according to the present disclosure, for example, when generating text summary content, the words in the input text are The importance level of the content of the text can be determined, and the technical effect is that the content of the generated text can be determined based on the importance level of the word with respect to the text content. In the present disclosure, the principle has been explained using an example of generating a summary, but the content of the present disclosure is not limited thereto. The text processing method according to the present disclosure can also be applied to other application scenes such as text expansion, text rewriting, etc. without departing from the principles of the present disclosure.

Note that a method or apparatus according to an embodiment of the present disclosure may be implemented by the computing device architecture shown in FIG. 5. FIG. 5 shows the architecture of the computing device. As shown in FIG. 5, computing device 500 includes a bus 510, one or at least two CPUs 520, read only memory (ROM) 530, random access memory (RAM) 540, a communication port 550 connected to a network, Input/output components 560, hard disks 570, etc. may also be included. A storage device, such as a ROM 530 or a hard disk 570, in the computing device 500 may contain various data or files according to the present disclosure utilized in processing and/or communicating the method for detecting a target in a video, as well as by the CPU. Program instructions to be executed may be stored. Computing device 500 may also include a user interface 580. Of course, the architecture shown in FIG. 5 is merely exemplary, and when implementing a different device, one or at least two components of the computing device shown in FIG. May be omitted.

Embodiments of the present application may also be implemented as a computer-readable storage medium. A computer readable storage medium according to embodiments of the present application stores computer readable instructions. When the computer readable instructions are executed by the processor, the methods according to the embodiments of the present application described with reference to the figures above can be performed. Computer-readable storage media include, for example, but not limited to, volatile memory and/or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disks, flash memory, and the like.

It should be understood by those skilled in the art that various modifications and improvements can be made to what is disclosed herein. For example, the various devices or components described above may be implemented in hardware, or in software, firmware, or a combination of some or all of these.

Additionally, as indicated in this application and the claims, terms such as "1," "one," and/or "one type," and/or "the" are used to clearly indicate otherwise from the context. Unless indicated otherwise, reference may be made in the plural rather than in the singular. In general, the terms "comprising" and "having" merely imply inclusion of the steps and elements explicitly identified and do not constitute an exclusive enumeration of such steps and elements. The method or apparatus may include other steps or elements.

Additionally, although this specification makes various references to several units of a system according to embodiments of the present disclosure, any number of different units may be used and executed on the client and/or server. The units are merely exemplary, and different units may be used for different aspects of the systems and methods.

Flowcharts are also used in disclosing the present invention to explain operations performed by systems according to embodiments of the present invention. Note that the operations described above or below do not necessarily have to be performed in exact order. Conversely, various steps may be performed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or operations of one or more steps may be removed from these processes.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Terms as defined in common dictionaries should be construed to have meanings consistent with the contextual meanings of the relevant art, and unless expressly defined herein, ideal or extreme should not be interpreted in a formal manner.

Although the present invention has been described above, the present invention is not limited thereto. Although several exemplary embodiments of the present invention have been described, it will be readily apparent to those skilled in the art that many changes can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. should be understood. Accordingly, all such modifications are intended to be included within the scope of the invention as defined in the claims. The foregoing is illustrative of the invention, and the invention is not to be understood as being limited to the particular embodiments disclosed, and modifications to the disclosed embodiments and other embodiments may be disclosed in the accompanying patents. It is to be understood that it is intended to be included within the scope of the claims. The invention is limited by the claims and their equivalents.

Claims (8)

a preprocessing unit arranged to perform preprocessing on the source text to generate a plurality of word vectors for the plurality of words;
a sentence vector determining unit arranged to determine a plurality of sentence vectors based on the plurality of initial recommendation weight vectors and the plurality of word vectors;
adjusting the plurality of initial recommendation weight vectors based on the relevance of each sentence vector to other sentence vectors of the plurality of sentence vectors to determine a recommendation probability distribution for the plurality of words; The recommended probability fixed unit to be placed,
an output unit arranged to determine the word to be output based on the recommended probability distribution ,
The recommendation probability determining unit further includes a relevance determining subunit,
The relationship determining subunit is
For each sentence vector, combine the sentence vector with other sentence vectors to generate a combined sentence vector,
arranged to determine the relationship between the sentence vector and the other sentence vector by processing the combined sentence vector using a relationship matrix,
The recommendation probability determination unit further includes an adjustment subunit,
The adjustment subunit includes:
A recommendation coefficient of the sentence vector is determined based on the relationship between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors, and here, the recommendation coefficient of the sentence vector is and each of the other sentence vectors among the plurality of sentence vectors,
For each of the initial recommended weight vectors, adjust the initial recommended weight vectors using the recommendation coefficients of the sentence vectors corresponding to the initial recommended weight vectors, and obtain adjusted word probability vectors;
arranged to determine a recommended probability distribution of the plurality of words based on the adjusted word probability vector;
Text processing device. The sentence vector determination unit is
processing the plurality of word vectors using an encoding neural network to determine a current encoded hidden state vector corresponding to each word vector;
arranged to determine, based on each initial recommendation weight vector and the current encoded hidden state vector, a sentence vector corresponding to the initial recommendation weight vector;
The text processing device according to claim 1. The output unit is
determining a current decoding hidden state vector using a decoding neural network based on the current encoding hidden state vector;
determining a current word probability distribution using the current encoded hidden state vector and the current decoded hidden state vector;
arranged to determine a word to be output based on the current word probability distribution and the recommended probability distribution;
The text processing device according to claim 2. the current word probability distribution includes a generation probability distribution and an attention probability distribution;
The output unit is
adjusting the attention probability distribution using the recommended probability distribution and determining the adjusted attention probability distribution;
determining an output word probability distribution by weighting and adding the generation probability distribution and the adjusted attention probability distribution;
arranged so that the word with the maximum probability within the output word probability distribution is determined as the word to be output;
The text processing device according to claim 3. the current word probability distribution includes a generation probability distribution and an attention probability distribution;
The output unit is
determining weights used for the generation probability distribution, the attention probability distribution, and the recommendation probability distribution, and determining an output word probability distribution based on the weights;
arranged so that the word with the maximum probability of the output word probability distribution is determined as the word to be output;
The text processing device according to claim 3. a text processing device preprocessing the source text to generate a plurality of word vectors for the plurality of words;
the text processing device determining a plurality of sentence vectors based on the plurality of initial recommendation weight vectors and the plurality of word vectors;
The text processing device adjusts the plurality of initial recommendation weight vectors based on the relationship between each sentence vector and other sentence vectors among the plurality of sentence vectors to determine recommendation probabilities for the plurality of words. determining the distribution;
The text processing device determines a word to be output based on the recommended probability distribution ,
The text processing device adjusts the plurality of initial recommendation weight vectors based on the relationship between each sentence vector and other sentence vectors among the plurality of sentence vectors. By combining the sentence vector with another sentence vector to generate a combined sentence vector, and processing the combined sentence vector using a relationship matrix, the sentence vector and the other sentence vector are combined. determining the association with the vector;
The text processing device may further adjust the plurality of initial recommended weight vectors based on the relationship between each sentence vector and other sentence vectors among the plurality of sentence vectors. determining a recommendation coefficient of the sentence vector based on the relationship between the sentence vector and each of the other sentence vectors of the plurality of sentence vectors, wherein the recommendation coefficient of the sentence vector is , expressed as the sum of the relationships between the sentence vector and each of the other sentence vectors among the plurality of sentence vectors, and for each of the initial recommended weight vectors, the initial recommended weight vector adjusting the initial recommendation weight vector using the recommendation coefficient of the sentence vector corresponding to the sentence vector to obtain an adjusted word probability vector; and a recommendation probability distribution of the plurality of words based on the adjusted word probability vector. determining the
Text processing methods. a processor;
a memory in which computer readable program instructions are stored;
performing the text processing method of claim 6 when the computer readable program instructions are executed by the processor;
Text processing device. A computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable storage medium comprising:
when the computer readable instructions are executed by a computer, causing the computer to perform the text processing method of claim 6 ;
Computer readable storage medium.

Images