JP2018185771A - Sentence pair classification apparatus, sentence pair classification learning apparatus, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a class relating to the relationship of a sentence pair in consideration of interpretation of words.SOLUTION: A convolution unit 238 outputs a feature matrix Fobtained by performing convolution processing on a feature matrix Foutput from a word vectorization unit 32, a feature matrix F' output by a sentence pair attention unit 32, and a feature matrix F" output by a word interpretation expanded sentence pair attention section 36, in each layer of a neural net, as an output of a layer i, to each of sentences of the sentence pair. A classification unit 240 classifies each of the sentence pairs included in a set sentence pairs to a class relating to the relationship of the sentence pair on the basis of feature matrices Fand Ffor each of the sentence pair output by a last layer B of the neural net.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a sentence pair classification apparatus, a sentence pair classification learning apparatus, a method, and a program, and in particular, a sentence pair classification apparatus, a sentence pair classification learning apparatus for classifying sentence pairs of two or more sentences into classes, The present invention relates to a method and a program.

  Determining whether a sentence is an answer to a question sentence (answer sentence selection), determining whether two sentences have the same meaning (paraphrase identification), determining whether sentence 1 to sentence 2 can be inferred (implications) If the classification of the relationship class of sentence pairs such as recognition) can be performed accurately by artificial intelligence, it can be applied to a wide range of services such as information retrieval, question answering, and intelligent agent dialogue.

  As a conventional technique for classifying sentence pair classes, techniques such as Non-Patent Document 1 have been proposed so far.

  In a conventional method such as Non-Patent Document 1, a vector similarity matrix is calculated for each word included in each sentence, and the relationship classes of the two sentences are classified based on the similarity matrix. The word vector can be learned from a large-scale document corpus by a method such as word2vec described in Non-Patent Document 2.

Wenpeng Yin, Hinrich Schutze, Bing Xiang, Bowen Zhou: ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs.Transactions of the Association for Computational Linguistics, Volume 4: 259-272 (2016) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.Distributed Representations of Words and Phrases and their Compositionality.In Proceedings of NIPS, 2013.

  In the conventional method, for example, two words, “Where can I buy a baseball ticket” and “Where is a baseball ticket office?” Have the same meaning (in other words) by using vectorized words. In the task of determining whether or not the “ticket” and “admission ticket” word vectors are highly similar, it can be determined correctly.

  However, in the paraphrase judgment, “How long can the cooling off be done?” And “How long can I cancel the contract unconditionally?”, The word “Cooling off” and “No” “Condition” “Contract” “Cancellation” The similarity of each word of “” becomes low, which causes a reduction in the accuracy of the paraphrase determination. The same problem occurs even in the task of answer sentence selection and implication recognition task.

  The present invention has been made to solve the above problems, and it is an object of the present invention to provide a sentence pair classification apparatus, method, and program capable of obtaining a class relating to the relationship between sentence pairs in consideration of interpretation. And

  It is another object of the present invention to provide a sentence pair classification learning device, method, and program capable of learning parameters for obtaining a class related to the relationship between sentence pairs in consideration of interpretation.

  In order to achieve the above object, a sentence pair classification device according to a first invention includes a word dividing unit that divides each sentence of a sentence pair into a series of words, and for each of the sentences of the sentence pair, Each of the divided words is vectorized on the basis of a word vector storage unit that stores a vector of each word, and is obtained by vectorizing a word vectorization unit that outputs a feature matrix for each word of the sentence, and each of the neural networks In a layer, a feature matrix for each of the words for each of the sentences in the sentence pair, or a feature matrix for matching the feature matrix output by the previous layer for each of the sentences in the sentence pair, In each sentence pair attention section to be output for each sentence of the sentence pair and each layer of the neural network, the number of words corresponding to the layer is the same. And a feature matrix for each of the words included in the sentence related to the chunk included in one sentence of the sentence pair, obtained by searching the word sentence storage unit that stores the word sentence for the chunk. In each layer of the neural network, a word extension sentence pair attention unit that outputs a feature matrix related to matching with a feature matrix related to each of the words of the other sentence of the sentence pair, and to each of the sentences of the sentence pair, Feature matrix obtained by performing convolution processing on the feature matrix output by the word vectorization unit, the feature matrix output by the sentence pair attention unit, and the feature matrix output by the word extension sentence pair attention unit Is output as the output of the layer to each of the sentences of the sentence pair; Output by the last layers of Rarunetto, based on the feature matrix for each of the sentence pairs is configured to include a, a classification unit which classifies the class of the relationship of the sentence pairs.

  In the sentence pair classification apparatus according to the first invention, the word vectorization unit, the sentence pair attention unit, the extended sentence pair attention unit, and the convolution unit use a parameter matrix learned in advance. A matrix may be obtained.

  Further, the sentence pair classification learning device according to the second invention, for each of the sentence pairs included in the sentence pair set including each of the sentence pairs to which the correct label indicating the class related to the relationship of the sentence pair is given, A word dividing unit for dividing each sentence of the sentence pair into a series of words; and for each sentence of the sentence pair included in the sentence pair set, each of the divided words is a vector of each word A word vectorization unit that outputs a feature matrix for each word of the sentence obtained by vectorization based on a word vector storage unit that stores the word, and in each layer of the neural network, for each of the sentences of the sentence pair A feature matrix for each of the words or a feature matrix for matching a feature matrix output by the previous layer for each of the sentences in the sentence pair is represented by the sentence pair. A sentence pair attention unit for outputting to each of the sentences; and a word sentence storage unit for storing a word sentence for the chunk in each layer of the neural network, with respect to the chunk in which the word is connected by the number of words corresponding to the layer. Relating to the matching between the feature matrix for each word contained in the word sentence related to the chunk included in one sentence of the sentence pair and the feature matrix for each word in the other sentence of the sentence pair, obtained by searching An extended sentence pair attention unit that outputs a feature matrix for each of the sentences of the sentence pair, and at each layer of the neural network, the feature matrix output by the word vectorization unit and the sentence pair attention unit The feature matrix output and the feature matrix output by the word extension sentence pair attention unit A convolution part that outputs a feature matrix obtained by performing convolution processing as an output of the layer for each of the sentences of the sentence pair, and for each of the sentence pairs included in the sentence pair set, Based on the feature matrix for each of the sentence pairs output by the last layer of the neural network, classify a class related to the relationship of the sentence pairs, and based on the classification result and the correct answer label, the classification result A class classifying unit that calculates a loss related to the sentence pair, and the sentence pair attention unit and the interpretation extended sentence pair attention based on the loss related to the classification result calculated for each of the sentence pairs included in the sentence pair set. And a learning unit that learns a parameter matrix for obtaining a feature matrix in the convolution unit.

  In the sentence pair classification method according to the third invention, the word dividing unit divides each sentence of the sentence pair into a series of words, and the word vectorizing unit applies to each of the sentences of the sentence pair, A step of outputting a feature matrix for each word of the sentence obtained by vectorizing each of the divided words based on a word vector storage unit storing a vector of each word; and a sentence pair attention unit, In each layer of the neural network, a feature matrix for each of the words for each of the sentences in the sentence pair, or a feature for matching the feature matrix output by the previous layer for each of the sentences in the sentence pair A matrix is output for each of the sentences of the sentence pair, and an extended sentence pair attention unit is provided at each layer of the neural network. The word sentence related to the chunk included in one sentence of the sentence pair obtained by searching the word sentence storage unit for storing the word sentence for the chunk for the chunks in which the number of words corresponding to the layer is connected. Outputting a feature matrix relating to matching between a feature matrix relating to each of the words included in the sentence pair and a feature matrix relating to each of the words in the other sentence of the sentence pair, for each of the sentences in the sentence pair, and convolution For each layer of the neural network, the feature matrix output by the word vectorization unit, the feature matrix output by the sentence pair attention unit, and the feature matrix output by the word extension sentence pair attention unit A feature matrix obtained by performing convolution processing is used as an output of the layer, and each of the sentences in the sentence pair And a step of classifying a class related to the relationship between the sentence pairs based on the feature matrix for each of the sentence pairs output by the last layer of the neural network. It is characterized by performing including.

  In the sentence pair classification method according to the third aspect of the invention, the word vectorization unit, the sentence pair attention unit, the interpretation extended sentence pair attention unit, and the convolution unit use a parameter matrix learned in advance. A matrix may be obtained.

  In the sentence pair classification learning method according to the fourth invention, the word division unit includes the sentence pair included in a sentence pair set including each sentence pair to which a correct label indicating a class related to the relationship between sentence pairs is given. Dividing each sentence of the sentence pair into a sequence of words, and a word vectorizing unit for each of the sentences of the sentence pair included in the sentence pair set. A step of outputting a feature matrix for each word of the sentence obtained by vectorizing each word based on a word vector storage unit storing a vector of each word; and a sentence pair attention unit, each of the neural network In the layer, the feature matrix for each of the words for each of the sentences in the sentence pair, or the feature output by the previous layer for each of the sentences in the sentence pair. A step of outputting a feature matrix relating to matrix matching to each of the sentences of the sentence pair, and a word extension sentence pair attention unit, in each layer of the neural network, the number of words corresponding to the layer. For the connected chunks, a feature matrix for each of the words included in the sentence related to the chunk included in one sentence of the sentence pair obtained by searching the word sentence storage unit that stores the word sentence for the chunk, and Outputting a feature matrix for matching with a feature matrix for each of the words of the other sentence of the sentence pair for each of the sentences of the sentence pair, and a convolution unit in each layer of the neural network The feature matrix output by the vectorization unit, the feature matrix output by the sentence pair attention unit, Outputting a feature matrix obtained by performing a convolution process on the feature matrix output by the word extension sentence pair attention unit, as an output of the layer, to each of the sentences of the sentence pair; The class classification unit, for each of the sentence pairs included in the sentence pair set, the sentence pair relationship based on the feature matrix for each of the sentence pairs output by the last layer of the neural network. Classifying a class related to sex, calculating a loss related to the classification result based on the classification result and the correct answer label, and a learning unit calculated for each of the sentence pairs included in the sentence pair set Based on the loss related to the classification result, the sentence pair attention unit, the extended sentence pair attention unit, and the convolution unit And a step of learning a parameter matrix for obtaining a characteristic matrix.

  A program according to the fifth invention is a program for causing a computer to function as each part of the sentence pair classification device according to the first invention or the sentence pair classification learning device according to the second invention.

  According to the sentence pair classification apparatus, method, and program of the present invention, the feature matrix output from the word vectorization unit, the feature matrix output from the sentence pair attention unit, and the extended sentence pair attention unit in each layer of the neural network The feature matrix obtained by performing the convolution process on the feature matrix output by is output to each sentence of the sentence pair as the output of the layer, and the class classification unit outputs the sentence pair included in the sentence pair set. For each, a class related to the relationship between sentence pairs considering the interpretation by classifying into a class related to the relationship between sentence pairs based on the feature matrix for each sentence pair output by the last layer of the neural network. Can be obtained.

  According to the sentence pair classification learning apparatus, method, and program of the present invention, in each layer of the neural network, the feature matrix output from the word vectorization unit, the feature matrix output from the sentence pair attention unit, and the extended sentence pair attention The feature matrix obtained by performing the convolution process on the feature matrix output by the unit is output as a layer output to each sentence of the sentence pair, and the class classification unit includes the sentence pair included in the sentence pair set. For each of the above, classify the class related to the relationship of the sentence pair based on the feature matrix for each of the sentence pairs output by the last layer of the neural network, and relate to the classification result based on the classification result and the correct answer label. The loss is calculated, and the learning unit calculates the loss based on the loss related to the classification result calculated for each sentence pair included in the sentence pair set. By learning the parameter matrix for obtaining the matrix, it is possible to learn the parameters for determining the class of the relationship of the sentence pairs in consideration of the interpretation of a word, the effect is obtained that.

It is a block diagram which shows the structure of the sentence pair classification learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of a word vector memory | storage part. It is an example of a word storage part. It is a flowchart which shows the sentence pair classification learning process routine in the sentence pair classification learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the sentence pair classification learning process routine in the sentence pair classification learning apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the sentence pair classification device which concerns on embodiment of this invention. It is a flowchart which shows the sentence pair classification | category processing routine in the sentence pair classification | category apparatus which concerns on embodiment of this invention. It is a flowchart which shows the sentence pair classification | category processing routine in the sentence pair classification | category apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline according to Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described.

  In the embodiment of the present invention, in view of the above-mentioned problems of the prior art, the interpretation of each word that appears in the input sentence is used. For example, the cooling-off excuses are “A legal system that allows you to withdraw or cancel a contract unconditionally without explanation for a certain period only” and “How long can you cancel a contract unconditionally?” By calculating the word similarity matrix between them and using it for sentence pair class classification, it becomes possible to classify sentence pair classes with high precision without strongly depending on the precision of word vectors.

<Configuration of sentence pair classification learning device according to an embodiment of the present invention>

  Next, the configuration of the sentence pair classification learning device according to the embodiment of the present invention will be described. As shown in FIG. 1, a sentence pair classification learning device 100 according to an embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program and various data for executing a sentence pair classification learning processing routine to be described later. And a computer including Functionally, the sentence pair classification learning device 100 includes an input unit 10 and a calculation unit 20 as shown in FIG.

  The input unit 10 receives a sentence pair set including each sentence pair to which a correct answer label indicating a class related to the relationship between sentence pairs is given.

  The calculation unit 20 includes a word vector storage unit 22, an interpretation sentence storage unit 24, a parameter matrix storage unit 26, a word division unit 30, a word vectorization unit 32, a sentence pair attention unit 34, and an extended sentence pair. An attention unit 36, a convolution unit 38, a class classification unit 40, and a learning unit 42 are included. Details of the processing of each processing unit will be described in detail in the description of the operation.

As shown in FIG. 2, the word vector storage unit 22 stores a set of a word x and a word vector e, and the dimension number of e is E ⁽¹⁾ dimension.

  The word sentence storage unit 24 stores a chunk character string in which at least one word is concatenated and a word sentence corresponding to the chunk character string. For example, as shown in FIG. 3, a sentence corresponding to a chunk character string “cooling off” is stored.

In the parameter matrix storage unit 26, W _a ⁽ⁱ⁾ used in the sentence pair attention unit 34, W _b ⁽ⁱ⁾ used in the interpretation extended sentence pair attention unit 36, W _c ⁽ⁱ⁾ , b _c ⁽ used in the convolution unit 38. ⁱ⁾ Each parameter matrix (i = 1,..., B) of W _d and b _d used in the class classification unit 40 is stored.

  For each sentence pair included in the sentence pair set received by the input unit 10, the word dividing unit 30 divides each sentence of the sentence pair into a series of words.

The word vectorization unit 32 is obtained by vectorizing each of the divided words for each sentence of the sentence pair included in the sentence pair set based on the word vector storage unit 22 that stores the vector of each word. , Output a feature matrix F _j for each word of the sentence.

The sentence pair attention unit 34 uses a parameter matrix W _a ⁽ⁱ⁾ in each layer (i = 1,..., B) of the neural network, and _a feature matrix F regarding each word for each sentence in the sentence pair. _The feature matrix F _j ⁽ⁱ⁾ ′ relating to the matching of the feature matrix F _j ⁽ⁱ⁾ output by the previous layer is obtained for each sentence of _j or sentence pair, and for each sentence of the sentence pair Output.

The word expansion sentence pair attention unit 36 uses the parameter matrix W _b ⁽ⁱ⁾ in each layer of the neural network to store the word sentence corresponding to the chunk for the chunk in which the number of words i corresponding to the layer is connected. obtained by searching the sentence storage section 24, the feature matrix G _k for each of the words included in the interpretation of a word statements about the chunk included in one sentence of sentence pairs, feature matrix for each of the words of the other sentence of the sentence pairs A feature matrix F _j ⁽ⁱ⁾ ″ relating to matching with F _h is obtained and output for each sentence in the sentence pair.

The convolution unit 38 outputs the feature matrix F _j output from the word vectorization unit 32 and the sentence pair attention unit 34 using the parameter matrices W _c ⁽ⁱ⁾ and b _c ⁽ⁱ⁾ in each layer of the neural network. wherein the matrix F _j ⁽ⁱ⁾ ', and interpretation of a word feature matrix extended sentence pairs attention unit 36 has output F _j ^(i)' obtained by performing the convolution processing on the 'feature matrix F _j a ^{(i + 1),} As the output of layer i, it outputs to each sentence of the sentence pair.

The class classification unit 40 outputs, for each sentence pair included in the sentence pair set, feature matrices F ₁ and F ₂ for each sentence pair output by the last layer B of the neural network, a parameter matrix W _d , Based on b _d , class related to sentence pair relationship is classified, and loss L related to the classification result is calculated based on the classification result and the correct answer label.

The learning unit 42 is characterized in the sentence pair attention unit 34, the expanded sentence pair attention unit 36, and the convolution unit 38 based on the loss related to the classification result calculated for each sentence pair included in the sentence pair set. The parameter matrices W _a ⁽ⁱ⁾ , W _b ⁽ⁱ⁾ , W _c ⁽ⁱ⁾ , b _c ⁽ⁱ⁾ , W _d , b _d for obtaining the matrix are learned.

<Operation of Sentence Pair Classification Learning Device According to Embodiment of the Present Invention>

Next, the operation of the sentence pair classification learning device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a sentence pair set including each sentence pair to which a correct answer label indicating a class relating to the relationship between sentence pairs is received, the parameter matrices W _a ⁽ⁱ⁾ , W _b ⁽ⁱ⁾ , W _c ^{(i )} , B _c ⁽ⁱ⁾ , W _d , b _d are initialized and stored in the parameter matrix storage unit 26, and the sentence pair classification learning device 100 executes a sentence pair classification learning processing routine shown in FIGS. .

  In step S100, the number of epochs is initialized to n = 1.

In step S102, learning data (sentence pair set consisting of sentence pairs with correct labels (clusters)) is divided into each of mini-batches including M sentence pairs determined at random. In the present embodiment, the maximum value of the number M of sentence pairs included in the mini-batch is M _max = 50.

  In step S104, a mini-batch is selected.

  In step S106, m, which represents the mini-batch sentence pair number, is set to m = 1.

  In step S108, the word dividing unit 30 sets a variable i representing an index of a block representing a neural network layer to i = 1 for each sentence (sentence j; j = 1 or 2) of the m-th sentence pair. .

  In step S110, the word dividing unit 30 divides the sentence j of the sentence pair into word sequences. For example, a sentence “when the investment trust can be cooled off until” is divided into a series such as “investment trust” “in” “cooling” “off” “ha” “when” “until” “can”. When the number of divided words is greater than T, the word dividing unit 30 outputs only T tokens from the beginning. If the number is less than T, the special word “PAD” is added to the end of the sequence and output. In this embodiment, T = 100.

In step S 112, the word vectorization unit 32 searches the word vector storage unit 22 for each word included in the word sequence (x ₁ , x ₂ ,..., X _r ) of the sentence output from the word division unit 30. Then, each sentence j of the sentence pair is converted into a feature matrix relating to each word of the sentence j in the following equation (1).

... (1)

The size of the matrix of F _j ⁽¹⁾ is E ⁽¹⁾ × T. In this embodiment, E ⁽¹⁾ = 100.

In the case of a word not included in the word vector storage unit 22 and the special word “PAD”, the word vector is an E ⁽¹⁾ -dimensional zero vector.

  Next, in step S200, the number of blocks B (i = 1,..., B) representing the neural network layer is set. In this embodiment, B = 2.

In step S300, the sentence pair attention unit 34 determines the feature matrix F _{j of} the sentence 1 and sentence 2 obtained in step S112, or the sentence 1 and sentence 2 obtained in the convolution processing of the previous layer in step S502 described later. From the feature matrix F _j ⁽ⁱ⁾ , an attention matrix A having An, m in the following equation (2) as an element is created.

... (2)

  Here, the function match is a function that receives vectors x and y and outputs a scalar value in order to output a matching score of a word (or chunk), and is 1 / (1+ | xy−). [:, n] is an operation for extracting the vector in the nth row without considering the column direction, and [:, m] is an operation for extracting the vector in the mth row without considering the column direction. Further, cosine similarity may be used as a match function. The size of the attention matrix A is T × T.

Next, in step S302, the attention matrix A is converted into feature matrices F ₁ ⁽ⁱ⁾ ′, F ₂ ⁽ⁱ⁾ ′ relating to matching between sentences 1 and 2 in the following equation (3).

... (3)

Here, W _a ⁽ⁱ⁾ is a parameter matrix, and is an E ⁽ⁱ⁾ × T matrix. A ^t represents a transposed matrix of the matrix A. In the present embodiment, E ⁽²⁾ = 100 and E ⁽³⁾ = 100.

In step 400, the word extension sentence pair attention unit 36, for each sentence j of the sentence pair, from the value of the layer index i and the word sequence (x ₁ , x ₂ ,.., X _r ) of the sentence j, A series of chunks is created by concatenating words of number i. When i = 1, the word series and the chunk series are the same. When i = 2, the chunk sequence is ((x ₁ , x ₂ ,..., x _r ) PAD). The length of the chunk sequence is always T. At the end of the chunk sequence, i−1 “PAD” words are added. When B is 3 or more, for example, when i = 3, the chunk sequence is ((x ₁ , x ₂ ,..., X _r ) PAD, PAD).

  In step 402, the word extension sentence pair attention unit 36 selects a chunk sequence element k (k = 1, 2,...).

In step S404, the word extension sentence pair attention unit 36, for each sentence j of the sentence pair, a character string (chunk character string; for example, (x ₁ , When x ₂ ) = “cooling”, “off” is “cooling off”), the word storage 24 is searched, and when the word corresponding to the chunk character string is stored, the following expression (4) is used: Obtain a feature matrix G _k for each of the words for the indicated sentence. The acquisition method is the same as that in step S112.

... (4)

In step S406, the word extension sentence pair attention unit 36 obtains the following from the sentence matrix G _{k of the} sentence sentence corresponding to the sentence h and the chunk element k when h = 3-j for each sentence j of the sentence pair: 5) Create an attention matrix A of the formula.

... (5)

In step S408, the word extension sentence pair attention unit 36 converts the attention matrix A created in step S406 into a feature matrix F _j ⁽ⁱ⁾ ″ related to the wording of the following equation (6) for each sentence j of the sentence pair. To do.

... (6)

Here, wall_pooling is a value obtained by averaging the non-zero values in the column direction (may be the maximum value in the column direction) for each row. W _b ⁽ⁱ⁾ is a parameter matrix and is an E ⁽ⁱ⁾ × T matrix.

In step 410, the word extension sentence pair attention unit 36, for each sentence j of the sentence pair, if there is no chunk character string corresponding to the chunk sequence element k in the sentence sentence storage unit 24, the feature matrix F _j ^{( i)} Assign a zero vector to the corresponding part of ''.

... (7)

  In step S412, the expanded sentence pair attention unit 36 determines whether or not the processing has been completed for all elements k. If completed, the process proceeds to step S500. If not completed, the process returns to step S402 to return to the next element. Select k and repeat the process.

In step S500, the convolution unit 38 performs a convolution process on each sentence (sentence j; j = 1 or 2) of the sentence pair. Each feature matrix F _j ⁽ⁱ⁾ , F _j ⁽ⁱ⁾ ′, F _j ⁽ⁱ⁾ ″ is an E ⁽ⁱ⁾ × T matrix. These matrices are converted into 3 × E ⁽ⁱ⁾ × T third-order tensor F, and the filter size is 3 × 2, the padding width in the row direction 0, the column direction 1, the stride width 1, and the number of input channels E ^(i). Then, convolution processing is performed for the number E ^{(i + 1)} of output channels, and H is output by the following equation (8).

... (8)

Here, H is a tensor of 1 × T × E ⁽ⁱ⁾ . σ represents a sigmoid function, and * represents a convolution process. W _c ⁽ⁱ⁾ and b _c ⁽ⁱ⁾ are parameter matrices.

In step S502, the convolution unit 38 converts H obtained by the convolution processing of step S500 into an i + 1 feature matrix F _j ^{(i + 1)} by the following pooling processing for each sentence j of the sentence pair.

... (9)

  Here, w2_pooling is a value obtained by averaging the non-zero values in the column direction at the window size 2 for each row (may be the maximum value in the column direction).

In the final block (i = B) of the layer, wall_pooling is used instead of w2_pooling. The size of F output from the final layer is 1 × E ^{(B + 1)} .

  In step S504, it is determined whether i = B. If i = B, the process proceeds to step S600. If i = B, the process proceeds to step S506, i = i + 1 is set, and the process returns to step S300 to perform the process. repeat.

In step S600, the class classifying unit 40 receives a vector v obtained by converting F ₁ and F ₂ output from the final block of the neural network into a vector and connecting them, and inputs sentence-related class classification (number of classes = class). C).

(10)

Here, the size of W _d is a C × 2E ^{(B + 1)} -dimensional matrix, b _d is a C-dimensional vector, and softmax is a softmax function. y is a C-dimensional vector.

In step 602, the class classification unit 40 calculates a loss related to the output y. When the index of the correct class is t∈ {1,..., C} and the output related to the class t is y _t , the loss L of the correct class t is calculated by the following equation (11).

(11)

  In step S700, the learning unit 42 determines whether m = M. If m = M, the learning unit 42 proceeds to step S704, and if m = M, sets m = m + 1 in step S702 and returns to step S108 for processing. repeat.

In step S704, the learning unit 42 adds up the loss L of the correct class t for each of the sentence pairs calculated in steps 108 to 700 for the mini-batch, and uses the probabilistic gradient descent method W _a of the sentence pair attention unit 34. ^(i), interpretation of a word W _b of the extended sentence pairs attention section 36 ^(i), W _c of the convolution portion ^{_{38 (i), b c (}} i) of the classification unit 40 W _d, b _d each parameter matrix of the (i = Optimize for 1, ..., B). The optimization method is not limited to the stochastic gradient descent method, and other optimization methods may be used.

  In step S706, it is determined whether or not the processing has been completed for all mini-batches. If the processing has been completed, the process proceeds to step S708. If the processing has not been completed, the process returns to step S104 to select the next mini-batch. repeat.

  In step S708, it is determined whether or not n = N (N = 100). If n = N, the process ends. If not n = N, n = n + 1 in step S710 and the process returns to step S102 to perform the process. repeat.

As described above, according to the sentence pair classification learning device according to the embodiment of the present invention, the feature matrix F _j output from the word vectorization unit 32 and the sentence pair attention unit 34 output in each layer of the neural network. the features matrix F _j ⁽ⁱ⁾ ', and interpretation of a word extended statement feature matrix pairs attention unit 36 has output F _j ^(i)' obtained by performing the convolution processing on the 'feature matrix F _j a ^{(i + 1)} , The output of layer i is output for each sentence of the sentence pair, and the class classification unit 40 outputs the sentence pair included in the sentence pair set by the last layer B of the neural network. Based on the feature matrices F ₁ and F ₂ for each sentence pair, classify the class related to the relationship between the sentence pairs, calculate the loss L related to the classification result based on the classification result and the correct answer label, the learning unit 42, Sentence pair Based on the loss related to the classification result calculated for each sentence pair included in the group, the parameter matrix for finding the feature matrix is learned, and the class related to the relationship between the sentence pairs in consideration of the interpretation is obtained. Can learn the parameters.

<Configuration of sentence pair classification apparatus according to an embodiment of the present invention>

  Next, the configuration of the sentence pair classification apparatus according to the embodiment of the present invention will be described. As shown in FIG. 6, the sentence pair classification apparatus 200 according to the embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a sentence pair classification processing routine described later, Can be configured with a computer including Functionally, the sentence pair classification apparatus 200 includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG.

  The input unit 210 receives a sentence pair for which a class is to be obtained as test data.

  The calculation unit 220 includes a word vector storage unit 222, an interpretation sentence storage unit 224, a parameter matrix storage unit 226, a word division unit 230, a word vectorization unit 232, a sentence pair attention unit 234, and an extended sentence pair. An attention unit 236, a convolution unit 238, and a class classification unit 240 are included. Details of the processing of each processing unit will be described in detail in the description of the operation.

  The word vector storage unit 222 stores the same as the word vector storage unit 22 of FIG.

The word sentence storage unit 224 stores the same as the word sentence storage unit 24 of FIG. In the parameter matrix storage unit 226, W _a ⁽ⁱ⁾ used in the sentence pair attention unit 234, W _b ⁽ⁱ⁾ used in the word extension sentence pair attention unit 236, and the convolution unit learned by the sentence pair classification learning device 100. It used ^{_{238 W c (i), b}} c (i) used in the classification unit 240 W _d, each parameter matrix of _{b d (i = 1, ...} , B) is stored.

  The word dividing unit 230 divides each sentence of the sentence pair into a word series for each sentence pair received by the input unit 10.

The word vectorization unit 232, for each sentence of the sentence pair, each word of the sentence obtained by vectorizing each of the divided words based on the word vector storage unit 222 that stores the vector of each word A feature matrix F _j for is output.

The sentence pair attention unit 234 uses a parameter matrix W _a ⁽ⁱ⁾ in each layer (i = 1,..., B) of the neural network, and _a feature matrix F relating to each word for each sentence in the sentence pair. _The feature matrix F _j ⁽ⁱ⁾ ′ relating to the matching of the feature matrix F _j ⁽ⁱ⁾ output by the previous layer is obtained for each sentence of _j or sentence pair, and for each sentence of the sentence pair Output.

The word extension sentence pair attention unit 236 stores, in each layer of the neural network, the word sentence for the chunk with respect to the chunk in which the number of words corresponding to the number i corresponding to the layer i is connected using the parameter matrix W _b ^(i). A characteristic matrix G _k for each word included in the word sentence related to the chunk included in one sentence of the sentence pair obtained by searching the word sentence storage unit 224, and a characteristic regarding each word of the other sentence in the sentence pair A feature matrix F _j ⁽ⁱ⁾ ″ for matching with the matrix F _h is obtained and output for each sentence in the sentence pair.

The convolution unit 238 outputs the feature matrix F _j output from the word vectorization unit 232 and the sentence pair attention unit 234 using the parameter matrices W _c ⁽ⁱ⁾ and b _c ⁽ⁱ⁾ in each layer of the neural network. wherein the matrix F _j ⁽ⁱ⁾ ', and interpretation of a word extended statement feature pairs attention unit 236 has output matrix F _j ^(i)' seek 'obtain a convolution processing performed on the feature matrix F _j ^{(i + 1)} , Output to each of the sentences in the sentence pair as the output of layer i.

The class classification unit 240 outputs, for each sentence pair included in the sentence pair set, feature matrices F ₁ and F ₂ for each sentence pair output by the last layer B of the neural network, a parameter matrix W _d , Based on b _d , the sentence pairs are classified into classes, and the classification results are output to the output unit 250.

<Operation of sentence pair classification apparatus according to embodiment of the present invention>

  Next, the operation of the sentence pair classification apparatus 200 according to the embodiment of the present invention will be described. When a sentence pair is received as test data in the input unit 210, the sentence pair classification apparatus 200 executes a sentence pair classification processing routine shown in FIGS. In addition, when test date of a plurality of sentence pairs, steps S800 to 802 may be performed for each sentence pair sentence.

  In step S800, the test data is divided into mini-batches including one sentence pair.

  Next, the sentence pair divided in step S800 is subjected to the same processing as in steps S108 to S600 shown in FIGS. 4 and 5 to obtain a C-dimensional vector y for each class.

  In step S802, the class corresponding to the t-th element having the largest value among the elements of the C-dimensional vector y obtained in step S600 is output to the output unit 250 as the sentence pair classification result.

As described above, according to the sentence pair classification apparatus according to the embodiment of the present invention, the feature matrix F _j output from the word vectorization unit 32 and the sentence pair attention unit 34 output in each layer of the neural network. wherein the matrix F _j ⁽ⁱ⁾ ', and interpretation of a word feature matrix extended sentence pairs attention unit 36 has output F _j ^(i)' obtained by performing the convolution processing on the 'feature matrix F _j a ^{(i + 1),} As the output of layer i, it is output for each sentence of the sentence pair, and the class classification unit 40 outputs the sentence output by the last layer B of the neural network for each sentence pair included in the sentence pair set. Based on the feature matrices F ₁ and F ₂ for each of the pairs, a class relating to the relationship between sentence pairs can be obtained by classifying the sentence pair into a class relating to the relationship between sentence pairs.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 22 Word vector memory | storage part 24 Word sentence memory | storage part 26 Parameter matrix memory | storage part 30 Word division part 32 Word vectorization part 34 Sentence pair attention part 36 Word extension sentence pair attention part 38 Convolution part 40 Class classification part 42 Learning unit 100 Sentence pair classification learning device 200 Sentence pair classification device 210 Input unit 220 Operation unit 222 Word vector storage unit 224 Word sentence storage unit 226 Parameter matrix storage unit 230 Word division unit 232 Word vectorization unit 234 Sentence pair attention unit 236 Word interpretation Extended sentence pair attention section 240 Class classification section 250 Output section

Claims (7)

A word divider that divides each sentence of a sentence pair into a sequence of words;
A feature matrix for each word of the sentence obtained by vectorizing each of the divided words for each of the sentences of the sentence pair based on a word vector storage unit that stores a vector of each word. An output word vectorization unit;
In each layer of the neural network, a feature matrix for each of the words for each of the sentences in the sentence pair, or a feature for matching the feature matrix output by the previous layer for each of the sentences in the sentence pair A sentence pair attention unit for outputting a matrix to each of the sentences of the sentence pair;
In each layer of the neural network, included in one sentence of the sentence pair, obtained by searching the word sentence storage unit for storing the word sentence for the chunk for the chunk in which the word is connected by the number of words corresponding to the layer. For each of the sentences in the sentence pair, a feature matrix for matching the feature matrix for each of the words included in the sentence related to the chunk and the feature matrix for each of the words of the other sentence of the sentence pair. A verb expansion sentence pair attention part to be output;
In each layer of the neural network, a convolution process is performed on the feature matrix output by the word vectorization unit, the feature matrix output by the sentence pair attention unit, and the feature matrix output by the word extension sentence pair attention unit. A convolution unit that outputs, as an output of the layer, a feature matrix obtained by performing for each of the sentences of the sentence pair;
A class classification unit that classifies a class related to the relationship between the sentence pairs based on the feature matrix for each of the sentence pairs output by the last layer of the neural network;
Sentence pair classifier including   The sentence pair classification device according to claim 1, wherein the word vectorization unit, the sentence pair attention unit, the word extension sentence pair attention unit, and the convolution unit obtain a feature matrix using a parameter matrix learned in advance. A word that divides each sentence of the sentence pair into a series of words for each of the sentence pairs included in the sentence pair set including each of the sentence pairs to which a correct answer label indicating a class relating to the relationship of sentence pairs is given. A dividing section;
For each of the sentences of the sentence pair included in the sentence pair set, each of the divided words is obtained by vectorizing based on a word vector storage unit that stores a vector of each word. A word vectorization unit that outputs a feature matrix for each word;
In each layer of the neural network, a feature matrix for each of the words for each of the sentences in the sentence pair, or a feature for matching the feature matrix output by the previous layer for each of the sentences in the sentence pair A sentence pair attention unit for outputting a matrix to each of the sentences of the sentence pair;
In each layer of the neural network, included in one sentence of the sentence pair, obtained by searching the word sentence storage unit for storing the word sentence for the chunk for the chunk in which the word is connected by the number of words corresponding to the layer. For each of the sentences in the sentence pair, a feature matrix for matching the feature matrix for each of the words included in the sentence related to the chunk and the feature matrix for each of the words of the other sentence of the sentence pair. A verb expansion sentence pair attention part to be output;
In each layer of the neural network, a convolution process is performed on the feature matrix output by the word vectorization unit, the feature matrix output by the sentence pair attention unit, and the feature matrix output by the word extension sentence pair attention unit. A convolution unit that outputs, as an output of the layer, a feature matrix obtained by performing for each of the sentences of the sentence pair;
For each of the sentence pairs included in the sentence pair set, classify the class related to the relationship between the sentence pairs based on the feature matrix for each of the sentence pairs output by the last layer of the neural network. A class classification unit that calculates a loss related to the classification result based on the classification result and the correct answer label;
Based on the loss related to the classification result calculated for each of the sentence pairs included in the sentence pair set, a feature matrix is generated in the sentence pair attention unit, the word extension sentence pair attention unit, and the convolution unit. A learning unit for learning a parameter matrix for obtaining;
Sentence pair classification learning device including A word dividing unit dividing each sentence of a sentence pair into a sequence of words;
Each of the sentences obtained by vectorizing a word vector for each of the sentences of the sentence pair based on a word vector storage unit that stores a vector of each word for each of the sentences of the sentence pair Outputting a feature matrix for the words of
The sentence pair attention unit is output by the previous layer for each of the words of the sentence pair or for each of the sentences of the sentence pair in each layer of the neural network. Outputting a feature matrix for feature matrix matching for each of the sentences of the sentence pair;
The word extension sentence pair attention part is obtained by searching the word sentence storage part for storing the word sentence for the chunk for each chunk of the neural net connected with the number of words corresponding to the layer. A feature matrix for matching a feature matrix for each of the words included in the interpretation sentence for the chunk included in one sentence of the sentence pair and a feature matrix for each of the words of the other sentence of the sentence pair, Outputting for each of said sentences;
In each layer of the neural network, the convolution unit includes the feature matrix output from the word vectorization unit, the feature matrix output from the sentence pair attention unit, and the feature matrix output from the word extension sentence pair attention unit. Outputting a feature matrix obtained by performing convolution processing on each of the sentences of the sentence pair as an output of the layer;
Classifying a class related to the relationship between the sentence pairs based on the feature matrix for each of the sentence pairs output by the last layer of the neural network;
Sentence pair classification method including.   5. The sentence pair classification method according to claim 4, wherein the word vectorization unit, the sentence pair attention unit, the word extension sentence pair attention unit, and the convolution unit obtain a feature matrix using a previously learned parameter matrix. For each of the sentence pairs included in the sentence pair set including each of the sentence pairs to which the correct answer label indicating the class relating to the relationship between the sentence pairs is given, each of the sentences of the sentence pair Dividing into series,
A word vectorization unit vectorizes each of the divided words for each of the sentences of the sentence pair included in the sentence pair set based on a word vector storage unit that stores a vector of each word. Outputting a resulting feature matrix for each word of the sentence;
The sentence pair attention unit is output by the previous layer for each of the words of the sentence pair or for each of the sentences of the sentence pair in each layer of the neural network. Outputting a feature matrix for feature matrix matching for each of the sentences of the sentence pair;
The word extension sentence pair attention part is obtained by searching the word sentence storage part for storing the word sentence for the chunk for each chunk of the neural net connected with the number of words corresponding to the layer. A feature matrix for matching a feature matrix for each of the words included in the interpretation sentence for the chunk included in one sentence of the sentence pair and a feature matrix for each of the words of the other sentence of the sentence pair, Outputting for each of said sentences;
In each layer of the neural network, the convolution unit includes the feature matrix output from the word vectorization unit, the feature matrix output from the sentence pair attention unit, and the feature matrix output from the word extension sentence pair attention unit. Outputting a feature matrix obtained by performing convolution processing on each of the sentences of the sentence pair as an output of the layer;
The class classification unit, for each of the sentence pairs included in the sentence pair set, the sentence pair relationship based on the feature matrix for each of the sentence pairs output by the last layer of the neural network. Classifying a class related to sex, and calculating a loss related to the classification result based on the classification result and the correct answer label;
The learning unit calculates the sentence pair attention unit, the word extension sentence pair attention unit, and the convolution unit based on a loss related to the classification result calculated for each of the sentence pairs included in the sentence pair set. Learning a parameter matrix for obtaining a feature matrix in
A sentence pair classification learning method including The program for functioning a computer as each part of the sentence pair classification | category apparatus of Claim 1 or Claim 2, or the sentence pair classification | category learning apparatus of Claim 3.