JP6620950B2 - Word learning device, word learning method, and word learning program - Google Patents

Description

  The present invention relates to a word learning device, a word learning method, and a word learning program that learn a word vector from a set of sentences.

  Conventionally, as a technique for expressing the meaning of a word as a word vector, a CVOW (Counterinous Bag-of-Words) model and a Skip-gram model that are generally used in software called “word2vec” have been proposed (non-null) Patent Document 2). These models are models that receive a set of sentences and learn and output a d-dimensional vector for each word included in each sentence in the set of sentences.

  Here, an outline of the Skip-gram model superior to the CBOW model will be described.

First, for a sentence x = w ₁ ,..., W _{T in} a set X of input sentences,

... (1)

Consider the probability model. Here, c represents the length of the history considering the neighboring words of the word w _t . When t + j is smaller than 1, w _{t + j} considers a virtual head word such as <s>, and when t + j is larger than T, w _{t + j} considers a virtual end word such as </ s>.

In the Skip-gram model,

... (2)

As a model. Here, v and v ′ are word vectors of words represented by subscripts, and W is a set of all words included in the document X. Learning word vectors v and v 'that maximize the probability model of equation (1) above.

John Duchi, Elad Hazan, and Yoram Singer.Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12 (Jul): 2121-2159, 2011. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.Efficient estimation of word representations in vector space.arXiv preprint arXiv: 1301.3781, 2013.

  In a word vector learning model such as the Skip-gram model, since a probability model is considered using products, sums, etc. in the Euclidean space, each dimension element of a word vector generally acquired is not related. However, if there is a relationship between the dimensions of the word vector as in a discrete signal and the meaning of the word vector dimension can be given, more information can be stored in the word vector. .

  The present invention has been made in view of the circumstances as described above, and provides a word learning device, a word learning method, and a word learning program that can learn a word vector having a relationship between dimensions. The purpose is to do.

  In order to achieve the above object, the word learning device of the present invention receives a set of sentences and a set of words as input, and includes target words included in each sentence of the set of sentences and surroundings around the target words. A word extraction unit with a peripheral context for extracting a word with a peripheral context including a word, a word vector of the target word for all the words with a peripheral context extracted by the word extraction unit with a peripheral context, and the peripheral So as to optimize the objective function represented using the score value calculated by the product with the vector obtained by performing the operation using cyclic convolution or cyclic cross-correlation for each word vector of the word, A word vector learning unit that learns each word vector of the target word and each word vector of the surrounding words, and the word vector learned by the word vector learning unit Comprising an output unit for outputting.

The word vector learning unit sets θ as a set of word vectors including each word vector of the target word and each word vector of the peripheral words, and a pair of a word x with a peripheral context and a correct label y ( When the set of x, y) is X and the score value for the word x with surrounding context is η _x , the objective function is a logistic regression model L (θ) formulated by the following equation: The set of word vectors θ that minimizes the function may be learned as an optimal set of word vectors θ ^.

However, the word vector of the target word w _t

And the word vector of the surrounding words in the left context of the target word w _t

And the word vector of the surrounding words in the right context of the target word w _t

Then, the score value η _x is calculated by the following equation.

  In order to achieve the above object, a word learning method of the present invention is a word learning method in a word learning device having a word extraction unit with a surrounding context, a word vector learning unit, and an output unit, the word extraction with a surrounding context. A section having a set of sentences and a set of words as input, and extracting a word with a peripheral context including a target word included in each sentence of the set of sentences and a peripheral word around the target word; The word vector learning unit cyclically convolves the word vector of the target word with respect to all the words with the surrounding context extracted by the word extracting unit with the surrounding context and the word vectors of the surrounding words. Alternatively, the target unit may be optimized so as to optimize an objective function represented by using a score value calculated by a product with a vector obtained by performing an operation using cyclic cross-correlation. A step of learning of each word vector, and the word vectors of each of the peripheral words, the output unit comprises, a step step of outputting the word vectors learned by the word vector learning unit.

In the step of learning by the word vector learning unit, a set of word vectors including each word vector of the target word and each word vector of the peripheral word is set as θ, and the word x with the peripheral context and the correct label y Logarithmic regression model L (θ) formulated by the following equation, where X is the set of pairs (x, y) and _x and the score value for the word x with surrounding context is _x The set of word vectors θ that minimizes the objective function may be learned as an optimal set of word vectors θ ^.

However, the word vector of the target word w _t

And the word vector of the surrounding words in the left context of the target word w _t

And the word vector of the surrounding words in the right context of the target word w _t

Then, the score value η _x is calculated by the following equation.

  In order to achieve the above object, a program of the present invention is a program for causing a computer to function as each part of the word learning device.

  According to the present invention, it is possible to learn a word vector having a relationship between dimensions.

It is a block diagram which shows the functional structure of the word learning apparatus which concerns on embodiment. It is a flowchart which shows the flow of the word learning process which concerns on embodiment.

  Hereinafter, the present embodiment will be described with reference to the drawings.

  The word learning device according to the present embodiment performs a calculation between discrete vectors (discrete signals) called (cyclic) convolution and (cyclic) cross-correlation, thereby obtaining word vectors having meanings of each dimension element. Learn the model to acquire.

  Here, assuming that two d-dimensional word vectors v and v ′, convolution * is defined by the following equation (3).

... (3)

Also cross-correlation

Is defined by the following equation (4).

(4)

Note that convolution is vv ′ = v ′ * v, which is a commutative operation.

This is a non-commutative operation.

  FIG. 1 is a block diagram showing a functional configuration of a word learning device 10 according to the present embodiment. As illustrated in FIG. 1, the word learning device 10 according to the present embodiment includes an input unit 12, a word extraction unit 14 with surrounding context, a word vector learning unit 16, and an output unit 18.

The input unit 12 inputs a sentence set X ′, a word set W = {w ¹ ,..., W ^N } included in each sentence in the sentence set X ′, and the history length l of the surrounding context. . Here, N is the number of words.

The word extraction unit with surrounding context 14 sets sentences included in the sentence set X ′ as x ′ = {w ₁ ,..., W _T }. All of

Respect, the word with surrounding context _{x = {w t-l,} ..., w t-1, w t, w t + 1, ..., w t + l} taken out. If the peripheral context is to the left than w _1, using a special word <s>, if the peripheral context is to the right than w _T, using a special word </ s>.

  The word extraction unit 14 with the surrounding context performs the above-described processing for extracting the word with the surrounding context for all sentences included in the sentence set X ′, and “1” representing the extracted word x with the surrounding context and the correct label. Is defined as a pair (x, 1), and the set of pairs is defined as X. The set X of pairs is used as learning data for a model proposed below.

  In addition, as disclosed in Non-Patent Document 2, the Skip-gram model or the like generally uses a word x with a peripheral context that is not included in the sentence set X ′ in order to effectively perform vector learning. The negative example is generated by sampling and mixed into the set X of pairs (Negative Smpling).

  Similarly, in this embodiment, a negative example such as a pair (x, −1) in combination with a word x with a surrounding context that did not appear in the sentence set X ′ and “−1” representing a correct answer label is generated by sampling. Then, a set X of pairs is created by mixing with the set X of pairs.

Word vector learning unit 16, a word with a certain neighborhood context _{x = {w t-l,} ..., w t-1, w t, w t + 1, ..., w t + l} with respect to using the convolution and cross-correlation follows Perform the calculation.

First, peripheral contexts composed of peripheral words around the target word are changed to a left context c _l = {w _t−l ,..., W _t−1 } and a right context c _r = {w _{t + 1 ,.} Division and convolution are performed, and the word vector v _cl ′ of the left context and the word vector v _cr ′ of the right context are calculated using the following equations (5) and (6).

... (5)

... (6)

Next, using the following equation (7) using convolution or the following equation (8) using cross-correlation, the score value η _x of the target word w _{t in} the word with surrounding context is calculated. That is, the score value η _x performs a calculation using cyclic convolution or cyclic cross-correlation on the word vector v _wt of the target word w _t and each of the word vectors v _cl ′ and v _cr ′ of the neighboring word w _j. It is calculated by the product with the vector obtained by this.

... (7)

(8)

  In the above equation (7), since convolution is a commutative operation, it is possible to associate the left and right neighboring words without distinction. On the other hand, in the above equation (8), there is a left-right distinction because cross-correlation is used. Since there is no difference in essence regardless of whether the modeling is performed by using the formula (7) or the formula (8), a case will be described in the present embodiment where the formula (7) (cyclic convolution) is used.

In the present embodiment, the logistic function σ is applied to the score value η _x calculated by the above formula (7) or (8), and the probability value p _{θ represented} by the following formula (9) is considered.

... (9)

here,

Is a model parameter, and φ _wt (c _l , c _r ) is a binary function indicating whether or not the word w = {c _l , w _t , c _r } with surrounding context exists on the target data It is.

The word vector learning unit 16 formulates the logistic regression model shown in the following equation (10) so as to minimize the optimum parameter.

That is, the word vector v _wt of each of the target words w _t and the word vectors v _cl ′ and v _cr ′ of each of the surrounding words w _j are learned.

(10)

  This can be optimized by a probabilistic gradient descent method such as Adagrad (see Non-Patent Document 1 above). The optimization procedure is as follows.

(Procedure 0) The parameter θ uses the dimensionality D of each word vector,

From

Each dimension of the word vector is initialized using a random number generated from a uniform distribution with the interval.

(Procedure 1) The iteration number counter i is set to 0, and the following (Procedure 2) is repeated M (M is an arbitrary natural number) times.

(Procedure 2)

On the other hand, the following (procedure 2-1) to (procedure 2-5) are calculated. _{Here, x,} c l, of _{c r} defined as described above.

Here, the gradient vector for the target word and each neighboring word

Are all initialized with a zero vector.

(Procedure 2-1) Word vector of target word w _t in counter i

Is calculated using the following equation (11).

... (11)

(Procedure 2-2)

A word vector of neighboring words w _j in counter i

Is calculated using the following equation (12).

(12)

here,

And

(Procedure 2-3)

Similarly, the word vector of the neighboring word w _j in the counter i

Is calculated using the following equation (13).

... (13)

here,

And

(Procedure 2-4) Word vector of target word w _t in counter (i + 1)

Is updated as the following equation (14).

... (14)

Also,

Also, a word vector of neighboring words w _j in the counter (i + 1)

Is updated as the following equation (15).

... (15)

(Procedure 2-5) The counter is updated so that counter i = i + 1.

  The output unit 18 outputs a parameter (a set of word vectors) θ learned by the word vector learning unit 16.

  Note that the word learning device 10 according to the present embodiment is configured by a computer device including, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores various programs. Moreover, the computer which comprises the word learning apparatus 10 may be provided with memory | storage parts, such as a hard disk drive and a non-volatile memory. In the present embodiment, the CPU reads and executes a program stored in a storage unit such as a ROM or a hard disk, whereby the hardware resources and the program cooperate to realize the above-described function.

  The flow of the word learning process by the word learning device 10 according to the present embodiment will be described using the flowchart shown in FIG. In the present embodiment, the word learning process is started at a timing when predetermined information for starting execution of the word learning process is input to the word learning device 10, but the timing at which the word learning process is started is this. For example, the word learning process may be started at the timing when the input image is input.

  In step S101, the input unit 12 inputs a sentence set X ′ and a word set W included in each sentence of the sentence set X ′.

  In step S103, the word extraction unit 14 with the surrounding context selects any sentence included in the sentence set X ′.

  In step S105, the word extraction unit with surrounding context 14 targets each word included in the selected sentence, extracts 1 word adjacent on the right side as a right context, and extracts 1 word adjacent on the left side. Extract as left context.

  In step S107, the word extraction unit 14 with a surrounding context generates a word x with a surrounding context including the target word and the word extracted in step S105.

  In step S109, the word extraction unit 14 with the surrounding context generates a set X of pairs (x, y) of the word x with the surrounding context and the correct answer label y.

  In step S111, the word extraction unit with surrounding context 14 determines whether or not there is an unprocessed sentence, that is, among sentences included in the sentence set X ′, there is a sentence that has not been processed in steps S103 to S109. Determine whether or not. If it is determined in step S111 that there is an unprocessed sentence (S111, Y), the process returns to step S103, and the processes of steps S103 to S111 are performed on the unprocessed sentence. If it is determined in step S111 that there is no unprocessed sentence (S111, N), the process proceeds to step S113.

  In step S113, the word extraction unit 14 with the surrounding context adds a pair (x, y) of the word x with the surrounding context and the correct label “−1” not included in the sentence set X ′ to the set X.

  In step S115, the word vector learning unit 16 initializes each dimension of the word vectors of each target word and each peripheral word using random numbers generated from the uniform distribution.

  In step S117, the word vector learning unit 16 resets the counter i = 0.

  In step S119, the word vector learning unit 16 adds 1 to the counter i.

  In step S121, the word vector learning unit 16 uses the above expression (11), the above expression (12), and the above expression (13) for each word x with surrounding context, Calculate the gradient of the word vector.

  In step S123, the word vector learning unit 16 uses the gradients of the word vectors of the target word and each peripheral word, the above formula (14), and the above formula (15) for each word x with surrounding context. The word vectors of the target word and each peripheral word are updated, and a set of word vectors θ is calculated.

  In step S125, the word vector learning unit 16 determines whether or not the counter i is greater than or equal to M (M is a predetermined natural number of 2 or more). If the counter i is greater than or equal to M in step S125 (S125, Y), the process proceeds to step S127. If the counter i is smaller than M in step S125 (S125, N), the process returns to step S119, and the processes of steps S119 to S125 are performed.

  In step S127, an optimal word vector set θ is output, and the execution of the word learning process program is terminated. In the present embodiment, the estimated dimension is displayed on display means such as a display, or the dimension information is output by storing data indicating the estimated dimension in the storage means.

  As described above, in this embodiment, a word with a peripheral context that includes a set of sentences and a set of words as input and includes target words included in each sentence of the set of sentences and peripheral words around the target words. To extract. Further, a word vector of the target word for all the extracted words with surrounding context, and a vector obtained by performing an operation using cyclic convolution or cyclic cross-correlation on each word vector of the surrounding words, Each word vector of the target word and each word vector of surrounding words are learned so as to optimize the objective function expressed using the score value calculated by the product of. Then, the learned word vector is output.

  Word vectors learned by the Skip-gram model are applied to many problems of natural language processing such as machine translation, syntax analysis, morphological analysis, and document classification. The set of word vectors θ learned by the word learning device 10 according to the present embodiment can also be applied to the many problems described above. In addition, since the set of word vectors θ obtained by the word learning device 10 according to the present embodiment has a meaning in each dimension, more information can be incorporated into the vector.

In the present embodiment, the case where the above equation (7) (cyclic convolution) is used when calculating the score value η _x of the target word w _t has been described, but the present invention is not limited to this, and the above equation (8) (Cyclic cross-correlation) may be used. In this case, the following formula (16) is used instead of the above formula (11), the following formula (17) is used instead of the above formula (12), and the following formula (18) is used instead of the above formula (13). Should be used.

... (16)

... (17)

... (18)

  Further, in the present embodiment, the operation of the components of the functions shown in FIG. 1 is constructed as a program and installed and executed on a computer used as the word learning device 10, but is not limited thereto, and distributed via a network. You may let them.

  Further, the constructed program may be stored in a portable storage medium such as a hard disk or a CD-ROM, and installed in a computer or distributed.

DESCRIPTION OF SYMBOLS 10 Word learning apparatus 12 Input part 14 Word extraction part 16 with a surrounding context Word vector learning part 18 Output part

Claims (5)

Peripheral context-extracted word extraction that uses a set of sentences and a set of words as input, and extracts a word with a peripheral context including target words included in each sentence of the set of sentences and peripheral words around the target word And
For all the words with surrounding context extracted by the word extracting unit with surrounding context,
It is expressed using a score value calculated by the product of the word vector of the target word and a vector obtained by performing an operation using cyclic convolution or cyclic cross-correlation on each word vector of the surrounding words. A word vector learning unit that learns each word vector of the target word and each word vector of the surrounding words so as to optimize the objective function
An output unit for outputting the word vector learned by the word vector learning unit;
Word learning device including The word vector learning unit sets θ as a set of word vectors including each word vector of the target word and each word vector of the peripheral words, and a pair (x, When the set of y) is X and the score value for the word x with surrounding context is η _x , the logistic regression model L (θ) formulated by the following equation is used as the objective function, and the objective function is The word learning device according to claim 1, wherein the word vector set θ to be minimized is learned as an optimum word vector set θ ^.

However, the word vector of the target word w _t

And the word vector of the surrounding words in the left context of the target word w _t

And the word vector of the surrounding words in the right context of the target word w _t

Then, the score value η _x is calculated by the following equation.

A word learning method in a word learning device having a word extraction unit with a peripheral context, a word vector learning unit, and an output unit,
The word extraction unit with a peripheral context receives a set of sentences and a set of words, and includes a target word included in each sentence of the set of sentences and a peripheral word around the target word Extracting a word;
The word vector learning unit for all the words with surrounding context extracted by the word extracting unit with surrounding context,
It is expressed using a score value calculated by the product of the word vector of the target word and a vector obtained by performing an operation using cyclic convolution or cyclic cross-correlation on each word vector of the surrounding words. Learning each word vector of the target word and each word vector of the surrounding words so as to optimize the objective function
The output unit outputting the word vector learned by the word vector learning unit;
Word learning method including In the step of learning by the word vector learning unit, θ is a set of word vectors including each word vector of the target word and each word vector of the peripheral words, and a word x with a peripheral context and a correct label y When the set of pairs (x, y) is X and the score value for the word x with surrounding context is η _x , a logistic regression model L (θ) formulated by the following equation is used as the objective function: The word learning method according to claim 3, wherein the set of word vectors θ that minimizes the objective function is learned as an optimal set of word vectors θ ^.

However, the word vector of the target word w _t

And the word vector of the surrounding words in the left context of the target word w _t

And the word vector of the surrounding words in the right context of the target word w _t

Then, the score value η _x is calculated by the following equation.

  The program for functioning a computer as each part of the word learning apparatus of Claim 1 or 2.