JP6586026B2 - Word vector learning device, natural language processing device, method, and program - Google Patents

Description

  The present invention relates to a word vector learning device, a natural language processing device, a method, and a program, and more particularly, to a word vector learning device, a natural language processing device, a method, and a program for learning a word vector related to a word.

  Since each word is a discrete symbol and not based on a physical phenomenon or the like, it is not so simple to quantitatively express the similarity between words. For comparison, for example, voice is generally regarded as time-series data of frequency on a computer. Therefore, the similarity between arbitrary speech sections can be measured to some extent by calculating the distance between the vectorization of various feature quantities (continuous values) that can be calculated from the frequency. Similarly, similarity between images can be easily calculated to some extent by calculating a distance between pixel information vectorized as feature amounts. In this way, the similarity between waveforms, colors, and those based on physical phenomena can be handled relatively naturally on a computer, but discrete symbols such as languages can be used. The similarity between things that do not conform to the physical phenomenon described in (1) cannot simply be handled on a computer. From this background, various methodologies have been devised so far to calculate the similarity between discrete symbols such as words. One of them is a distributed semantic expression method. This is a method of allocating one vector to each word and trying to express the semantic similarity between words with the distance between the vectors, as in the case of speech and images. It can be said that this method has a very high affinity for computers because it expresses the semantic proximity between words by calculating the distance in the vector space.

  FIG. 8 shows an outline of the similarity between words by the distributed semantic expression.

Here, the i-th word is represented as w _i . Also, the vector assigned to the i-th word w _i is denoted by r _i . Hereinafter, a vector assigned to a word is specifically referred to as a “word vector”. That is, the word vector of word w _i is r _i . At this time, as a calculation on a computer, the similarity between two words w _i and w _j is generally defined by an inner product between word vectors of w _i and w _j or a cosine distance.

In this case, the larger the value, the more similar the words w _i and w _j are. This has the advantage that words that are semantically similar can be handled during processing in various applications of language processing such as translation, dialogue, document summarization, and document proofreading. As a result, it is shown that better results can be obtained than a processing method that does not use semantic proximity between words.

  Here, many methods have been proposed for acquiring word vectors for each word. As a basic methodology, first, for each word in a sentence, the context information of that word is defined. There are no specific rules for the context information and various information can be used, but most simply, the words appearing around each word are mostly handled as context information. Changing the context definition does not significantly affect the word vector estimation algorithm itself. Therefore, in the following discussion, the word context information is a word that appears in the vicinity.

  In recent years, methods that can be processed at such a high speed as to be able to deal with large-scale data obtained from web have become mainstream (see Non-Patent Document 1 and Non-Patent Document 2). The reason why the method that can handle large-scale data is the main reason is that the more data there is, the more accurately the language event can be captured, so it can be theoretically expected that the estimation accuracy of similarity will be improved. is there. Here, as a representative of the conventional method, a method for acquiring a word vector according to Non-Patent Document 2 will be described.

In this method, in order to express a case where a word appears as the context of a certain word, another vector different from the word vector is assigned to each word. For convenience, this is called a “context vector” in contrast to the word vector. That is, the i-th word w _i has two vectors, a word vector r _i and a context vector c _i . The vocabulary number is V. At this time, let (r _i ) _{i = 1} ^{V be} a vector list in which all word vectors are arranged in order from i = 1 to V. Similarly, (c _i ) _{i = 1} ^V is a list of vectors in which all context vectors are arranged in order from i = 1 to V. In order to simplify the notation, R = (r _i ) _{i = 1} ^V and C = (c _i ) _{i = 1} ^V. Further, the number of times that the j-th word becomes a context with respect to the i-th word is X _{i, j} . At this time, the acquisition of the distributed semantic expression according to Non-Patent Document 2 can be formulated by the problem of minimizing the following objective function.

However, {circumflex over (R)} (^ r _i ) _{i = 1} ^V and {circumflex over (C)} (^ c _i ) _{i = 1} ^V represent estimation results of word vectors and context vectors. Φ is a function for calculating a weighting factor from X _{i, j} .

Finally obtained ^ r _i is a word vector of the i-th word. This is a word vector used in the similarity calculation of equation (2). It is also used in natural language processing application applications such as translation and document proofreading.

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, Efficient Estimation of Word Representations in Vector Space.Proceedings of Workshop at ICLR, 2013. Jeffrey Pennington, Richard Socher, and Christopher Manning, Glove: Global Vectors for Word Representation.Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.

  The problem of the method used in the current mainstream as in Non-Patent Document 2 described above is that when the “number of words” to be handled and the “dimension number” of the output vector space are increased, the result of the word vector finally obtained The amount of memory required to hold the data is large.

  The file size is the same as the memory occupancy during execution. Memory occupancy can be a significant problem in computing environments with limited resources, such as portable terminals. In addition, even when executing on a general computer, it is desirable that the memory occupancy is as small as possible from the viewpoint of not affecting other programs as much as possible when simultaneously executing multiple on a recent multi-core computer. .

  The present invention has been made to solve the above-described problems, and provides a word vector learning apparatus, method, and program capable of learning a word vector that can reduce a necessary memory capacity. With the goal.

  It is another object of the present invention to provide a natural language processing apparatus and program for performing natural language processing based on the semantic similarity of words using learned word vectors.

  In order to achieve the above object, the word vector learning device according to the first invention is based on document data, and for each word, a D-dimensional word vector related to the word and the word as a context of another word A word vector learning device for learning a D-dimensional context vector representing appearance, wherein each of the word vector and the context vector is divided into B blocks having a dimensionality F, Based on the set of real vectors and the number of times one word has appeared as the context of the other word for the word pair in the document data, each block of the word vector and context vector is the S types of real vectors. The objective function representing the likelihood of each block of the word vector and the context vector is optimized under any of the constraints To, and is configured to include a learning unit, which estimates the real value vector of each block of word vectors and the context vector for each of the words.

Further, in the word vector learning device according to the first invention, the learning unit represents each block r _i of the word vector for each word i so as to optimize the objective function represented by the following equation: _{, b} and the word j, an update unit that updates the real vector of each block c _{j, b} of the context vector, and an iterative determination unit that repeats the update by the update unit until a predetermined iteration end condition is satisfied. May be included.
Here, D represents a set of pairs of the word appearing in the document data and a word appearing as the context of the word, α _{i, b} and β _{i, b} are Lagrange undetermined multipliers, and u _{i , b} , v _{j, b} are auxiliary parameters that are real vectors of dimension F, ρ is a constant, ζ represents a set of S types of real vectors of dimension F, and X _{i, j} Represents the number of times word j appears as context with respect to word i.

  A natural language processing device according to a second invention uses the word vector of each word learned by the word vector learning device according to claim 1 or 2 as an input document, to the word vector. A natural language processing unit that performs natural language processing based on the semantic similarity between the words based on.

  A word vector learning method according to a third aspect of the present invention is based on document data, and for each word, a D-dimensional word vector related to the word, and a D-dimensional representing that the word appears as the context of another word. A word vector learning method in a word vector learning device for learning a context vector, wherein the learning unit divides each of the word vector and the context vector into B blocks having a dimension number F, and S types having a dimension number F. Each block of the word vector and the context vector is the S types of real numbers, based on the set of real vectors and the number of times one word appears as the context of the other word in the document data. An objective function representing the likelihood of each block of each of the word vector and the context vector under the constraint of any of the vectors As optimization, and executes comprising the step of estimating a real-valued vector of each block of word vectors and the context vector for each of the words.

Also, in the word vector learning device according to the first invention, the step of estimating by the learning unit is performed for each word i so that the update unit optimizes the objective function represented by the following equation: Updating the real vector of each block c _{j, b} of the context vector for each block r _{i, b} and word j of the word vector until the iteration determination unit satisfies a predetermined iteration termination condition And repeating the updating by the updating unit.
Here, D represents a set of pairs of the word appearing in the document data and a word appearing as the context of the word, α _{i, b} and β _{i, b} are Lagrange undetermined multipliers, and u _{i , b} , v _{j, b} are auxiliary parameters that are real vectors of dimension F, ρ is a constant, ζ represents a set of S types of real vectors of dimension F, and X _{i, j} Represents the number of times word j appears as context with respect to word i.

  In the natural language processing device according to the fourth invention, the natural language processing unit uses the word vector of each word learned by the word vector learning device according to claim 4 or 5 for the inputted input document. And performing a natural language process based on a semantic similarity between words based on the word vector.

  A program according to the fifth invention is a program for causing a computer to function as each part constituting the word vector learning device according to the first invention or the natural language processing device according to the second invention.

  According to the word vector learning device, method, and program of the present invention, each of the word vector and the context vector is divided into B blocks having the dimension number F, and a set of S kinds of real vectors having the dimension number F; Based on the number of times one word has appeared as the context of the other word, the likelihood of each block of each of the word vector and the context vector is determined under the constraint that each block is one of S types of real vectors. Learning word vectors that can reduce the required memory capacity by estimating the real-valued vector of each block of word vectors and context vectors for each word so as to optimize the objective function to represent The effect of being able to be obtained is obtained.

  In addition, according to the natural language processing apparatus and the program of the present invention, it is possible to perform natural language processing based on the semantic similarity of words using a learned word vector.

It is a figure which shows the example of the word list and context co-occurrence information. It is a figure which shows an example of a word vector. It is a block diagram which shows the structure of the word vector learning apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the word vector learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the word vector learning process routine in the word vector learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the vector update process routine in the word vector learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the natural language processing routine in the natural language apparatus which concerns on embodiment of this invention. It is a figure which shows the outline | summary of the similarity between words by a distributed semantic expression.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline according to Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described.

  In essence, the present invention compresses the size of a word vector based on the general idea that the amount of required resources (file size, memory occupancy, etc.) is better in any computing environment. Tackle the problem.

  In the embodiment of the present invention, the similarity between words in the above equation (3) is calculated by the cosine similarity, so that the amount of information held by the word vector is reduced without changing the value of the cosine similarity as much as possible. This is based on the idea of obtaining a word vector with a reduced capacity as a whole.

  First, a word vector having a dimension number D is divided into several blocks. For example, if the “number of elements in a block” of the word vector is F and the “number of blocks” is B, the relationship of D = BF holds. For example, if D = 256 and F = 32, B = 8. Next, the number of types of value sets that each block can take is determined in advance. For example, S is the “number of value set types”.

  At this time, one block in the value set S is selected as each block in the word vector. That is, one word vector is expressed by a combination of value sets in which B blocks are individually selected one by one. At this time, the number of types that the word vector can take is SB, and one of them corresponds to the problem of being assigned to the word vector. For example, when 16 types of value sets are prepared in the previous example, one word vector can take 168 = 4,294,967,296 values. In general, the number of words is in the millions, and no more than tens of millions. Therefore, if there are about 4.3 billion types, it is considered that the distributed expression of words can be expressed sufficiently. Therefore, a method of acquiring a reduced word distribution semantic expression from the data by solving the optimization problem as shown in the above formulas (2) and (3) is used.

  In the embodiment of the present invention, a word vector learning apparatus to which the problem of acquiring a distributed semantic expression of a word using an electronic text described in a natural language existing on a web is applied as an example will be described.

  First, it is assumed that a sentence generated by a general user posted to an SNS site on a certain day is acquired and a word vector is generated. At this time, the vocabulary number V to be used is the number of types of all words appearing in the acquired data. Alternatively, the vocabulary may be determined for words that appear more than a certain frequency.

  FIG. 1 shows an example of word list and context co-occurrence information. For the sake of simplicity in the example shown here, it is assumed that word breaks and the like can be easily obtained using a commonly used tool or the like. In the case of Japanese, there are tools that can be used for free, and in the case of English, a blank separator can be used as a word separator. Here, I will give an example in Japanese.

  The dimension number D of the word vector is set to 256, for example. Further, it is assumed that the vocabulary number V is 1 million words. An example of a word vector is shown in FIG.

  When the conventional method is used with this setting, 1 million 256-dimensional word vectors are constructed.

  This word vector can be obtained by solving the optimization problem of the above equation (2).

Next, consider a case where the present invention is applied to an embodiment of the present invention. First, the b-th partial vector when the i-th word vector r _i is divided for each block is described as r _{i, b} . At this time, assuming that the number of blocks is B, the vectors from b = 1 to b = B of r _{i, b} match r.

  In the embodiment of the present invention, a word vector is obtained by solving the optimization problem of the following equation (4) instead of the above equation (3).

  Here, ζ represents a set of S kinds of real vectors having a dimension number F.

  A major difference from the conventional method is that each word vector is divided into blocks, and a block-type vector is optimized. Another significant feature is that learning is performed while restricting the value set of each block to S types.

  Next, in order to simplify the processing, the equation (4) is modified as follows.

  Equations (4) and (5) are equivalent. It is simply an equation modification for performing processing by decomposing into vectors r and u and c and v as processing during optimization. When the equality constraint of r = u, c = v is relaxed using the extended Lagrangian relaxation method, the following equation (6) is obtained (Reference 1: Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed). Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning, 2011.).

In the embodiment of the present invention, equation (6) is used as an objective function. Note that A = (α _i ) _{i = 1} ^V and B = (β _j ) _{j = 1} ^V.

  The processing flow of the embodiment of the present invention is shown below.

(input)
The learning data D and the tuning parameters D, B, F, and T determined manually are input.

(Initialization)
The optimization variables R and C are initialized with random numbers, and the variable t = 0 for managing repetition is initialized.

(Process 1: Update)
The optimization parameters R and C are updated without restriction using a gradient method or the like.

(Process 2: Parameter adjustment)
The optimization parameter R; C is adjusted so that the number of types of value sets is S in all word vector blocks.

(Process 3)
Update the Lagrange multiplier.

(Process 4: End determination)
If the optimization has converged or if the predetermined number of times T set in advance has been reached or if the determination end determination is satisfied, the process proceeds to the next process. If not, t = t + 1 and return to processing 1.

(Process 5: Reduction)
The data holding method is optimized for the obtained ^ w.

<Configuration of Word Vector Learning Device According to Embodiment of the Present Invention>

Next, the configuration of the word vector learning device according to the embodiment of the present invention will be described. As shown in FIG. 3, the word vector learning device 100 according to the embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a word vector learning processing routine described later, Can be configured with a computer including Functionally, the word vector learning device 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 is learning data.

(Hereinafter referred to as document data). The document data is a set of pairs (w _i , w _j ) of words and words that appear as the word context. w _i and w _j are a target word w _i and a context word w _j . The input unit 10 receives tuning parameters D, B, F, and T determined manually. These values do not change during the subsequent processing.

  The calculation unit 20 includes a learning unit 30 and a vector storage unit 40.

The learning unit 30 divides each of the word vector and the context vector in the document data into B blocks having a dimension number F, and sets one of the S types of real vectors having the dimension number F and one word pair in the document data. Based on the number of times X _{i, j} that the word has appeared as the context of the other word, each block of the word vector and the context vector is subject to the restriction that the word vector and the context are any one of the S real vectors. Estimate a word vector for each word and a real value vector for each block of the context vector to optimize the objective function representing the likelihood of each block of each vector.

  The learning unit 30 includes an update unit 32 and an iterative determination unit 34.

  The updating unit 32 represents a set of pairs of words that appear in the document data and words that appear as the context of the word, so as to optimize the objective function, expressed by the above equation (6).

And the auxiliary parameters u _{i, b} , v _{j, b of} each block, which is a real vector of dimension F _, and Lagrange undetermined multipliers α _{i, b} , β _{i, b} for each block, Update the real vector of each block c _{j, b} of the context vector for each block r _{i, b} and word j for each word vector.

  The iterative determination unit 34 causes the updating unit 32 to repeat the update until a predetermined iteration end condition is satisfied. Here, the iteration end condition is that the updating of the updating unit 32 is repeated the number of repetitions T.

  Specific processing of the updating unit 32 will be described below.

  The update unit 32 first initializes the optimization variables R, C, U, and V with random numbers. Also, a variable t for managing the number of repetitions is initialized with t = 0. Similarly, α and β are initialized as α = 0 and β = 0.

  At this time, only the word vector set R and the context vector set C are considered as optimization variables, and the objective function when U and V of the auxiliary parameter set are fixed is expressed by the following equation (7).

The gradient between the word vector block r _{i, b} and the context vector block c _{j, b} in the equation (7) can be written as the following equations (8) and (9).

Therefore, the gradient vector is used to update the word vector block r _{i, b} and the context vector block c _{j, b} according to the above equations (8) and (9).

  Next, the auxiliary parameters u and v are updated. Regarding equation (6), only the auxiliary parameters u and v are considered as optimization variables, and the objective function when u and v are fixed is as shown in equation (10) below.

  However,

And

This problem is equivalent to the k-means clustering problem based on Euclidean distance. Therefore, for example, the auxiliary parameters u _{i, b} , v _{j, b} are updated by solving equation (10) using a general k-means clustering method.

Next, R, C, U, and V are fixed, and Lagrange undetermined multipliers α _{i, b} and β _{i, b} are updated as optimization variables.

From this relationship, the following update formula is obtained. Using the gradient method, Lagrange undetermined multipliers α _{i, b} and β _{i, b} are updated in accordance with equations (13) and (14).
Here, η represents a learning rate.

  Next, the end determination by the repetition determination unit 34 is performed. Basically, the end determination is made based on whether or not a preset number of repetitions T has been reached. When t = T, it is determined that the process is finished, and the process proceeds to save the word vector. When t <T, t = t + 1 and the process returns to the first process of the updating unit 32.

  Finally, the learning unit 30 stores the learned word vector and the real value vector of the context vector updated by the update unit 32 in the vector storage unit 40 in an appropriate format. Since each block of the word vector takes one of S types of value sets, it can be described by [logS] bit. Therefore, one word vector is B [logS] bit. As a result, the word vector of all vocabularies is VB [logS] bit. Similarly, context vectors for all vocabularies can also be stored with VB [logS] bit. Also, since one value set is composed of F double-precision floating point numbers (64 bits), the entire S value set can be expressed by 64 SF bits.

  Finally, the word vectors and context vectors of all vocabularies can be stored with 2VB [logS] + 64SF bits. Therefore, when V = 1,0000,00, B = 8, S = 16, and F = 32, 2 × 1,0000,00 × 8 × 4 + 64 × 16 × 32 = 64,032,768 bits ( In about 8 MB), word vectors and context vectors of all vocabularies can be stored.

  <Configuration of Natural Language Processing Device According to Embodiment of the Present Invention>

  Next, the configuration of the natural language processing apparatus according to the embodiment of the present invention will be described. As shown in FIG. 4, the natural language processing apparatus 200 according to the embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program and various data for executing a natural language processing routine described later. It can be configured with a computer including. Functionally, the natural language processing apparatus 200 includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG. In the present embodiment, the natural language processing device 200 will be described by taking as an example a case where an unknown word is replaced with a word having a high similarity based on the word vector learned by the word vector learning device 100. However, the present invention is not limited to this, and summarization, document proofreading, and the like may be performed using the replaced word.

  The input unit 210 receives text to be translated.

  The calculation unit 220 includes a natural language processing unit 230 and a vector storage unit 240.

  The vector storage unit 240 stores the same as the vector storage unit 40.

  The natural language processing unit 230 includes a replacement unit 232 and a translation unit 234.

  The replacement unit 232 extracts an unknown word that is not in an existing dictionary (not shown) that stores the word from the words of the text received by the input unit 210, and a context vector for the word stored in the vector storage unit 40 Based on the above, the word in the dictionary having the highest similarity to the unknown word is estimated. And the text which replaced the unknown word with the word in the estimated dictionary is produced | generated.

  The translation unit 234 translates the text in which the word is replaced by the replacement unit 232 using an existing method, outputs the translated text to the output unit 250, and ends the processing.

  In the natural language processing apparatus 200, when performing other natural language processing, a word similar to a word appearing in a specific document is extracted from the dictionary and included in the processing target, thereby increasing information and accuracy. It is possible to improve. At this time, for each word that appears, the above formula (1) is calculated and a word having a high similarity is included in the process.

<Operation of the word vector learning device according to the embodiment of the present invention>

  Next, the operation of the word vector learning device 100 according to the embodiment of the present invention will be described. When the document data and the tuning parameters D, B, F, and T are received by the input unit 10, the word vector learning device 100 executes a word vector learning processing routine shown in FIG.

  First, in step S100, the optimization variables R, C, U, and V are initialized with random numbers. Also, a variable t for managing the number of repetitions is initialized with t = 0. Similarly, α and β are initialized as α = 0 and β = 0.

  Next, in step S102, a set of pairs of a word that appears in the document data and a word that appears as the context of the word so as to optimize the objective function represented by the above equation (6).

And the auxiliary parameters u _{i, b} , v _{j, b of} each block, which is a real vector of dimension F _, and Lagrange undetermined multipliers α _{i, b} , β _{i, b} for each block, Update the real vector of each block c _{j, b} of the context vector for each block r _{i, b} and word j for each word vector.

  Specifically, step S102 is performed by the following steps S120 to S124 shown in FIG.

In step S120, the word vector block r _{i, b} and the context vector block c _{j, b} are updated according to the above equations (8) and (9) using the gradient method.

  In step S122, the auxiliary parameters u and v are updated according to the equation (10).

In step S124, R, C, U, and V are fixed, and Lagrange undetermined multipliers α _{i, b} and β _{i, b} are updated as optimization variables according to the equations (13) and (14).

  In step S104, it is determined whether the iteration end condition is satisfied. When t = T, it is determined that the process is finished, and the process proceeds to the storage of the word vector in step S110. When t <T, t = t + 1 is set in step S106, and the process returns to step S102.

  In step S108, the learned word vector and the real value vector of the context vector updated in step S102 are stored in an appropriate format in the vector storage unit 40, and the process ends.

  As described above, according to the word vector learning device according to the embodiment of the present invention, each of the word vector and the context vector is divided into B blocks having the dimensionality F, and S types having the dimensionality F are selected. Based on the set of real vectors and the number of times one word has appeared as the context of the other word, each block can be either of the S types of real vectors, with the constraint that each of the word vectors and context vectors. A word that can reduce the required memory capacity by estimating the word vector for each word and the real value vector of each block of the context vector so as to optimize the objective function representing the likelihood of each block Can learn vectors.

<Operation of Natural Language Processing Device According to Embodiment of the Present Invention>

  Next, the operation of the natural language processing apparatus 200 according to the embodiment of the present invention will be described. When the input unit 210 accepts a text to be translated, the natural language processing apparatus 200 executes a natural language processing routine shown in FIG.

  In step S200, an unknown word is extracted from the text to be translated received by the input unit 210.

  In step S202, based on the word vector of each word stored in the vector storage unit 240, the word in the dictionary having the highest similarity to the unknown word extracted in step S200 is estimated, and the translation target Is generated by replacing an unknown word with a word in the estimated dictionary.

  In step S204, translation is performed based on the text generated in step S202, and the text is output to the output unit 250 and the process is terminated.

  As described above, according to the natural language processing apparatus of the embodiment of the present invention, it is possible to perform translation processing based on the semantic similarity of words using learned word vectors.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

10, 210 Input unit 20, 220 Arithmetic unit 30 Learning unit 32 Update unit 34 Repetition determination unit 40 Vector storage unit 50, 250 Output unit 100 Word vector learning device 200 Natural language processing device 230 Natural language processing unit 232 Replacement unit 234 Translation unit 240 Vector storage

Claims (8)

A word vector learning device that learns, for each word, a D-dimensional word vector related to the word and a D-dimensional context vector indicating that the word appears as a context of another word based on document data. ,
Each of the word vector and the context vector is divided into B blocks of dimension F, and a set of S types of real vectors of dimension F and one word for the word pair in the document data Based on the number of occurrences of the context, each block of each of the word vector and context vector is subject to a constraint that each block of the word vector and context vector is one of the S kinds of real vectors. A word vector learning apparatus including a learning unit that estimates a word vector and a real value vector of each block of a context vector for each of the words so as to optimize an objective function representing likelihood. The learning unit
For each word vector block r _{i, b} for each word _{i and} for each block c _{j, b} for the context vector for each word j, so as to optimize the objective function represented by An updater for updating a real vector;
The word vector learning device according to claim 1, further comprising: an iterative determination unit that repeats updating by the updating unit until a predetermined iterative termination condition is satisfied.

Here, D represents a set of pairs of the word appearing in the document data and a word appearing as the context of the word, α _{i, b} and β _{i, b} are Lagrange undetermined multipliers, and u _{i , b} , v _{j, b} are auxiliary parameters that are real vectors of dimension F, ρ is a constant, ζ represents a set of S types of real vectors of dimension F, and X _{i, j} Represents the number of times word j appears as context with respect to word i. A natural based on semantic similarity between words based on the word vector using the word vector of each word learned by the word vector learning device according to claim 1 or 2 with respect to the input document. A natural language processing apparatus including a natural language processing unit that performs language processing. A word vector in a word vector learning device that learns, for each word, a D-dimensional word vector related to the word and a D-dimensional context vector indicating that the word appears as a context of another word based on document data A learning method,
The learning unit divides each of the word vector and the context vector into B blocks having a dimension number F, and one word for a word pair in the document data and a set of S kinds of real vectors having the dimension number F. Based on the number of times the word vector and context vector appear as contexts of the other word, each block of the word vector and context vector is subject to any of the S types of real vectors. A word vector learning method comprising: estimating a word vector for each of the words and a real value vector of each block of the context vector so as to optimize an objective function representing the likelihood of each block. The step of estimating by the learning unit includes:
Each block r _{i, b} of the word vector for each word _{i and} each block c of the context vector for each word j so that the updating unit optimizes the objective function represented by updating _{j, b} real vectors;
5. The word vector learning method according to claim 4, further comprising: a step of repeating the update by the update unit until a repetition determination unit satisfies a predetermined repetition end condition.

Here, D represents a set of pairs of the word appearing in the document data and a word appearing as the context of the word, α _{i, b} and β _{i, b} are Lagrange undetermined multipliers, and u _{i , b} , v _{j, b} are auxiliary parameters that are real vectors of dimension F, ρ is a constant, ζ represents a set of S types of real vectors of dimension F, and X _{i, j} Represents the number of times word j appears as context with respect to word i. The natural language processing unit uses the word vector of each word learned by the word vector learning device according to claim 4 or 5 for the inputted input document, and makes semantics between words based on the word vector. A natural language processing method including a step of performing natural language processing based on a similar degree of similarity. The computer program to function as each unit constituting the word vectors learning equipment according to claim 1 or claim 2. The program for functioning a computer as each part which comprises the natural language processing apparatus of Claim 3.