JP6375210B2 - Model construction apparatus and program - Google Patents

Description

  The present invention relates to a model construction apparatus and program capable of performing latent topic analysis considering a hierarchy between topics at high speed.

  In recent years, as a method of analyzing discrete data such as documents and purchase history, the generation process of documents expressed in bag-of-words (bug of words) is stochastically modeled, which is a potential factor that cannot be observed directly Topic models that enable high-accuracy clustering based on the topic are attracting attention.

  A feature of the topic model is that a single document is expressed as a mixture of multiple topics. The typical method is latent dirichlet allocation (LDA), which is an information search, voice It is applied to various data mining fields such as recognition and QA system. Here, LDA is disclosed in Non-Patent Document 1, and LDA application to a QA system is disclosed in Patent Document 1, respectively.

  In addition, as disclosed in Patent Document 2 and Non-Patent Document 2, there is a hierarchical topic model in which topics have a hierarchical structure in order to make the classification result easier to use.

JP 2013-143066 JP 2013-134750 A

D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3: 993-1022, 2003. W Li, A McCallum, "Pachinko allocation: DAG-structured mixture models of topic correlations," Proc ICML, 2006

  In Non-Patent Document 2 (this method is abbreviated as “PAM”), a super topic node is determined from the upper level to the lower level using Gibbs sampling so that the pachinko falls, assuming a hierarchical structure of topics. The relationship between each word appearing in the document and the topic is sought. The PAM method has a problem that many calculation processes are required to obtain the relationship between each document and topic.

  FIG. 1 is a diagram illustrating an example of a hierarchical structure of the topic. Higher topics include “medicine” and “economy”, and lower topics include “regenerative medicine”, “local medicine”, “new drug development”, “financial crisis”, and “trade liberalization”. The upper and lower nodes having a hierarchical relationship are connected by arrows as shown in the figure. “New drug development” is a common low-level topic for both “medicine” and “economy”. Although the example of FIG. 1 has an upper and lower two-layer structure, an n-layer structure (n ≧ 2) is generally possible.

  FIG. 2 is a diagram for illustrating a calculation process when obtaining a relationship between each document and a topic in the hierarchical structure in PAM. In PAM, the probability of the path zi from the root node to the corresponding word is calculated, and θ (D) and φ (W (W) are calculated based on the probability P (zi | w) from each topic node to the word w obtained from the probability. ) Will be updated. Here, θ (D) and φ (W) are respectively the topic ratio for each document and the word distribution for each topic in the latent topic analysis.

  As shown in the example of FIG. 2, the path zi exists as a number of combinations because it follows a root node, a plurality of supertopic nodes s, a plurality of topic nodes κ, and a plurality of word nodes ν. A process is required. The super topic node s corresponds to the upper topic, and the topic node κ corresponds to the lower topic.

  Patent Document 2 uses a hierarchical topic model to find words useful for interactive search. However, it is assumed that the hierarchical topic model to be used has already been constructed, and Patent Document 2 does not mention a method for obtaining the relationship between words and topics that actually appear in each document. Therefore, as in the conventional method such as Non-Patent Document 2, many calculation processes are required in the process of classifying documents by hierarchical topics.

  In view of the above-described problems of the conventional technology, an object of the present invention is to provide a model construction apparatus and program capable of constructing a hierarchical topic model at high speed.

  In order to achieve the above object, the present invention is a model construction apparatus that performs latent topic analysis on a series of target data expressed in bugs of words, and outputs the analysis result as a model, the topic of each target data An initial setting unit that sets an initial value for a weight matrix, a topic weight matrix for each word, and Gibbs sampling sequentially for each matrix that is set with the initial value, as the output model An update calculation unit that obtains each matrix, and in the output model, a lattice structure is given as a hierarchical structure between topics, and the update calculation unit performs the Gibbs sampling sequentially When updating from the old topic to the new topic for each element of the topic weight matrix of each target data, Distance between the old topic which comprises carrying out in limited to a predetermined value or less in the lattice structure.

  In addition, the present invention is a program that causes a computer to function as the model construction device.

  According to the present invention, a lattice structure connection is provided as a hierarchical structure between topics, and topic update candidates in the Gibbs sampling process are limited to topics with a short distance on the hierarchical structure. Can be built.

It is a figure which shows the example of the hierarchical structure of a topic. It is a figure for showing the calculation process at the time of calculating | requiring the relationship between each document and a topic for hierarchical structure. It is a functional block diagram of the model construction device concerning one embodiment. It is a figure which shows the example of the initialization process of matrix (D) in normal LDA etc. FIG. It is a figure which shows the example of the initialization process of matrix (phi) (W) in normal LDA etc. FIG. It is a flowchart of the Gibbs sampling process in normal LDA etc. It is a figure which shows the example for demonstrating the update process in FIG. It is a figure which shows the example of the lattice structure used as a hierarchical structure.

  FIG. 3 is a functional block diagram of the model construction device according to an embodiment. The model construction device 10 includes an initial setting unit 1 and an update calculation unit 2. The model construction apparatus 10 reads a document group as an input, and outputs a model as a result of performing a latent topic analysis such as LDA.

  In the present invention, the calculation framework for constructing the output model is the same as the calculation framework disclosed in Non-Patent Document 1 and used in ordinary LDA or the like that does not consider the hierarchical structure between topics. . Here, in particular, a model that considers the hierarchical structure under the same computational load as ordinary LDA, etc. that does not consider the hierarchical structure, by calculating with the method described later in detail within the common framework. Can be output. That is, it is possible to output a model that considers the hierarchical structure in the same way as PAM without accompanying a large calculation load that repeats calculation for a large number of paths as in PAM of Non-Patent Document 2. Therefore, each unit 1 and 2 of the model construction device 10 specifically performs the following processing.

  First, the initial setting unit 1 reads a group of documents each expressed as a bug of word, and performs the same initialization process as when calculating a normal topic model such as LDA. That is, by assigning topics randomly to each document and every word that appears, a matrix θ (D), D = {d1, d2,…} representing the weight of the topic in each document d1, d2,. Matrix φ (W), W = {w1, w2,…} representing topic weights in w1, w2,... are set as initial values, and matrix θ (D), φ (W) as the initial values are set. To update calculation unit 2.

  FIG. 4 is a diagram illustrating an example of the initialization process of the matrix θ (D) in a normal LDA or the like. (1) shows that the hyper parameters α and β are set to 0.1 and 0.01, respectively, as an example of LDA calculation setting information, and the word frequency (bug of (Word) information.

Then, as shown separately in (21), (31) and (22), (32), the document 1 and the document 2 are processed in common. First, as shown in (21) and (22), topics are set at random for the number of words appearing for each document. For example, for document 1, as shown in (1), there are 5 words A, 3 words B, and 2 words C, respectively. Only 4 topics [0], [1], [2], [3] are randomly assigned.
Five topics assigned to word A… [3], [0], [3], [1], [2]
Three topics assigned to word B… [1], [0], [3]
Two topics assigned to word C… [3], [1]

  Note that topics are appropriately expressed in the form of “array” as in [0], [1], [2], [3]. (The expression is an expression for clarifying the description of the update process described later.)

Next, as shown in (31) and (32), the topic ratio is obtained by counting the number of the randomly assigned topics for each topic type. For example, for document 1, as shown in (31) and below, the number of topics is aggregated to obtain the topic ratio.
Topic ratio of document 1 (topic0, topic1, topic2, topic3) = (2, 3, 1, 4)

  Finally, the topic ratios θ (D) of all initialized documents are obtained by enumerating the topic ratios for each document in a matrix format across all the documents, as shown in (4).

  FIG. 5 is a diagram illustrating an example of initialization processing of the matrix φ (W) in a normal LDA or the like, and is a diagram illustrating an example corresponding to the example of FIG. First, as shown in (21) and (22), information prepared at the initialization of θ (D), in which topics are set at random for the number of words appearing for each document, is processed. That is, (21) and (22) are common to FIGS.

  Next, as shown in (51) and (52), the information for (21) and (22) is totaled for each word and topic in Document 1 and Document 2, respectively. Obtain the distribution φ (W).

For example, in (51), the appearance frequency of word A in document 1 is 5 times, and topics are randomly assigned as follows.
Five topics assigned to word A… [3], [0], [3], [1], [2]

Accordingly, as shown in the first line of the tabular data portion of (51), the topic allocation count of word A in document 1 is as follows.
Number of topics assigned to word A in document 1 (topic0, topic1, topic2, topic3) = (1, 1, 1, 2)

Then, the number of allocations is similarly obtained for the remaining words B, C, and D of the document 1, and the result of all the words of the document 1 is totaled to obtain the word distribution as shown in (51). For example, as shown in the first column of the tabular data portion of (51), the word distribution of topic 0 is as follows.
Topic 0 word distribution (word A, word B, word C, word D) = (1, 1, 0, 0)

  Finally, by summing up the word distribution obtained for each document in (51) and (52) over all documents, the word distribution for each topic in all documents initialized as shown in (6) φ (W) is obtained.

  In the update calculation unit 2, by performing sequential update processing on the initialized θ (D) and φ (W) obtained by the initial setting unit 1 as described above, from the model construction device 10 Find the final θ (D) and φ (W) as outputs. Here, the update processing framework itself can use the same processing as the Gibbs sampling process performed in ordinary LDA, etc., and the method specific to the present invention is included in the contents of each successive update processing. Used.

  Accordingly, in the following, the Gibbs sampling process in a normal LDA or the like will be described first, and then the update process unique to the present invention will be described.

  FIG. 6 is a flowchart of a Gibbs sampling process in a normal LDA or the like, and FIG. 7 is a diagram illustrating an example for explaining an update process in the flowchart. The example of FIG. 7 corresponds to the example shown in FIGS.

  When the flow of FIG. 6 is started, first, in step S1, the counter i of the document di that is a counter for controlling the target of the update process in the flow, the counter j of the number of occurrences of words in the document di, and loop processing The number counter k is set to an initial value (generally 0 or 1), and the process proceeds to step S2.

  In step S2, Gibbs sampling is performed, and then the process proceeds to step S3. That is, a topic corresponding to the j-th word in the document di specified by the counter (i, j) at the time is newly determined based on the matrices φ (W), θ (D) at the time, The matrices φ (W) and θ (D) are updated according to the newly determined topic. Details of step S2 will be described later with reference to FIG.

  In step S3, it is determined from the value of the counter j whether processing for the number of occurrences of all words in the document di has been completed. If it has been completed, the value of the counter j is reset to the initial value, and step S4 is performed. If not completed, the process proceeds to step S31. In step S31, the value of the counter j is incremented by 1, and the process returns to step S2. Accordingly, the loop processing L3 for each document di as shown in the figure is configured by the determination in step S3.

  In step S4, it is determined from the value of counter i whether processing has been completed for all documents di. If completed, the value of counter i is reset to the initial value, and then the process proceeds to step S5. If not, the process proceeds to step S41. In step S41, the value of the counter i is incremented by 1, and the process returns to step S2. Accordingly, the loop processing L4 for every document di (i = 1, 2,...) As shown in the figure is configured by the determination in step S4.

  In step S5, it is determined whether or not the conditions for completion of the entire loop processing in FIG. 6 are satisfied. If satisfied, the flow ends. If not satisfied, the process proceeds to step S51 and the value of the counter k is set. After incrementing by 1, return to step S2. Accordingly, the loop processing L5 as the entire flow as shown in the figure is configured by the determination in step S5.

  Here, as the completion condition of the entire loop processing, the counter k has reached a predetermined value, the matrices φ (W), θ (D) obtained at the immediately preceding k−1th time, and the current k The fact that the difference from the matrix φ (W), θ (D) obtained in the first round is less than a predetermined value can be used.

  According to the flow of FIG. 6 as described above, the matrices φ (W) and θ (D) held at the time when it is determined that the completion condition is satisfied in step S5 are output as final results, that is, models. It will be.

  Here, the details of the processing in step S2 in FIG. 6, that is, Gibbs sampling will be described with reference to FIG. In FIG. 7, the step S2 (that is, the first step S2) immediately after starting the flow of FIG. 6 for the initialized matrices φ (W) and θ (D) shown as examples in FIGS. Processing will be described as an example. In FIG. 7, the processing contents are the same in the general cases after the first time as shown on the right side that the loop processing L3, L4, L5 common to FIG. 6 is performed.

  Since this is the initial processing, as shown in bold and underlined in (21) in FIG. 7 (common with (21) in FIGS. 4 and 5), the document 1 (counter i = initial value 1) [3], which is the first topic (counter j = initial value 1) in the topic array assigned by the total number of words, is to be updated. Then, as shown in (7), a new ID determination process, that is, a replacement process with a new topic is performed for [3] to be updated.

  In the determination process of (7), as shown in (510) and (310), the matrices φ (W) and θ (D) at that time are referred to. Note that (510) represents the reference location in the matrix φ (W) shown in (51) of FIG. 5, and (310) represents the reference location in (31) of FIG.

When a new ID is specifically determined in (7), the word A to which the topic [3] (j = 1) to be updated is assigned is referred to the matrix φ (W). A random variable that outputs topics with the probability of following the topic ratio is used. In the example of (7), the conditional probability is as follows.
P (topic0 | word A) = 1 / (1 + 2 + 1 + 3)
P (topic1 | word A) = 2 / (1 + 2 + 1 + 3)
P (topic2 | word A) = 1 / (1 + 2 + 1 + 3)
P (topic3 | word A) = 3 / (1 + 2 + 1 + 3)

  That is, a new topic ID is determined by rolling a “biased dice”, which is determined by the matrix φ (W) as described above. Here, it is assumed that the new ID is [2] by the dice. (In general, the new and old IDs may be the same as a result of the decision.)

  In (8), according to the decision in (7), [3], which is the old ID, is replaced with [2], which is the new ID, and the array of topics updated from the state in (21) is shown. Yes. Then, as shown in (511) and (311), the matrices φ (W) and θ (D) are also updated from the states of (510) and (310), respectively, according to the updated topic arrangement. It will be. Note that the updating of the corresponding part of the matrices φ (W) and θ (D) may be performed by the same aggregation process as that for the initial value setting described with reference to FIGS. In the example of (511) and (311), as a result of the topic [3] being replaced with [2], the frequency of [3] is reduced by 1 from (510) and (310) before the update, respectively, and [2 ] 'S frequency has increased by 1.

  The Gibbs sampling in ordinary LDA or the like has been described above with reference to FIGS. The update calculation unit 2 performs Gibbs sampling according to a flow structure that repeats sequentially as in FIG. 6. However, by adding constraints to each update process in step S 2, the calculation load is reduced and finally The model to be constructed can take into account the hierarchical structure. Specifically, the following (Constraint 1) and (Constraint 2) are added.

  (Constraint 1) Limit the number of topics to a power of 2, and give each topic a topic label that expresses the ID in binary. In addition, as a hierarchical structure between topics, nodes whose Hamming distance between the topic labels expressed in binary numbers is 1 are connected by edges, and nodes whose Hamming distance is n pass through n edges. The lattice structure as a graph that can be reached (the number of hops is n) is adopted. Here, the presence of an edge between nodes means that there is a hierarchical connection between topics corresponding to the node.

  FIG. 8 is a diagram illustrating an example of the lattice structure used as a hierarchical structure. FIG. 8 shows an example of a lattice structure in which nodes are represented by topic labels assigned to each topic ID when the number of topics is set to 2 to the fourth power (= 16).

  In the above description, reference has been omitted from the viewpoint of the description flow, but the initial setting unit 1 also follows the restriction on the number of topics in the above (Restriction 1). The initial value setting of the matrices φ (W) and θ (D) after that is the same as the normal method as described above.

  (Restriction 2) In the process of replacing the topic ID described with reference to FIG. 7 from the old ID to the new ID, the new ID candidates (transition destination candidates) are not all topics as in the normal method. The lattice structure is limited to those that can be reached with a predetermined n or less hops (transition between nodes) (that is, the Hamming distance is a predetermined number n or less).

  This limitation simplifies the calculation of the probability P (z | w) (where z is a topic ID and w is a word), and can reduce the calculation load. Here, the probability may be standardized within the limited range.

If the lattice structure of FIG. 8 is used, for example, if the topic label of the old ID to be updated is “0100” and the candidate for the new ID (transition destination) is a Hamming distance of 1 or less, Transition destination topic labels are limited to five types: “0100”, “0000”, “1100”, “0110”, or “0101”. Regarding the respective probabilities, the following probabilities obtained by referring to the matrix φ (W) may be standardized. Note that w is a word to which the topic with the old ID to be updated is assigned.
P (0100 | w), P (0000 | w), P (1100 | w), P (0110 | w), P (0101 | w)

  The model construction by the model construction apparatus 10 described above is the first embodiment. Next, another embodiment (second embodiment) of the model construction will be described.

  In the first embodiment, after assigning topic IDs in a lattice structure, the initial setting unit 1 sets initial values of the matrices φ (W) and θ (D), and the update calculation unit 2 performs Gibbs sampling of the initial values. Updated to obtain the final matrices φ (W), θ (D). On the other hand, in the second embodiment, topic IDs are sequentially allocated by a lattice structure, and the initial setting unit 1 and the update calculation unit 2 perform model construction for each allocation. I do.

Specifically, if the set of topic IDs assigned to the i-th (i = 1, 2,…, m) is written as G (i), the following relationship (a relationship that increases as topics are added sequentially) ) To set each set G (i) in advance. Here, G (m) allocated at the last m-th time is assumed to be the entire lattice structure used in the first embodiment composed of powers of 2.
G (1) ⊂G (2) ⊂G (3) ⊂… ⊂G (m-1) ⊂G (m)

  Then, in the order of G (1), G (2),..., G (m), the initial setting unit 1 and the update calculation unit 2 perform model construction in the same manner as in the first embodiment, and the last G ( The result obtained for m) is adopted as the model to be finally constructed. Here, calculation by Gibbs sampling at the time of model construction is the same as that of the first embodiment, and only the point that the set of allocated topic IDs is limited to G (i) is different from the first embodiment.

  Particularly in the second embodiment, if the matrix of the result of model construction for topic ID allocation G (i) is φ (W) [i], θ (D) [i], the i-th result φ ( W) [i], θ (D) [i] are used as initial values in the initial setting unit 1 for topic ID allocation G (i + 1) to be processed next. By using the initial value, the initial value can be set closer to the final result than when it is set at random, reducing the number of calculations until convergence to the final result and speeding up the processing. be able to.

  In the initial model construction with G (1), the initial value is set at random. Further, in the first model construction with G (1), the processing of the update calculation unit 2 may be the same processing as that of normal LDA or the like. That is, the above (Constraint 2) may be omitted.

  For the topic “G (1) ⊂G (2) ⊂G (3) ⊂… ⊂G (m-1) ⊂G (m)” that increases sequentially, for example, it can be set as follows: .

First, for G (1), if n is an even number that gives the total number of topics as the nth power of 2, _n C _{n / 2} (= n · (n-1) · (n-2) ... (n / 2 + 1) / (1 ・ 2 ・ 3 ・ ・ ・ (n / 2))), when n is an odd number _n C _{(n-1) / 2} (= n ・ (n-1) ・ (n- 2) ... (n + 1) / 2 / (1 · 2 · 3 ... (n-1) / 2)) IDs are selected. In the case of the example of FIG. 8 (n = 4), it is drawn in the middle part of the figure, and G (1) can be constituted by the six IDs listed below.
G (1) = {1100, 1010, 1001, 0110, 0101, 0011}

Note that _n C _i is generally i (i = 0, 1, 2) when each node of the lattice structure composed of 2 n nodes is divided into n + 1 stages as shown in FIG. ,…, N) corresponds to the number of nodes in the stage. Here, there is a relationship that the number of hops is 2 and there is no edge between the nodes in the i-th stage (except for i = 0 and n-th stage because there is only one node). The selection of G (1) corresponds to selecting a node in the middle stage as much as possible among the n + 1 stages. Accordingly, when n is an odd number, _n C _{(n + 1) / 2} may be used instead of the above _n C _{(n-1) / 2} .

The subsequent G (i) (i ≧ 2) is configured by newly adding a label ID that can be reached with a Hamming distance = 1 with respect to the label ID included in G (i−1). In the example of FIG. 8, G (2) and G (3) are configured as follows, and reach G (3) including the entire lattice structure including 2 4 = 16 label IDs. be able to.
G (2) = {1110, 1101, 1011, 0111} ∪G (1) ∪ {1000, 0100, 0010, 0001}
G (3) = {1111} ∪G (2) ∪ {0000}

  Instead of the above, for the subsequent G (i) (i ≧ 2), the Hamming distance for the label ID included in G (i-1) is a predetermined number n or less (the number of hops between nodes is predetermined). It may be configured by newly adding a label ID that can be reached in a few n or less). The above is an example of the predetermined number n = 1.

  As described above, according to the present invention, the implementation of hierarchical topic classification can be realized by slightly changing the implementation of the topic model construction method using the conventional Gibbs sampling without hierarchy. This implementation can speed up the update process of θ (D) and φ (W) compared to the conventional hierarchical topic model. In particular, in the case of the method of sequentially increasing the number of topic classifications in the second embodiment, it is possible to further increase the speed.

  Hereinafter, supplementary matters in the present invention will be described.

  (1) In both the first and second embodiments, the lattice structure in the model output as the final result by the model construction device 10 uses all the 2n label IDs as described above. In addition, only a predetermined part of the label IDs having the number of 2 to the nth power may be used. Even when only a part is used, the Gibbs sampling process is possible as described above. Also, the setting of G (1), G (2),..., G (m) in the second embodiment is also set so that the last G (m) is an ID set consisting of only the predetermined part. Is possible as well.

  (2) The input data to be model-constructed by the model building device 10 can be a bug-of-word expression (BoW expression) of each document configured as normal text, or any other target other than text The bug of word expression can be used in exactly the same way. For example, as is well known, the feature amount can be expressed as Bag of Visual Words, but this is a kind of bug of word expression, so input the image expressed in the bug of visual word. It may be data.

  (3) The present invention can also be provided as a program that causes a computer to function as the model construction device 10. The computer can be configured with known hardware such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU that reads and executes the program functions as each unit of the model construction device 10. .

  10 ... Model building device, 1 ... Initial setting unit, 2 ... Update calculation unit

Claims (8)

A model construction device that performs a latent topic analysis on a series of target data expressed in a bug of word and outputs the analysis result as a model,
An initial setting unit that sets initial values for the topic weight matrix of each target data and the topic weight matrix of each word;
An update calculation unit that obtains each matrix as the output model by sequentially performing Gibbs sampling on each matrix set with the initial value, and
In the output model, the lattice structure is given as a hierarchical structure between topics,
The update calculation unit, when performing the Gibbs sampling sequentially, updates the old topic candidate to the lattice when updating the old topic to the new topic for each element of the topic weight matrix of each target data. A model construction apparatus characterized in that the construction is limited to a structure in which the distance from the old topic is a predetermined value or less.   The lattice structure is given as a connection between topics by providing an edge between nodes where each topic is a node and the Hamming distance between labels in which the topic ID is expressed in binary is 1. The model construction apparatus according to claim 1.   The update calculation unit, when performing an update from the old topic to the new topic, limits the candidates for the new topic to those that can be reached from the old topic in the lattice structure with a predetermined number of hops or less. 3. The model construction apparatus according to claim 2, wherein the probability of selecting a candidate is obtained by normalization from a topic weight matrix of each target data at the time of updating. In the initial setting unit and the update calculation unit, a set G (i) is sequentially applied to a set G (i) (i = 1, 2,..., M) of a series of topics having the relationship of the following expression (1). ), The result of the latent topic analysis under the topic specified by the set G (m) is output as the model. Item 4. The model construction device according to any one of Items 1 to 3.
G (1) ⊂G (2) ⊂… ⊂G (m)… Equation (1)   In the initial setting unit, the latent topic analysis is performed under the topic specified by the set G (i + 1), based on the result of the latent topic analysis performed under the topic specified by the set G (i). The model construction apparatus according to claim 4, wherein the model construction apparatus is used as an initial value when performing the operation.   The topic constituting the set G (i + 1) is obtained by adding a topic that can be reached with a predetermined number of hops or less in the lattice structure to the topic constituting the set G (i). The model construction device according to claim 4 or 5.   The model construction apparatus according to claim 1, wherein the series of target data expressed in the bug of word is a document or other than a document.   A program that causes a computer to function as the model construction device according to any one of claims 1 to 7.