JP6770709B2 - Model generator and program for machine learning. - Google Patents

Description

The present invention relates to a machine learning model generator and a program.

Named entity recognition, which is also used as an elemental technology for applications that use natural language processing, is a technology that extracts proper expressions such as proper nouns contained in text, such as Support Vector Machine (SVM) and Conditional Random Fields (CRFs). A system using a machine learning method for dealing with identification problems (classification problems) such as is known.

Patent Document 1 below discloses a model parameter estimation method capable of estimating model parameters adapted to additional data while reducing the influence on the parameters of the existing model.

Non-Patent Document 1 below discloses Hidden-Unit CRF (HUCRF), which is a kind of derivative of CRFs in which a hidden unit layer is added to CRFs.

In Non-Patent Document 2 below, HUCRF is made to learn tentatively labeled data by clustering using canonical correlation analysis for an unlabeled data group, and the obtained parameters are actually labeled as initial values. A pre-learning method for performing additional learning with the data marked with is disclosed.

JP 2015-38709

L.maaten, M.welling, L.K.Saul "Hidden-Unit Conditional Random Fields" Int. Conf. On Artificial Intelligence & Statistics pp.479-488, 2011 Y.B.Kim, K.Stratos, R.Sarikaya, "Pre-training of Hidden-Unit CRFs" Association for Computational Linguistics, pp.192-198, 2015

In the identification problem, in general, supervised learning is performed by giving learning data (known data) to which a label (also called "tag") is already attached to learn, and model parameters obtained as a result of learning. Is used to identify unknown data (classification by assigning one of the labels). In order to achieve high accuracy by using such a machine learning method, a large amount of learning data (corpus or dictionary) is generally required.

For example, many documents in a company contain highly confidential information, and many documents cannot be understood without assuming the knowledge and rules unique to each company. Therefore, unique expressions are extracted for documents in a company. For that purpose, it is necessary to prepare a learning corpus independently. It is extremely costly to add a large amount of annotations (related annotation information) to such an in-house document, and as a result, even if an attempt is made to extract a unique expression for the in-house document, high accuracy is achieved. It is not realistic to prepare as much training data as possible.

Even in the case of a task for which a large amount of training data cannot be prepared, additional learning is performed for the target task with some of the model parameters (also called "models") pre-trained in another task that has a large amount of training data as initial conditions. By doing so, the learning accuracy (in the case of an identification problem, the accuracy when identifying unknown data using the obtained model) may be improved, but appropriate pre-learning conditions such that the accuracy is high. Is unknown, so it takes time to search for the condition.

An object of the present invention is to provide a machine learning model generator and a program capable of generating a model having higher learning accuracy as compared with the case where classes are not integrated at the time of pre-learning.

[Model generator for machine learning]
The present invention according to claim 1 comprises a first similarity calculation means for calculating the similarity between the classes based on the mutual relationship between the classes in which the relationship is defined in advance.
A second similarity calculation means that calculates the similarity between the classes based on the features of the classes, and
A determination means for determining whether or not the classes are similar based on the calculation results of the first calculation means and the second calculation means.
A class integration means for integrating classes determined to be similar by the determination means into one class,
Machine learning means for supervised machine learning processing,
It is a model generator for machine learning equipped with.

The present invention according to claim 2 is the machine learning model generation device according to claim 1, wherein the relationship is a hierarchical structure or a graph structure.

According to the third aspect of the present invention, the machine learning means includes a model parameter estimation means for estimating model parameters and a means for estimating posterior probabilities of data in each class of data from the learned model parameters. Item 2. The model generator according to item 1 or 2.

[program]
The present invention according to claim 4 includes a step of calculating the similarity between the classes as the first similarity based on the mutual relationship between the classes in which the relationship is defined in advance.
A step of calculating the similarity between the classes as a second similarity based on the features of the classes, and
A step of determining whether or not the classes are similar based on the first similarity and the second similarity, and
A step to integrate classes judged to be similar into one class,
Steps to perform supervised machine learning processing,
Is a program to make a computer execute.

According to the first aspect of the present invention, it is possible to generate a model in which the learning accuracy is higher than that in the case where the classes are not integrated at the time of pre-learning.

According to the second aspect of the present invention, in addition to the effect of the first aspect, it can be applied when the class relationship has a hierarchical structure or a graph structure.

According to the present invention according to claim 3, in addition to the effect of claim 1 or 2, it is determined whether or not the classes are similar based on the second similarity calculated based on the posterior probability. A model generator for machine learning is obtained.

According to the fourth aspect of the present invention, it is possible to generate a model in which the learning accuracy is higher than that in the case where the classes are not integrated at the time of pre-learning.

It is a block diagram which shows the functional structure of the model generation apparatus 10 which concerns on embodiment of this invention. It is a block diagram which shows the hardware structure of the model generation apparatus 10 which concerns on embodiment of this invention. It is a flowchart for demonstrating operation of the model generation apparatus 10 which concerns on embodiment of this invention. An example of the hierarchical structure of the class according to the first embodiment is shown. This is an example of the estimated probability of each instance in the class according to the first embodiment. It is a matrix which shows the correlation coefficient between learning classes which concerns on Example 1. The relationship between the hierarchical structure similarity between the learning classes according to the first embodiment and the hierarchical structure of the classes is shown. In the first embodiment, the hierarchical structure after the class integration processing is performed is shown. An example of the graph structure of the class according to the second embodiment is shown. This is an example of the estimated probability of each instance in the class according to the second embodiment. It is a matrix which shows the correlation coefficient between learning classes which concerns on Example 2. The relationship between the graph structure similarity between the learning classes according to the second embodiment and the graph structure of the classes is shown. In Example 2, the graph structure after the class integration process is shown.

FIG. 1 is a block diagram showing a functional configuration of the model generation device 10 according to the embodiment of the present invention. The model generation device 10 of the present embodiment includes a machine learning unit 11, a first similarity calculation unit 12, a second similarity calculation unit 13, a determination unit 14, a class integration unit 15, and an output unit 16. I have.

In the machine learning unit 11, machine learning processing is performed from the learning data 17 in which the labeled learning data is recorded, model parameters are estimated, and a class of which class the instance (learning data for each case) corresponds to. Estimate posterior probabilities for classification.

In the first similarity calculation unit 12, the vector amount generated based on the posterior probability of each instance estimated by the machine learning unit 11 (hereinafter, also referred to as “estimated probability”) is used as the feature amount of each class. The similarity between each class (“feature similarity”) is calculated based on the feature amount.

The second similarity calculation unit 13 calculates the similarity between each class (“relationship similarity”) based on the class relationship information 18 (for example, information on the “class hierarchy”). The initial value of the class relationship information is manually constructed based on the label of the learning data 16.

When the feature amount similarity and the relationship similarity exceed each predetermined threshold value, the determination unit 14 determines that the class pair is "similar".

The class integration unit 15 performs a process of integrating the class pairs determined to be “similar” by the determination unit 14 into one class, and updates the label of the learning data 17 and the information 18 of the relationship between the classes.

The output unit 16 outputs the model parameters estimated by the machine learning unit 11 when there is no class pair determined to be “similar” by the determination unit 14.

FIG. 2 is a diagram showing a hardware configuration of the model generator 10. The model generation device 10 includes a CPU 21, a memory 22, a storage device 23 such as a hard disk drive (HDD), and a display device 24, and these components are connected to each other via a control bus 25.

The CPU 21 executes a predetermined process based on a control program stored in the memory 22 or the storage device 23 (or provided from a storage medium (not shown) such as a CD-ROM), and the model generation device 10 Control the operation.

In carrying out the present invention, the model generation device 10 may further include various input interface devices such as a keyboard and a touch panel.

Next, with reference to the flowchart of FIG. 3, the operation of the model generator according to the present embodiment will be described by taking as an example the case where the feature amount of each class is calculated based on the estimated probability of each instance. In the examples shown below, when the predefined class relationships have the hierarchical structure shown in FIG. 4 (Example 1) and the graph structure shown in FIG. 9 (implementation). Example 2).

In step S1, each instance is executed after the machine learning process is executed in the lowest class of the hierarchical structure (“corporate name”, “political organization name”, “prefecture name”, “city / town / village name”, “personal name”). The estimated probability is calculated for, and the process proceeds to step S2. Hereinafter, the case where the estimated probability of each instance becomes the value shown in FIG. 5 will be described as an example. (The estimated probability of the data of instance 1 is corporation name: 0.1, political organization name: 0.1, prefecture name: 0.4, city / town / village name: 0.3, person name: 0.1. )

The feature quantity of each class can be calculated as, for example, a vector quantity having the value of the estimation probability of each instance as the value of each dimension.

In step S2, after the features of each learning class are generated from the estimated probabilities of each instance, the correlation coefficient between the learning classes is calculated based on the generated features of each learning class. In FIG. 6, it is represented as a correlation matrix of features.

In step S3, which can be performed in parallel with steps S1 and S2, the degree of hierarchical similarity between the learning classes is calculated based on the hierarchical structure of the classes. For example, by setting the similarity between classes (brother classes) existing under the same upper hierarchy (parent) to "1" and the similarity between other classes to "0", the similarity in the hierarchical structure is set. The degree can be calculated.

Taking the hierarchical structure of FIG. 4 as an example, the combinations of "corporate name" and "political organization name" and "prefecture name" and "city / town / village name" are the same parent ("organization name" and "organization name", respectively. Since the parent is the place name), the similarity in the hierarchical structure is "1", and for combinations other than these, the similarity in the hierarchical structure is "0". In FIG. 7, the similarity between each learning class is represented as a matrix.

After the completion of steps S2 and S3, the process proceeds to step S4, and in step S4, the correlation coefficient of the feature amount calculated in step S2 and the similarity in the hierarchical structure calculated in step S3 are each equal to or more than a predetermined threshold value. The learning class pairs that are are extracted as similar class pairs. As an example, when the threshold value of the correlation coefficient of the feature amount is 0.5 and the threshold value of the similarity in the hierarchical structure is 0.5, the pair of "city name" and "prefecture name" is similar. It will be extracted as a class pair.

In step S5 following step S4, it is determined whether or not a similar class pair exists, and if there is a similar pair, the process proceeds to step S6 and there is no similar class pair. If so, the process proceeds to step S7.

In step S6, the similar class pairs extracted in step S4 are integrated into one class. Specifically, the label of the class relationship information (hierarchical structure information) and the learning data recorded as the hierarchical structure is updated. If the pair of "city name" and "prefecture name" is a similar class pair, both are integrated, and the class name (label) after integration is, for example, a higher hierarchy name in the hierarchical structure ( Example: "place name") can be used (Fig. 8). After the class integration process, the process proceeds to steps S1 and S3.

In steps S1 and S3 that proceed again, the machine learning process and the calculation of the hierarchical similarity between the learning classes are performed again based on the updated hierarchical structure information and the labels of the learning data, respectively. Due to the update of the hierarchical structure, "place name" was added, and "city / town / village name" and "prefecture name" were deleted. As a result, the lowest class of the hierarchical structure was "corporate name", "political organization name", and " It becomes "place name" and "personal name".

In this way, the loop is repeated until there are no similar pairs, and finally in step S7, the model is output and the process ends.

In the second embodiment, the task of estimating the author from the text of the novel is assumed, and the relationship between the authors defined from the teacher-apprentice / friend relationship has a graph structure as shown in FIG. 9 ( It shows that there is a teacher-apprentice relationship and a friendship between the classes (novelist name) connected by a line).

In step S1, after the machine learning process is executed in all the classes constituting the graph structure, the estimated probability is calculated for each instance, and the process proceeds to step S2. Hereinafter, the case where the estimated probability of each instance becomes the value shown in FIG. 5 will be described as an example. (The estimated probabilities of the data of instance 1 are novelist A: 0.1, novelist B: 0.1, novelist C: 0.4, novelist D: 0.3, novelist E: 0.1. It has become.)

The feature quantity of each class can be calculated as, for example, a vector quantity having the value of the estimation probability of each instance as the value of each dimension.

In step S2, after the features of each learning class are generated from the estimated probabilities of each instance, the correlation coefficient between the learning classes is calculated based on the generated features of each learning class. In FIG. 11, it is represented as a correlation matrix of features.

In step S3, which can be performed in parallel with steps S1 and S2, the degree of similarity in the graph structure between the learning classes is calculated based on the graph structure of the classes. For example, the similarity in the graph structure can be calculated by setting the similarity between the classes directly connected by the line to "1" and setting the similarity between the other classes to "0".

Taking the graph structure of FIG. 12 as an example, for example, "Novelist A" has a degree of similarity in graph structure of "1" for combinations with "Novelist B", "Novelist C", and "Novelist D". , And the degree of similarity with "Novelist E" is "0". In FIG. 12, the similarity between each learning class is represented as a matrix.

After the completion of steps S2 and S3, the process proceeds to step S4, and in step S4, the correlation coefficient of the feature amount calculated in step S2 and the similarity in the graph structure calculated in step S3 are equal to or higher than a predetermined threshold value. The learning class pair that is is extracted as a similar class pair. As an example, when the threshold value of the correlation coefficient of the feature amount is 0.5 and the threshold value of the similarity on the graph structure is 0.5, the pair of "Novelist C" and "Novelist D" are similar. It will be extracted as a class pair.

In step S5 following step S4, it is determined whether or not a similar class pair exists, and if there is a similar pair, the process proceeds to step S6 and there is no similar class pair. If so, the process proceeds to step S7.

In step S6, the similar class pairs extracted in step S4 are integrated into one class. Specifically, the label of the class relationship information (graph structure information) and the learning data recorded as the graph structure is updated. As an example, FIG. 13 shows a graph structure after the integration process when the pair of “Novelist C” and “Novelist D” are similar class pairs. As the class name (label) after the integration, for example, a name that combines both names before the integration can be used. After the class integration process, the process proceeds to steps S1 and S3.

In steps S1 and S3 that proceed again, the machine learning process and the calculation of the similarity in the graph structure between the learning classes are performed again based on the updated graph structure information and the labels of the learning data, respectively.

In this way, the loop is repeated until there are no similar pairs, and finally in step S7, the model is output and the process ends.

As described above, the present invention can be applied to a machine learning model generator and a program.

10 Model generator 11 Machine learning unit 12 First similarity calculation unit 13 Second similarity calculation unit 14 Judgment unit 15 Class integration unit 16 Output unit 17 Learning data 18 Class relationship information 21 CPU
22 Memory 23 Storage device 24 Display device 25 Control bus

Claims (4)

A first similarity calculation means that calculates the similarity between the classes based on the mutual relationship between the classes whose relationships are defined in advance.
A second similarity calculation means that calculates the similarity between the classes based on the features of the classes, and
A determination means for determining whether or not the classes are similar based on the calculation results of the first calculation means and the second calculation means.
A class integration means for integrating classes determined to be similar by the determination means into one class,
A machine learning means that performs supervised machine learning processing in a class integrated by the class integration means ,
A model generator for machine learning equipped with. The machine learning model generator according to claim 1, wherein the relationship is a hierarchical structure or a graph structure. The model generator according to claim 1 or 2, wherein the machine learning means includes a model parameter estimating means for estimating model parameters and a means for estimating posterior probabilities of data in each class of data from the learned model parameters. .. A step of calculating the similarity between the classes as the first similarity based on the mutual relationship of the classes whose relationships are defined in advance, and
A step of calculating the similarity between the classes as a second similarity based on the features of the classes, and
A step of determining whether or not the classes are similar based on the first similarity and the second similarity, and
A step to integrate classes judged to be similar into one class,
Steps to perform supervised machine learning processing in an integrated class ,
A program that lets your computer run.

Images