JP7189068B2 - MODEL CREATED SUPPORT METHOD AND MODEL CREATED SUPPORT SYSTEM - Google Patents

Description

The present invention relates to a model creation support method and a model creation support system.

To improve the accuracy of an analysis model (inference model) using machine learning techniques, it is important to select teacher data and tune analysis parameters in the analysis model. However, when the desired analysis accuracy is not obtained, a problem arises as to whether there is a problem with the teacher data or with the setting of the analysis parameters.

In this regard, as a technology to support the accuracy improvement of the analysis model, there is a method of automatically adding unsupervised data that can be inferred as "correct" with high reliability based on the trained model to the supervised data (Patent Document 1), and the judgment result. There is known a technique of presenting rules that have influenced the (Patent Document 2).

JP-A-2005-92253 JP 2017-58816 A

However, with the technique described in Patent Literature 1, even if data that can be inferred with high reliability by an already existing trained model is added to the training data, a significant improvement in accuracy cannot be expected. On the other hand, in the technique described in Patent Document 2, it is necessary to manually input information extraction rules (feature values) from sentences into the system. The problem is that there are practical limits to the judgment.

The present invention has been made in view of such circumstances, and its object is to provide a model creation support method and a model creation support system that can reliably improve the accuracy of models generated by machine learning. to do.

One of the present invention for solving the above problems is a model creation support system equipped with a processor and a memory, by performing machine learning for specifying each feature amount for a plurality of learning target data, input A learning process for generating an inference model for estimating a label to be set to data based on the feature amount of the input data, and a feature amount of the predetermined data specified by inputting predetermined data into the generated inference model. and, by determining the similarity with the feature quantity of the learning target data specified by the machine learning, the validity of the label estimation by the inference model is determined, and information indicating the content of the determination is output. and an evaluation process to be performed.

ADVANTAGE OF THE INVENTION According to this invention, the precision of the model produced|generated by machine learning can be improved reliably.

FIG. 1 is a diagram showing an example of the configuration of a document analysis system 100 (model creation support system) according to this embodiment. FIG. 2 is a diagram showing an example of the configuration of the analysis node 2. As shown in FIG. FIG. 3 is a diagram showing an example of the data format of the document data 2121. As shown in FIG. FIG. 4 is a diagram showing an example of the teacher dictionary data 2122. As shown in FIG. FIG. 5 is a diagram showing an example of the inference model parameters 2123. As shown in FIG. FIG. 6 is a diagram showing an example of the inference result data 2124. As shown in FIG. FIG. 7 is a diagram showing an example of the confirmation label extraction rule 2126. As shown in FIG. FIG. 8 is a diagram showing an example of the feature quantity difference data 2125. As shown in FIG. FIG. 9 is a diagram showing an example of the confirmation label data 2127. As shown in FIG. FIG. 10 is a flowchart illustrating an example of document analysis processing. FIG. 11 is a flowchart illustrating an example of learning processing. FIG. 12 is a flowchart for explaining details of the evaluation process. FIG. 13 is a flowchart for explaining the details of the feature quantity difference extraction process. FIG. 14 is a flowchart for explaining details of the evaluation process. FIG. 15 is a diagram showing an example of the label confirmation screen. FIG. 16 is a flowchart for explaining details of inference processing.

The model creation support system of this embodiment will be described below with reference to the drawings. In the following description, information may be described using expressions such as "xxx table", but these information may be expressed in a data structure other than a table. Therefore, in order to indicate that it does not depend on the data structure, the "xxx table" and the like are sometimes called "xxx information". When describing the contents of each piece of information, identification information expressed as “number” and “name” is used, but other types of identification information may be used. "XXX process" in the following description may be "XXX program". The description with the subject of "processing" in the following description may be the description with the processor as the subject. Part or all of the processing may be implemented by dedicated hardware. Various programs may be installed in each computer by a program distribution server or a computer-readable storage medium.

FIG. 1 is a diagram showing an example of the configuration of a document analysis system 100 (model creation support system) according to this embodiment. The document analysis system 100 creates an inference model for inferring the semantic content of each word by performing machine learning for inferring the semantic content of each word in document data to be learned. Analyze the semantic content of documents in The document analysis system 100 is installed, for example, in a predetermined data center. The document analysis system 100 includes a terminal 3 that holds document data to be analyzed, and an analysis node 2 that generates an inference model and infers the document to be analyzed sent from the terminal 3 according to the inference model. is configured with Note that the analysis node 2 and the terminal 3 are connected via a network switch 4 . For example, the analysis node 2 and the terminal 3 are communicably connected by a network switch 4 via a wired or wireless communication network such as LAN (Local Area Network), WAN (Wide Area Network), the Internet, or a dedicated line. be.

The analysis node 2 and terminal 3 are composed of personal computers, workstations, or the like.
FIG. 2 is a diagram showing an example of the configuration of the analysis node 2. As shown in FIG. The analysis node 2 includes a processing unit 21 such as a CPU (Central Processing Unit), a memory 22 such as a RAM (Random Access Memory) or ROM (Read Only Memory), an FC (Fibre Channel) disk, an SC
A disk device 27 such as an SI (Small Computer System Interface) disk, a SATA disk, an ATA (AT Attachment) disk or a SAS (Serial Attached SCSI) disk, an input device 24 including a keyboard, a mouse, a touch panel, etc., and a monitor (display). etc., and a communication device 26 for communicating with other devices. The processing unit 21 controls the operation of the analysis node 2 as a whole, and executes necessary processing based on a control program group 211 and a management table group 212 which are stored in the memory 22 and will be described later. The memory 22 is used to store a control program group 211 and a management table group 212 to be described later, and is also used as a work memory for the processing unit 21 . The communication device 26 is a communication interface compatible with the network switch 4 and performs protocol control when the analysis node 2 communicates.

The analysis node 2 has functions of a learning unit 2111 , an evaluation unit 2112 , a feedback unit 2113 , and an inference unit 2114 as a control program group 211 . Analysis node 2 also stores document data 2121 , teacher dictionary data 2122 , inference model 200 , inference result data 2124 , confirmation label extraction rule 2126 , feature quantity difference data 2125 , and confirmation label data 2127 .

The learning unit 2111 receives a predetermined learning process request from the terminal 3 .

The learning unit 2111 generates an inference model 200 that estimates a label to be set to input data based on the feature amount of the input data by performing machine learning to specify each feature amount for a plurality of learning target data. Generate.

Specifically, the learning unit 2111 performs machine learning to specify the weight value of the feature amount, thereby creating the inference model 200 that estimates the label to be set to the input data based on the weight value of the feature amount of the input data. Generate. Note that in this embodiment, this label is a personal name label set for a word determined to be a personal name.

The inference model 200 includes an accuracy calculation formula 201 for calculating an accuracy, which will be described later, and an inference model parameter 2123 that is a parameter group used in the inference model 200 .

The inference model 200 estimates the label from the feature amount of the input data based on the accuracy, which is a parameter for determining the type of label to be set for the input data. Information on the estimated label is stored in confirmation label data 2127, which will be described later.

Note that the learning unit 2111 stores document data to be learned and document data to be analyzed in the document data 2121 . The learning unit 2111 also stores words extracted by machine learning from the document data recorded in the document data 2121 and labels set to the words in the teacher dictionary data 2122 as dictionary data.
Here, examples of the document data 2121 and teacher dictionary data 2122 will be described.

<Document data>

FIG. 3 is a diagram showing an example of the data format of the document data 2121. As shown in FIG. The document data 2121 includes learning document data 3121 in which learning target data is recorded and production document data 3122 in which analysis target data is recorded. Each of the learning document data 3121 and the production document data 3122 includes one or more document data. In the example shown in the figure, the document data for learning 3121 records the sentence "Mr. sasaki runs every morning". The learning document data 3121 is, for example, news article data, and the article includes words representing people's names.

<Teacher dictionary data>
FIG. 4 is a diagram showing an example of the teacher dictionary data 2122. As shown in FIG. The teacher dictionary data 2122 is
It includes a positive case table 21221 and a negative case table 21222 .

The positive example table 21221 has items in a personal name dictionary column 212211 in which words determined to be personal names (hereinafter referred to as positive examples) are stored. In addition, the negative example table 21222 has items in a personal name dictionary column 212221 in which words determined not to be personal names (hereinafter referred to as negative examples) are stored. In the example shown in the figure, "sasaki" and "tanaka" are registered as words determined to be positive, and "hitachi" and "amazon" are registered as examples of words determined to be negative.

Next, the details of the inference model parameter 2123 will be described.
<Inference model parameters>
FIG. 5 is a diagram showing an example of the inference model parameters 2123. As shown in FIG. The inference model parameter 2123 has a weight column 21231 in which variable names indicating weight values are stored, and a value column 21232 in which variable values (weight values) related to the weight column 21231 are stored.
<Accuracy calculation formula>
Next, the accuracy calculation formula 201 will be described. The accuracy calculation formula 201 is a formula expressed by a feature amount and its weight value, and in this embodiment,

P=w1*X1+w2*X2+w3*X3

shall be P is the probability, w is the weight value, and X is the feature quantity. Here, the weight w1 of the feature quantity X1 is 0.5, the weight w2 of the feature quantity X2 is 0.8, and the weight w3 of the feature quantity X3 is -0.1.

In this embodiment, when the probability P of a word to be analyzed is equal to or greater than a first threshold value (0.85 in this embodiment), the label of that word is set to "T" (for example, The word has a high probability of being a person's name, for example, the word has many positive feature values). Further, when the probability P of a certain word is less than the first threshold and equal to or greater than the second threshold (0.25 in this embodiment), the label of the word is set to "Null" (for example, the word is It is uncertain whether the word is a person's name, such as having positive and negative feature amounts). Also, if the probability P of a word is less than the second threshold, the label of that word is set to "F" (for example, if the word has many negative feature values, the word is less likely to be a person's name).

Here, in the following, the range of accuracy P for which "T" is set is the first area, the range of accuracy P for which "Null" is set is the second area, and "F" is set. The range of accuracy P is called a third region.

Note that the learning unit 2111 records the data of the result of generating the inference model 200 and the data obtained when generating the inference model 200 in the inference result data 2124 .

Here, an example of the inference result data 2124 will be described.
<Inference result data>
FIG. 6 is a diagram showing an example of the inference result data 2124. As shown in FIG. The inference result data 2124 includes a candidate field 21241 in which learned or analyzed words are stored (the inference result data 2124 contains not only the words to be learned but also the inference model 200 to the words of the sentence to be analyzed). result is also stored), a learning flag column 21242 that stores information indicating whether the word related to the candidate column 21241 is a positive example (“positive”) or a negative example (“negative”), and a candidate A sentence column 21243 storing sentences (learning document data 3121) in which words related to the column 21241 are registered, and information of feature amounts of words related to the candidate column 21241 (for example, "t" is a positive feature amount, " f” is a negative feature amount), a label column 21245 stores label information set for the word related to the candidate column 21241, and the probability P of the word related to the candidate column 21241 is and a certainty column 21246 to be stored.

The feature amount column 21244 stores the feature amounts of the words present in the sentences associated with the sentence column 21243 among the feature amounts of the words associated with the candidate column 21241 . Specifically, for example, in the feature amount item list in the feature amount column 21244, a word that exists in the sentence related to the sentence column 21243 is registered ("t").

Further, in the present embodiment, any one of "T", "F", and "Null" is set in the label column 21245. FIG. In the example shown in the figure, the word "sasaki" is data to be learned as a positive example ("positive"). This word is recorded in the sentence "Mr. sasaki runs every morning" (learning document data 3121), and this sentence includes the positive feature quantity of the element "run". As a result, the accuracy P of the word "sasaki" is "0.96", which is equal to or higher than the first threshold, so a label "T" indicating that the word is highly likely to be a personal name is set.

Next, as shown in FIG. 2, the evaluation unit 2112 receives a predetermined processing request from the terminal 3. FIG.
Then, the evaluation unit 2112 inputs predetermined data (a word to be learned or a new additional word) to the inference model 200 generated by the learning unit 2111, thereby identifying the characteristics of the predetermined data. The validity of the label estimation by the inference model 200 is determined by determining the similarity between the amount and the feature amount of the data to be learned, which has already been specified by machine learning in the learning unit 2111, and the content of the determination is determined. output the information shown. Note that the evaluation unit 2112 stores information about the similarity of feature amounts in the feature amount difference data 2125 . Also, the evaluation unit 2112 stores the determination result of the validity of the label setting in the confirmation label data 2127 .

Specifically, the evaluation unit 2112 sets a plurality of judgment rules for judging the similarity between feature amounts according to the accuracy, and judges the validity of label estimation by the inference model 200 based on the set judgment rules. judge. Note that the evaluation unit 2112 stores this determination rule in a confirmation label extraction rule 2126, which will be described later.

In addition, the evaluation unit 2112 evaluates the similarity between the feature amount of the predetermined data and the feature amount of the learning target data based on the feature amount that both data have in common (hereinafter referred to as duplicate feature amount) The validity of the label estimation is determined by specifying the feature amount that only the data has (hereinafter referred to as the difference feature amount).

Furthermore, the evaluation unit 2112 determines the validity of the weight value in the inference model 200 by determining the similarity between the weight value of the feature amount of the predetermined data and the weight value of the feature amount of the learning target data. . Note that the overlapping feature amount and the difference feature amount are stored in the feature amount difference data 2125, which will be described later.

Specific examples of the confirmation label extraction rule 2126 and the difference feature data 2125 will now be described.
<Confirmation label extraction rule>
FIG. 7 is a diagram showing an example of the confirmation label extraction rule 2126. As shown in FIG. The confirmation label extraction rule 2126 is information that stores determination rules, and stores a confirmation purpose column 21261 that stores information indicating the purpose of changing the label and information that specifies the content of the label change (label operation). a label operation column 21262, a region column 21263 storing information specifying a region to which a word to be changed in label belongs, and a rule column 21264 storing a determination rule.

The confirmation label extraction rule 2126 stores a determination rule for improving label accuracy, which is applied to words whose accuracy P belongs to the first region (first determination rule). Further, the confirmation label extraction rule 2126 stores a determination rule for improving recall, which is applied to words whose probability P belongs to the second region (second determination rule). Further, the confirmation label extraction rule 2126 stores a determination rule for improving recall, which is applied to words whose probability P belongs to the third region (third determination rule). Further, the confirmation label extraction rule 2126 stores determination rules for improving precision and recall, which are applied to all words (fourth determination rule).

For example, the purpose of the first determination rule is to "improve precision", that is, to find words that are incorrectly labeled even though they are not personal names. In this case, the first decision rule is "T ⇒ F", specifically, a word is incorrectly labeled (T)
change the label so that the word is not labeled (F). The first determination rule targets the words in area "1".

Here, the feature amount difference data will be described.
<Feature amount difference data>
FIG. 8 is a diagram showing an example of the feature quantity difference data 2125. As shown in FIG. The feature amount difference data 2125 includes a candidate column 21251 storing words to be learned or analyzed, a sentence column 21252 storing sentences (learning document data 3121) in which words related to the candidate column 21251 are registered, and candidate A label column 21253 storing information (“T”, “F”, “Null”) specifying the label assigned to the word related to the column 21251, a positive example relationship column 21254, and a negative example relationship column. It has 21255 items.

The relationship with positive example column 21254 lists the feature amounts (hereinafter referred to as positive example 21254a in which the overlapped feature amount) is stored, and the feature amount of the words related to the candidate column 21251 that are not included in the positive example words registered in the teacher dictionary data 2122 (hereinafter referred to as the positive example and a difference column 21254b in which a difference feature amount) is stored. In addition, the negative example relationship column 21255 includes feature quantities (hereinafter referred to as 21255a in which negative example overlapping feature amounts are stored, and among feature amounts possessed by words related to the candidate column 21251, feature amounts not possessed by negative example words registered in the teacher dictionary data 2122 (hereinafter referred to as difference column 21255b in which negative example difference feature amounts) are stored.

In the example shown in the figure, the word "sasaki" has a given label of "T", has "run" as a positive case overlap feature amount, and has "purchase" as a negative case difference feature amount. is doing.

Next, the confirmation label data will be explained.
<Confirmation label data>
FIG. 9 is a diagram showing an example of the confirmation label data 2127. As shown in FIG. The confirmation label data 2127 has a candidate column 21271 , a sentence column 21272 , a label column 21273 , a duplicate difference column with positive examples 21274 , a duplicate difference column with negative examples 21275 , and a confirmation label column 21276 .

Of these, the candidate column 21271 , text column 21272 , label column 21273 , positive example overlap difference column 21274 , and negative example overlap difference column 21275 are the same as the feature amount difference data 2125 . The confirmation label column 21276 stores information (confirmation label) indicating the determination result of the appropriateness of label setting for the word in the candidate column 21271 . For example, if there is a question about the appropriateness of the word label setting, "o" is stored in the corresponding confirmation label column 21276. FIG.

Next, as shown in FIG. 2, the feedback unit 2113 receives from the user correction of the inference model 200 generated by the learning unit 2111 based on the information (confirmation label) indicating the content of the determination by the evaluation unit 2112 .

Specifically, feedback section 2113 accepts correction of the weight value specified by learning section 2111 from the user.

The inference unit 2114 receives an inference request including the production document data 3122 from the terminal 3, and uses the inference model 200 to set labels for the words in the sentence indicated by the production document data 3122 (inference of words related to personal names). ), the semantic contents of the sentences related to the production document data 3122 are analyzed. Note that the inference unit 2114 registers this result in the inference result data 2124 .

The functions of the analysis node 2 described above are realized by the hardware of the analysis node 2 or by the processing unit 21 of the analysis node 2 reading and executing each program stored in the memory 22 or the disk device 27. Realized. In addition, these programs are, for example, secondary storage devices, non-volatile semiconductor memories, hard disk drives, storage devices such as SSD, or IC cards, SD cards, DVDs, etc. Stored on a data storage medium.

<<Processing>>
Next, document analysis processing for analyzing a document to be analyzed, which is performed by the document analysis system 100, will be described.
<Document analysis processing>
FIG. 10 is a flowchart illustrating an example of document analysis processing. First, the analysis node 2 executes a learning process of generating an inference model 200 that estimates the label corresponding to the input word by performing predetermined machine learning on each word in the sentence to be learned (SP1 ). Then, the analysis node 2 evaluates the appropriateness of label setting by the generated inference model 200 and executes evaluation processing for setting confirmation labels (SP2). Note that the analysis node 2 receives correction (feedback) of the inference model 200 from the user based on the confirmation label set by the evaluation process. Based on the revised inference model 200 and the labels, the analysis node 2 performs an inference process of estimating the label corresponding to each word in the document to be analyzed (SP3).
Details of each process will be described below.

<Learning processing>
FIG. 11 is a flowchart illustrating an example of learning processing. First, the learning unit 2111 of the analysis node 2 receives a request for learning processing from the user (SP901). Specifically, for example, the analysis node 2 receives the learning document data 3121 and the teacher dictionary data 2122 from the terminal 3 .

The learning unit 2111 registers the received document data for learning 3121 in the document data 2121 (SP902). The learning unit 2111 also registers the teacher dictionary data 2122 (SP902).

The learning unit 2111 generates the inference model 200 by machine learning based on the teacher dictionary data 2122 and the document data 2121 (SP903). The learning unit 2111 registers the result in the inference result data 2124 (SP904).

Specifically, for example, the learning unit 2111 extracts all words registered in the teacher dictionary data 2122 (hereinafter referred to as candidate words) from each sentence registered in the learning document data 3121 . Then, the learning unit 2111 extracts other words (other words in the learning document data 3121) appearing in a predetermined range of the extracted candidate words at a predetermined frequency or more as positive feature amounts by machine learning. (Specifically, a positive value is set for the weight value of the feature amount). On the other hand, the learning unit 2111 extracts other words (other words in the document data for learning 3121) that do not appear in the predetermined range of the extracted candidate words with a predetermined frequency or more as negative feature amounts by machine learning. (Specifically, a negative value is set for the weight value of the feature amount). This method is described in, for example, Ce Zhang, “DeepDive: A Data Management System for Automatic Knowledge Base Construction,” Doctoral dissertation of University of Wisconsin-madison, Mar.
2015.”

Specifically, for example, the learning unit 2111 replaces the candidate words (positive examples) “sasaki” and “tanaka” registered in the positive example table 21221 of the teacher dictionary data 2122 with “sasaki” in the learning document data 3121. runs every morning” and “today is the birthday of tanaka”. Then, the learning unit 2111 extracts the words "run" and "birth" around "sasaki" and "tanaka" as positive feature amounts for "sasaki" and "tanaka", respectively. Further, for example, the learning unit 2111 selects the candidate words (negative examples) “hitachi” and “amazon” registered in the negative example table 21222 of the teacher dictionary data 2122 as The person is Mr. odaira" and "I bought this clothes on Amazon". Then, the learning unit 2111 extracts the candidate word "founding" that does not appear around "hitachi" as a negative feature amount that does not appear around the person's name for "hitachi."

As described above, the learning unit 2111 specifies the feature amount for all combinations of all words in the teacher dictionary data 2122 and sentences in the learning document data 3121, thereby calculating the accuracy calculation formula 201. automatically generate an inference model 200 containing Note that the contents of the inference model 200 are registered in the inference model parameter 2123 .
For example, the accuracy calculation formula 201 is as follows.

Accuracy P = w1 * “Run” + w2 * “Birth” + w3 * “Foundation” +…
First threshold = 0.85
Second threshold = 0.25

Here, w1, w2, and w3 are weight values for feature amounts. By generating such an inference model 200 by machine learning, a weight value for each feature amount is determined. For example, a positive value is set as the weight value w2 for a feature quantity (for example, "birth") that statistically appears frequently around words of a person's name. In addition, a negative value is set as the weight value w3 for a feature amount that statistically does not appear frequently around words of a person's name (for example, "startup").

Next, the analysis node 2 inputs predetermined data (additional learning target words) that are not registered in the teacher dictionary data 2122 to the inference model 200, thereby specifying the feature amount of the additional learning words. At the same time, the accuracy P of the additional learning word is calculated, and the corresponding label is set (SP905). The analysis node 2 thereby completes the inference model 200 . It should be noted that words that have already been learned in the learning process may be reused instead of additional learning target words.

For example, the analysis node 2 inputs "suzuki" to the inference model 200, calculates its accuracy P, and registers the value (for example, "0.88") in the accuracy column 21246 of the inference result data 2124. . Since the accuracy P is greater than or equal to the first threshold, analysis node 2 registers the label "T" indicating that "suzuki" is likely to be a person's name in the label column 21245 of the inference result data 2124. Further, for example, the analysis node 2 assigns a label “F” indicating that the word with the calculated probability P less than the second threshold is unlikely to be a person's name.
is registered in the label column 21245. In addition, the analysis node 2 assigns a label "Null" indicating that it is uncertain whether or not the word is a person's name for the word for which the calculated probability P is less than the first threshold and equal to or greater than the second threshold. is registered in the label column 21245. The learning process ends here.

Next, details of evaluation processing for evaluating the generated inference model 200 will be described.
<Evaluation processing>

FIG. 12 is a flowchart for explaining details of the evaluation process. First, the evaluation unit 2112 of the analysis node 2 receives an evaluation processing request from the user (SP1001). Specifically, for example, it receives a predetermined input from the terminal 3 .

When the analysis node 2 receives the evaluation processing request, the feature amount of the word specified in the course of the learning process and the feature amount obtained by inputting predetermined data to the inference model 200 generated as a result of the learning process. is executed (SP1002). Then, the analysis node 2 executes confirmation label extraction processing for setting confirmation labels for words that satisfy a predetermined condition based on the result of the feature amount difference extraction processing (SP1003). Details of these processes will be described later.

The analysis node 2 displays a confirmation label presentation screen displaying the confirmation label set by the confirmation label extraction process, and executes a confirmation label presentation process for receiving a predetermined instruction from the user (SP1004). The details of the confirmation label presenting process will be described later.

The analysis node 2 executes feedback processing to input the received instruction to the inference model 200 or the inference result data 2124 (SP1005). The evaluation process ends here.

Here, the details of the feature quantity difference extraction process will be described.
<Feature amount difference extraction processing>

FIG. 13 is a flowchart for explaining the details of the feature quantity difference extraction process. First, the evaluation unit 2112 of the analysis node 2 generates a new record in the feature amount difference data 2125, and fills the candidate field 21251, the text field 21252, and the label field 21253 of the generated record with the inference result data generated in the learning process. The values of the candidate column 21241, text column 21243, and label column 21245 of 2124 are copied (SP1101).

Next, the evaluation unit 2112 determines the difference between the feature amount of the word determined to be a positive example in the course of the learning process and the feature amount of the word specified as a result of inputting the predetermined word into the inference model 200. Alternatively, information about duplication is registered in the feature amount difference data 2125 (SP1102).

That is, first, the evaluation unit 2112 registers information about the feature amount of the positive example word in the feature amount difference data 2125 . Specifically, the evaluation unit 2112 identifies all the feature amounts for which "t" is registered in the feature amount column 21244 of the record of the word whose learning flag column 21242 is "positive" in the inference result data 2124, Each specified feature amount is registered in the duplication column 21254a of the relation column 21254 with the positive example of each record of the feature amount difference data 2125 .

In addition, the evaluation unit 2112 registers information about feature amounts that positive example words do not have in the feature amount difference data 2125 . Specifically, the evaluation unit 2112 identifies all the feature amounts for which the feature amount column 21244 is unregistered in the record of the word whose learning flag column 21242 is “positive” in the inference result data 2124, and each identified feature amount is registered in the difference column 21254b of the relation column 21254 with the positive example of each record of the feature quantity difference data 2125.

Next, the evaluation unit 2112 registers information about the feature amount of the negative example word in the feature amount difference data 2125 (SP1103).

That is, first, the evaluation unit 2112 registers information about the feature amount of the negative example word in the feature amount difference data 2125 . Specifically, the evaluation unit 2112 identifies all the feature amounts for which "t" is registered in the feature amount column 21244 of the record of the word whose learning flag column 21242 is "negative" in the inference result data 2124, Each identified feature amount is registered in the overlap column 21255a of the relationship column 21255 with the negative example of each record of the feature amount difference data 2125 .

In addition, the evaluation unit 2112 registers, in the feature amount difference data 2125, information about feature amounts that negative example words do not have. Specifically, the evaluation unit 2112 identifies all the feature amounts for which the feature amount column 21244 is unregistered in the record of the word of which the learning flag column 21242 is “negative” in the inference result data 2124, and each identified feature amount is registered in the difference column 21255b of the relation column 21255 with the negative example of each record of the feature quantity difference data 2125.

In this way, the analysis node 2 identifies duplicate feature amounts (positive duplicate feature amount and negative duplicate feature amount) and differential feature amounts (positive differential feature amount and negative differential feature amount). The feature quantity difference extraction process is thus completed.

Next, details of the confirmation label extraction process will be described.
<Confirmation label extraction processing>
FIG. 14 is a flowchart for explaining the details of the confirmation label extraction process. The evaluation unit 2112 of analysis node 2 determines whether or not to set a confirmation label for each word.

First, the evaluation unit 2112 applies the first determination rule to determine the appropriateness of label setting for each word, and corrects the label (SP1201). That is, when a word belonging to the first region satisfies the first determination rule, the evaluation unit 2112 changes the label of the word from "T" to "F". By applying the first determination rule, if the word is not a person's name but is erroneously labeled, the label can be removed.

That is, first, the evaluation unit 2112 acquires the words in the first region with the accuracy P and the contents of the first determination rule. Specifically, for example, the evaluation unit 2112 identifies all words for which the value of the accuracy column 21246 of the inference result data 2124 is equal to or greater than the first threshold among the words of the feature quantity difference data 2125, and extracts confirmation labels. The contents of the label operation column 21262 and the rule column 21264 of the record in which "1" is stored in the area column 21263 of the rule 2126 are acquired.

Then, regarding the feature amount of each identified word, the evaluation unit 2112 determines that the weight of the positive case difference feature amount is unnecessarily small compared to the weight of the positive case duplicate feature amount. (Focus on "difference from positive example"), and set a confirmation label for such a word.

Specifically, for example, the evaluation unit 2112 identifies all records in the confirmation label data 2127 in which a word whose accuracy P is equal to or greater than the first threshold is registered in the candidate column 21271, and determines whether each record is correct. All feature amounts registered in the difference column 21274b of the overlapping difference column 21274 with the example are specified. Then, the evaluation unit 2112 refers to the inference model parameter 2123 to identify the feature amount with the smallest weight value among the identified feature amounts, and the word having the identified feature amount is registered in the candidate column 21271. The confirmation label column 21276 of the record of the confirmation label data 2127 is set (register "○").

Next, the evaluation unit 2112 applies the second determination rule to determine the appropriateness of label setting for each word, and corrects the label (SP1202). That is, the evaluation unit 2112 changes the label of the word from "Null" to "T" when the second determination rule is satisfied for a word whose probability P belongs to the second region. By applying the second determination rule, if the word is a person's name but the word is not labeled by mistake, the word can be labeled.

That is, first, the evaluation unit 2112 acquires the words whose accuracy P is in the second region and the content of the second determination rule. Specifically, for example, the evaluation unit 2112 determines the content of the candidate column 21241 of the record in which the accuracy column 21246 stores the accuracy P that is less than the first threshold value and is equal to or greater than the second threshold value among the words of the inference result data 2124. Identify all words that are The evaluation unit 2112 also acquires the contents of the label operation column 21262 and the rule column 21264 of the record in which "2" is stored in the area column 21263 of the confirmation label extraction rule 2126. FIG.

Then, the evaluation unit 2112 determines whether or not the feature amount of each identified word has a probability P less than the first threshold value because the weight is small even though the specified word has a duplicate positive example feature amount. is determined (focusing on the “difference from negative example”), and a confirmation label is set for such a word.

Specifically, the evaluation unit 2112 compares the overlap difference between the negative example and the record in which the candidate field 21271 stores a word whose accuracy P is less than the first threshold value and is equal to or greater than the second threshold value in the confirmation label data 2127. All feature amounts registered in the difference column 21275b of the column 21275 are specified. Then, the evaluation unit 2112 refers to the inference model parameter 2123 to identify the feature amount having the maximum weight value among the identified feature amounts, and the word having the identified feature amount is registered in the candidate column 21271. The confirmation label column 21276 of the record of the confirmation label data 2127 is set (register "○").

Next, the evaluation unit 2112 applies the third determination rule to determine the appropriateness of label setting for each word, and corrects the label (SP1203). That is, the evaluation unit 2112 changes the label of the word from "F" to "T" when the third determination rule is satisfied for a word whose probability P belongs to the third region. By applying the third determination rule, if the word is a person's name but the word is not labeled by mistake, the word can be labeled.

That is, first, the evaluation unit 2112 acquires the words in the third region with the accuracy P and the content of the third determination rule. Specifically, for example, the evaluation unit 2112 identifies all words for which the value of the accuracy column 21246 of the inference result data 2124 is less than the second threshold among the words of the feature quantity difference data 2125, and extracts confirmation labels. The contents of the label operation column 21262 and the rule column 21264 of the record in which "3" is stored in the area column 21263 of the rule 2126 are acquired.

Then, regarding the feature amount of each identified word, the evaluation unit 2112 determines that the weight of the negative case difference feature amount is smaller than the weight of the negative case overlap feature amount, so that the probability P of the word is less than the second threshold. It is determined whether or not such a word exists (focusing on the "difference from negative example"), and a confirmation label is set for such a word.

Specifically, the evaluation unit 2112 identifies all records in the confirmation label data 2127 in which a word whose accuracy P is less than the second threshold is registered in the candidate column 21271, and identifies each record as a negative example. all the feature amounts registered in the difference column 21275b of the overlapping difference column 21275 of . Then, the evaluation unit 2112 refers to the inference model parameter 2123 to identify the feature amount having the maximum weight value among the identified feature amounts, and the word having the identified feature amount is registered in the candidate column 21271. The confirmation label column 21276 of the record of the confirmation label data 2127 is set (register "○").

Finally, the evaluation unit 2112 applies the fourth determination rule to determine the appropriateness of label setting for each word, and corrects the label (SP1204). That is, when all words (words in all regions) satisfy the fourth determination rule, the evaluation unit 2112 changes the current label of the word “T” to “F” and changes the current label to “F”. Labeled "F"
For words that are , change to "T".
By applying the fourth decision rule, a correct label can be assigned to a word when the word is a person's name but the word is not labeled by mistake, or vice versa.

That is, first, the evaluation unit 2112 acquires the words in all regions and the content of the fourth determination rule. All words in the feature amount difference data 2125 are specified, and the contents of the label operation column 21262 and the rule column 21264 of the record in which "4" is stored in the region column 21263 of the confirmation label extraction rule 2126 are acquired.

Then, for each identified word, the evaluation unit 2112 determines whether there are other words (already learned positive or negative examples) for which different labels are set despite having the same feature amount. Determine whether or not (focusing on the content of the label), and set a confirmation label for each word.

Specifically, for example, the evaluation unit 2112 refers to each record of the confirmation label data 2127 so that the list of feature amounts registered in the overlapping column 21274a of the overlapping difference column 21274 with the positive example is common. , two words with different labels registered in the label column 21273 (one is "T" and the other is "F") are specified. Then, the evaluation unit 2112 sets the confirmation label column 21276 of each record in which the identified word is registered in the candidate column 21271 (registers "○"). With this, the confirmation label extraction processing ends.

As described above, in the present embodiment, a determination rule exists for each area, but a plurality of determination rules may exist for each area. Further, in the present embodiment, the feature amount of each word is compared with either one of the feature amount of the word of the "positive example" and the feature amount of the word of the "negative example". The feature amount of the "positive example" word and the "negative example" are selected such that the word having the feature amount with the smallest difference from the feature amount of the word and the largest difference from the feature amount of the negative example word is selected. It is also possible to use a determination rule that combines the distances of the feature amounts of the words.

In addition, in the present embodiment, overlapping feature amounts and difference feature amounts are used as feature amounts used in determination rules. The determination rule may be based on the weight value of the feature amount, such as selecting the feature amount with the largest weight value.

Furthermore, in the present embodiment, the number of feature values is used as the determination rule. If they are different, there is a high possibility that a person's name and a company name are mixed, so set a confirmation label for such a word." may

<Confirmation label presentation process>
Next, details of the confirmation label presenting process will be described. The confirmation label presenting process displays a label confirmation screen showing the setting status of the confirmation label.
FIG. 15 is a diagram showing an example of the label confirmation screen. The confirmation label presentation screen 1000 displays information on words for which confirmation labels are set (hereinafter referred to as confirmation words) (that is, information on records in which "○" is registered in the confirmation label column 21276 of the confirmation label data 2127). This is the screen to display.

The confirmation label presentation screen 1000 includes a label confirmation screen 1010 having a word display column 1012 displaying confirmation words (candidate column 21271) and a sentence display column 1014 displaying sentences (sentence column 21272) including the confirmation word. . This label confirmation screen 1010 also has an OK button 1016 that the user selects when the confirmation word is a person's name, and an NG button 1018 that the user selects when the confirmation word is not a person's name. When the OK button 1016 is selected, the confirmation word label is set to "T", and when the NG button 1018 is selected, the confirmation word label is set to "F". This allows the user to modify the labels by the inference model 200. FIG.

The confirmation label presentation screen 1000 also includes a feature amount confirmation screen 1020 . A feature quantity confirmation screen 1020 is displayed when the NG button 1018 is selected on the label confirmation screen 1010 . The feature amount confirmation screen 1020 displays the feature amount list display field 1022 that displays the feature amounts of the confirmed words (that is, the words for which "t" is registered in the feature amount column 21244 of the record related to the confirmed word in the inference result data 2124). ).

Each feature quantity list display column 1022 includes an OK button 1024 that is selected by the user when the feature quantity displayed there is appropriate as a word for judging a person's name, and an OK button 1024 that the user selects if the feature quantity displayed there is displayed. and an NG button 1026 that is selected by the user when the word is not appropriate as a word for judging a person's name. When the NG button 1026 is selected, the user can modify the corresponding feature amount or related parameters using a predetermined edit screen (not shown). For example, it is possible to delete the record related to the corresponding feature quantity in the inference model parameter 2123, or change the value of the value column 21232 (for example, decrease the value). Also, a value other than “t” can be set in the feature amount column 21244 of the record related to the confirmation word in the inference result data 2124 . As a result, the content of the inference model 200 can be modified appropriately.

The confirmation label presentation screen 1000 also includes an influence confirmation screen 1030 . The influence confirmation screen 1030 displays other words whose feature amounts change when the feature amount for which the NG button 1026 is selected in the feature amount list display field 1022 or the parameter related thereto is modified. 1032 and a sentence display column 1034 for displaying sentences including the word related to the other word display column 1032 . That is, in the other word display column 1032, the word (candidate column 21241) having the feature amount for which the NG button 1026 was selected, retrieved from the inference result data 2124, and the content of the sentence (sentence column 21243) containing that word. is displayed.

With the feature amount confirmation screen 1020 and the effect confirmation screen 1030 described above, the user can correct the inference result by the same operation as the label confirmation screen 1010, and based on this result, adjust the weight of the inference model parameter 2123. It is possible to determine the feature quantity to be used.

The confirmation label presentation screen 1000 also includes a degree-of-change adjustment screen 1040 . The degree-of-change adjustment screen 1040 includes an accuracy change display screen 1050 , a labeled area display screen 1060 , a slide bar 1070 for adjusting the weight of the feature amount, and a save button 1080 .

Accuracy change display screen 1050 displays accuracy parameters (pr
ecision, recall) changes are displayed.

The labeled area display screen 1060 displays a two-dimensional graph representing the relationship between the feature amount of a word and the label assigned to the word. Specifically, the vertical axis 1062 and the horizontal axis 1064 of the graph respectively represent the feature values displayed on the feature value confirmation screen 1020 . A point 1066 on the graph represents a word. If a point 1066 is inside a circle 1068 displayed on the graph, the word associated with that point 1066 is labeled. If a point 1066 is outside the circle 1068 displayed on the graph, the word associated with that point 1066 is not labeled. A word column 1069 corresponding to the point 1066 is provided for the point 1066 .

Each axis of the two-dimensional graph of the labeled region display screen 1060 is a mapping transformation that, when a word has two or more feature amounts, compresses those feature amounts and converts them into a two-dimensional graph. It may be an axis after being processed.

The slide bar 1070 accepts a change of the weight value of each feature amount (the value column 21232 of the inference model parameter 2123) from the user. When the weight value is changed using the slide bar 1070, the word is labeled according to the weight value adjustment amount (for example, the label is "F
”), or the word is given a new label (eg, labeled “T”). The user can confirm the content of the change on the labeling area display screen 1060 .

The save button 1080 accepts setting of the current weight value set by the slide bar 1070 to the value column 21232 of the inference model parameter 2123 . The document analysis system 100 can perform machine learning again based on this modified weight value to generate a new inference model.

Next, details of the inference process will be described.
<Inference processing>
FIG. 16 is a flowchart for explaining details of inference processing. First, the inference unit 2114 of the analysis node 2 receives an inference request from the user (SP1301). Specifically, for example, the inference unit 2114 receives the production document data 3122 from the terminal 3 .

The inference unit 2114 registers the received production document data 3122 in the document data 2121 (SP1302). Then, the inference unit 2114, based on the inference model 200 (inference model parameters 2123) whose label has been corrected by the confirmation label or the like in the evaluation process, for each word in the sentence recorded in the production document data 3122, Words and sentences are analyzed (SP1303). Then, the inference unit 2114 registers the data obtained by analyzing the word in the inference result data 2124 (SP1304). The inference processing ends here.

As described above, the document analysis system 100 of the present embodiment performs machine learning to identify each feature amount for a plurality of data (words) to be learned, so that the label to be set in the input data (" T”, “F”, “Null”, etc.) is generated based on the feature amount of the input data, and predetermined data (additional learning words) is input to the generated inference model 200. The validity of the label estimation by the inference model 200 is determined by determining the similarity between the feature amount of the identified additional learning word and the feature amount of the learning target data identified by machine learning. Output information (confirmation label) that indicates the content of judgment. This allows the user to determine whether or not the inference model 200 should be modified. As a result, the accuracy of the inference model 200 generated by machine learning can be reliably improved.

That is, the document analysis system 100 of this embodiment verifies the inference model 200 by comparing the similarity between the feature amount of the teacher data and the feature amount of the inference model. Even a user who has little knowledge about judging the propriety of inference can easily modify the inference model 200 to improve its accuracy. That is, even a user without analytical knowledge can tune the inference model 200 with a small number of man-hours.

Although the embodiments of the present invention have been described above, the embodiments of the present invention are not limited to those illustrated, and various modifications can be made without departing from the gist of the invention.

For example, although a person's name is mentioned here as an attribute of data for which a label is set, other attributes may be targeted.

Further, each function described in this embodiment may be composed of one program or divided into two or more program parts. Moreover, these programs may be arranged in either the analysis node 2 or the terminal 3, or may be provided in another information processing device.

At least the following will be clarified by the above description of the present specification. That is, the inference model estimates a label from the feature amount of the input data based on accuracy, which is a parameter for determining the type of label to be set in the input data, and the model creation support system estimates the evaluation In the processing, a plurality of determination rules for determining similarity between the feature quantities are set according to the accuracy, and the validity of label estimation by the inference model is determined based on the set determination rules. good.

In this way, by judging the appropriateness of label estimation based on a plurality of judgment rules for judging the similarity between feature quantities according to the accuracy, which is a parameter for judging the type of label, Accurate determination according to the type becomes possible.

Further, in the evaluation process, the model creation support system determines the similarity between the feature amount of the predetermined data and the feature amount of the learning target data, and The validity of the label estimation may be determined by specifying the feature amount that only the label has.

In this way, when judging the validity of label estimation, by identifying the common points (overlap) and differences (differences) of the feature values that are the basis for label setting, the validity of label estimation can be accurately determined. can be determined. That is, the accuracy of the inference model 200 can be determined by using the distance information between the feature amounts of the teacher data and the output data (data spouted out by the inference model 200).

Further, the model creation support system may execute feedback processing for accepting correction of the generated inference model from the user based on the information indicating the content of the determination.

In this way, by providing feedback for receiving corrections of the generated inference model 200 from the user, it is possible to improve the inference model 200 and increase its reliability, for example.

Further, in the learning process, the model creation support system estimates a label to be set to the input data based on the weight value of the feature amount of the input data by performing machine learning for specifying the weight value of the feature amount. the weight in the inference model by determining similarity between the weight value of the feature amount of the predetermined data and the weight value of the feature amount of the data to be learned in the evaluation process; Validity of the values may be determined, and the feedback process may accept corrections to the identified weight values from the user.

Thus, in the inference model 200 that estimates the label to be set to the input data based on the weight value of the feature amount of the input data, the weight value of the feature amount of predetermined data (additional learning word) and the data to be learned By determining the similarity with the weight value of the feature amount and receiving the correction of the weight value from the user, detailed tuning of the inference model 200 becomes possible, and the reliability of the inference model 200 can be further increased.

100 document analysis system, 2 analysis nodes, 3 terminals, 200 inference model

Claims (10)

A model creation support system comprising a processor and memory,
A learning process that generates an inference model that estimates a label to be set to input data based on the feature amount of the input data by performing machine learning that specifies each feature amount for a plurality of learning target data;
Determining similarity between the feature amount of the predetermined data specified by inputting the predetermined data into the generated inference model and the feature amount of the learning target data specified by the machine learning. an evaluation process for determining the validity of the label estimation by the inference model and outputting information indicating the content of the determination;
, a model creation support method. The inference model estimates a label from the feature amount of the input data based on accuracy, which is a parameter for determining the type of label to be set in the input data,
In the evaluation process, the model creation support system sets a plurality of determination rules for determining similarity between the feature amounts according to the accuracy, and estimates labels by the inference model based on the set determination rules. determine the validity of
The model creation support method according to claim 1. In the evaluation process, the model creation support system determines the similarity between the feature amount of the predetermined data and the feature amount of the learning target data, and the feature amount that both data have in common and only one data Determining the validity of the label estimation by specifying the feature amount having
The model creation support method according to claim 1. 2. The model creation support method according to claim 1, wherein said model creation support system executes feedback processing for accepting correction of said generated inference model from a user based on information indicating said content of determination. The model creation support system includes:
generating an inference model for estimating a label to be set to input data based on the weight value of the feature amount of the input data by performing machine learning for specifying the weight value of the feature amount in the learning process;
In the evaluation process, the validity of the weight value in the inference model is determined by determining the similarity between the weight value of the feature amount of the predetermined data and the weight value of the feature amount of the learning target data. ,
Receiving a correction of the identified weight value from a user in the feedback process;
The model creation support method according to claim 4. having a processor and memory,
a learning unit that generates an inference model that estimates a label to be set in input data based on the feature amount of the input data by performing machine learning that identifies each feature amount for a plurality of learning target data;
Determining similarity between the feature amount of the predetermined data specified by inputting the predetermined data into the generated inference model and the feature amount of the learning target data specified by the machine learning. an evaluation unit that determines the validity of the label estimation by the inference model and outputs information indicating the content of the determination;
A model creation support system. The inference model estimates a label from the feature amount of the input data based on accuracy, which is a parameter for determining the type of label to be set in the input data,
The evaluation unit sets a plurality of determination rules for determining similarity between the feature amounts according to the accuracy, and determines the validity of label estimation by the inference model based on the set determination rules.
The model creation support system according to claim 6. The evaluation unit specifies similarity between the feature amount of the predetermined data and the feature amount of the learning target data as a feature amount that both data have in common and a feature amount that only one of the data has. determining the validity of the label estimate by
The model creation support system according to claim 6. 7. The model creation support system according to claim 6, further comprising a feedback unit that accepts a correction of said generated inference model from a user based on information indicating said content of determination. The learning unit generates an inference model for estimating a label to be set to input data based on the weight value of the feature amount of the input data by performing machine learning for specifying the weight value of the feature amount,
The evaluation unit determines the validity of the weight value in the inference model by determining similarity between the weight value of the feature amount of the predetermined data and the weight value of the feature amount of the learning target data. ,
the feedback processing unit receives a correction of the generated weight value from a user;
The model creation support system according to claim 9.

Images