JP2020154448A - Information extraction support device and information extraction support method and program - Google Patents

Abstract

To extract information with high accuracy even when a person extracting the information is not an expert.SOLUTION: An information support device according to an embodiment comprises an extraction part, an identification part, an assignment part, a learning inference part, and an output part. The extraction part extracts first information about document data from said document data containing multiple words. The identification part identifies relevant information from the first information as clue information. The assignment part assigns a label to the first information on the basis of a predetermined rule to indicate a correct or incorrect value. The learning inference part performs learning and inference of a rule for extracting new first information on the basis of the first information, a feature quantity used to extract the first information, and the label. The output part outputs a method for changing at least one of the feature quantity and the rules on the basis of the label and the correctness judgment which is a result of the correctness or incorrectness about the first information.SELECTED DRAWING: Figure 1

Description

An embodiment of the present invention relates to an information extraction support device, an information extraction support method, and a program.

For example, specific information in a document can be organized by using an information extraction technique that extracts attributes such as product name and product price and relationships between attributes such as the relationship between product name and product price. .. For example, a list of product specifications can be extracted from a document and summarized in a table. It is often realized by using machine learning technology, but in many cases, clues (features) useful for learning while performing evaluation, and positive examples (correct answers) and negative examples (incorrect answers) necessary for learning are given. I will improve the way. For example, in Patent Document 1, the learning performance is expanded by expanding the learning data, such as adopting the latter if the accuracy of the learning result by the data expanded by changing the word order of the dependency is better than the learning result by the original correct answer data. Technology has been proposed to improve.

Japanese Unexamined Patent Publication No. 2006-4399

However, if the side that extracts the information is not an expert, there is a problem that the influence of the contents generated in the improvement process is not taken into consideration.

The information support device of the embodiment includes an extraction unit, a specific unit, an addition unit, a learning inference unit, and an output unit. The extraction unit extracts the first information about the document data from the document data including a plurality of words. The specific unit specifies related information as clue information from the first information. The granting unit assigns a label indicating correctness to the first information based on a predetermined rule. The learning inference unit executes learning and inference of rules for extracting new first information based on the first information, the feature amount used for extracting the first information, and the label. The output unit outputs at least one method of changing the feature amount and the rule based on the label and the correctness determination which is the result of the correctness of the one information.

The figure which shows the example of the functional structure of the information extraction support apparatus of 1st Embodiment. The flowchart which shows the example of the information extraction support method of 1st Embodiment. The figure which shows the example which extracts as attribute information. The figure which shows the example which judges the correctness of the extracted attribute. The figure which shows the example of the change method by the correctness judgment of an attribute. The figure which shows the specific example which shows the result of a correctness judgment. The figure which shows the example of the hardware composition of the computer used for the information extraction support apparatus of 1st Embodiment. The figure which shows the example of the apparatus configuration of the information extraction support apparatus of 1st Embodiment.

For example, when extracting information using machine learning, the features to be used may be changed, and rules and criteria for labeling training cases as positive (correct) or negative (incorrect) may be linked. To do. As a result, it is difficult to understand how to correct it before and after learning. Furthermore, as the evaluation value (score) changes over time, the same evaluation may be performed many times for extraction results having similar properties. In this embodiment, based on the information extraction result and the transition of the evaluation result, the learning features and the method of changing the positive / negative example labeling to the training data are presented, and the target to be evaluated can be easily determined. Explain the technology that provides support.

The information extraction support device, the information extraction support method, and the embodiment of the program will be described in detail with reference to the attached drawings.

(First Embodiment)
First, the information extraction support device of the first embodiment will be described.

[Example of functional configuration]
FIG. 1 is a diagram showing an example of a functional configuration of the information extraction support device 101 of the first embodiment. The information extraction support device system 101 of the first embodiment includes an extraction unit 102, a specific unit 103, an addition unit 104, a learning estimation unit 105, and an output unit 106.

The extraction unit 102 acquires one or more target documents 107 to be analyzed. The target document is, for example, news articles, papers, patent specifications, etc. uploaded on a Web page or the Internet, and is document data (text data) expressed in a natural sentence composed of a plurality of words. The target document may be acquired by inputting from the user, or may be automatically collected. The document data may be any data that can extract attribute information including attributes and relationships between attributes. For example, the attribute indicates the type of information desired to be extracted by the user, and examples thereof include a product name, a price, a company name, a material name, and a characteristic value. The relationship between attributes is a product name and a product price. The extraction unit 102 extracts candidate attribute information from the acquired document according to a predetermined rule. For example, the words, sentences, titles in the document, the date and time and format in the metadata attached to the document, the number of figures and tables, etc. are extracted according to predetermined rules.

The identification unit 103 identifies the information related to the attribute information extracted by the extraction unit 102 as clues information representing features. The clue information includes meta information of the document itself from which it is extracted (for example, creation date and time, file format of the document, language, etc.), characteristics of the word itself (noun, verb, etc.), and a person's name. Classification information of unique expressions such as and place names, words and phrases that appear before and after words, ngram, etc., features of figures and tables included in documents (classification of figures such as photographs, graphs, illustrations, number of rows and columns of tables, etc. ) Etc. are included.

The giving unit 104 labels the attribute information extracted by the extracting unit 102 with known information in which the correct answer is labeled in advance for training or a temporary label for training with respect to the training case data using a predetermined rule. Label. Here, the "temporary label" indicates a label attached using a predetermined rule. For example, if labels for part of speech such as nouns and verbs are given by natural language processing, the label of "correct answer" can be given in most cases. On the other hand, when labeling the pronunciation of named entities such as personal names and place names as correct or incorrect, it may not be possible to deal with them with predetermined rules, but here it does not matter whether the answer is actually correct or not. , Grant according to the set rules.

The learning inference unit 105 has a correct / incorrect relationship represented by the training case based on the attribute information extracted by the extraction unit 102, the clue information extracted by the specific unit 103, and the label (training case) given by the giving unit 104. Learn the evaluation criteria. After learning, the attribute information extracted by the extraction unit 102 is inferred by calculating the probability of being the desired attribute information.

At the time of learning, for example, the priority of using the clue information accompanying the attribute information regarded as the correct example (correct answer) in the training case is set higher. The priority of using the clue information accompanying the attribute information determined to be a negative example (incorrect answer) may be set low. The extracted attribute information, clue information, and training case may be stored in the storage unit and acquired and used at the time of learning or inference.

A reception unit that accepts correct / incorrect judgments from the user may be provided to accept input of correct / incorrect judgments for the information extraction result.

The correct / incorrect judgment result input may be presented to the user so that only the information extraction result and the correctness, which are the judgment targets, can be checked, and the result input may be accepted, or only the correct answer may be accepted. Further, the determination result may be input by connecting to another system having a correctness determination function. When accepting the result input from the user, for example, a check box may be added to check whether the information extraction result is correct or incorrect, and the user may check either of them to accept the correct / incorrect judgment result. At this time, if either or both of the extracted attribute information, the relationship between the attributes, the clue information, and the bibliographic information of the extraction source document are presented together with the information extraction result, how the information was extracted. It's even better because it allows the user to guess.

Further, if necessary, the likelihood and certainty of the extraction result may be described together as a score.

The score is obtained by calculating the similarity using the Levenshtein distance and the Jaro Winkler distance of letters and words. Other methods may be used as long as the similarity of letters and words can be calculated. These scores and correct / incorrect determination results may be stored in the storage unit.

Based on the information on the tentative label of correctness and the label information of the correctness judgment on the extracted attribute information, the output unit 106 outputs at least one method of changing the feature amount or the rule (rule) used by the learning inference unit 105. To do.

Here, the information extraction process in the information extraction support device 101 and the change of the clue information extraction and the labeling rule through the determination result input will be described with reference to the flowchart of FIG.

In step S201, the extraction unit 102 acquires one or more target documents. An example of extracting attribute information from the target document and extracting clue information will be described with reference to FIG.

FIG. 3 is an example of extracting the material name, the characteristic value called its mobility, and the relationship between the two as attribute information from the text.

First, a plurality of words are extracted from the notation "... the mobility of Si is 5,700 cm2 / Vs ..." which is a part of the target document by using morphological analysis or the like. Next, the material name “Si” and the mobility value “5,700” from the extracted plurality of words are extracted as attribute information using predetermined rules.

Further, "Si" and "5,700" are specified as a pair of the material name and its mobility. The set A of the clue information for specifying the attribute information includes, for example, 3 words included between the values of "material" and "mobility", and "no", "mobility", and "ha" as words. The rules for that are set in advance.

Here, it is assumed that the set of clues information such as the number of words is changed by the learning and the execution of the estimation by the learning estimation unit 106. For example, a new set of clue information A'that the word "cm2 / Vs" appears immediately after the mobility value when attribute information is extracted from a new target document and learning is performed using these. Suppose it is added.

Based on the new set of clue information A', the attribute information is extracted from the target document, and the extraction result and the feature set and the score corresponding to the extraction result are calculated. An item to be evaluated is determined based on a predetermined rule from the extraction result based on the new feature set A', the past extraction result, and the label of the assigning unit 104 (step S202). For example, when the given label changes, the extraction result is set as the evaluation target.

In step S203, the result of the correctness determination for the evaluation target is received.

FIG. 4 is an example of determining (evaluating) the correctness of the extracted attributes. When the user makes a correct / incorrect judgment, for example, the extraction result is selected and displayed by a predetermined criterion or randomly, and the correct answer is set as ◯ and the incorrect answer is set as ×.

Here, FIG. 5 shows an example of the relationship between the set of clue information and the number of evaluations for which correct / incorrect judgment is performed. Here, it is assumed that the set of clues information, the score for each trial of the learning estimation unit 106, and the relationship between the evaluation results are stored in the storage unit as the evaluation results of a plurality of times.

For example, in the set A, the evaluation is correct (◯) in all three trials. When the clue information set of the set A is obtained, the evaluation result is set as the correct answer (○). In this case, there is no improvement proposal in terms of improving the accuracy of information extraction.

Specifically, as shown in the result of the correctness determination in FIG. 6, the evaluation result is correct based on the clue information of the set A even in different target documents or different sentences in the same document.

For example, in the first evaluation, the set of cases "Si" and "5,700" is set as "correct answer (○)", and the correctness judgment (evaluation) is set even when the "mobility" is different as in the second and subsequent evaluations. You may. This is because even if the notations of "material" and "mobility" are similar, if they are not exactly the same, the accuracy of the feature amount and the clue information can be expected to be improved. Preferably, it is preferable that patterns having different "materials" or "mobilities" can be evaluated as shown in FIG. 6 rather than the same pattern being evaluated many times.

Return to FIG. In the set B of FIG. 5, the same number of correct answers (○) and incorrect answers (×) appear in the correct / incorrect judgment (evaluation). Therefore, it is not possible to determine whether the current clue information set is suitable for information extraction. That is, there is a possibility that the addition of clue information or the current set of clue information is not suitable for the desired information extraction. In this case, the information extraction support device 101 outputs that additional clue information (feature amount) for correct / incorrect determination is required.

Further, in the set C of FIG. 5, since there are more incorrect answers (x) in the correctness judgment (evaluation), the data itself used for learning or the labeling rule may not be appropriate. In this case, the information extraction support device 101 outputs that the data used for learning needs to be changed. If there is a problem with the data used for training itself, a warning that training using new data is necessary may be output.

Further, in the middle of the evaluation, as shown in the set D of FIG. 5, neither the correct answer (◯) nor the incorrect answer (x) may be shown. The information extraction support device 101 may output the content prompting the evaluation.

Return to FIG. In step S204, a proposal for changing the feature extraction and labeling rules is presented based on changes in the past extraction results, extraction results, corresponding feature sets, scores, and evaluation results.

For the change proposal, for example, the evaluation history stored in the storage unit is used. The following modification is an example of setting the rules for long-distance supervised learning (Disstant supervised learning).

(Proposed change 1) Proposal / information extraction result e based on one evaluation result, feature set A of e, when the evaluation result is correct (○) count (feature set A, evaluation result ○)> threshold value and count If (feature set A, evaluation result ○) / count (feature set A, evaluation result ×)> threshold value, the rule “If the feature set is A, the answer is correct (○).

(Change proposal 2) Improvement proposal based on the results of multiple evaluations (change in characteristics)
-When the information extraction result e, the feature set A of e, the evaluation result is correct (○), and the feature set for the information extraction result e is A',
Rule "If the feature set is A', it is ○".

(Change proposal 3) Improvement proposal based on the results of multiple evaluations (change in score)
(3a) When the score, for example, the probability value decreases with the same feature, it is output that "an example of new clue information (feature) is needed".
For example, if the information extraction result e, the feature set A of e, the probability value 0.95, and the evaluation result are ○ in the first execution, and the probability value drops to 0.7 in the second execution, only the set A Then, since the accuracy is lowered, an example of "new feature" may be presented.
(3b) When the feature changes and the score, for example, the probability value decreases, "diff (A, A')" is output as a factor for reducing accuracy.
For example, in the first execution, the information extraction result e, the feature set A of e, the probability value 0.95, the evaluation result is ○, the feature set changes to A'in the second execution, and the probability value is (for example). When it drops to 0.7, it is considered that the feature set is easy to change, so the difference between the feature sets is output.
Further, in order to generate a highly versatile rule, a correct / incorrect judgment may be made for the intersection (A∩B) of the feature set A and the feature set B, and the rule may be created as a candidate.

Return to FIG. In step S205, one or more of the output change proposals are reflected.

As described above, the output change proposal is reflected, the correct / incorrect judgment (evaluation) is executed multiple times, and the extracted candidates, the clue information set, the number of labeled correct / incorrect answers, and the actual evaluation result are recorded. To do. The distribution change of the correctness judgment result for the recorded extraction candidate and the clue information set is calculated, and the clue information and the labeling rule change method are output according to the distribution change. It becomes possible to determine the target to be evaluated according to the content of evaluation / change, and it becomes possible to efficiently improve the accuracy.

As described above, it is possible to change the feature amount or the rule. Further, according to the first embodiment, it is possible to specify the evaluation target even when the feature amount and the method of changing the rule are not known, so that the information extraction side can extract information with high accuracy even if the person who extracts the information is not an expert. Will be possible.

Finally, an example of the hardware configuration of the computer used for the information extraction support device 101 of the first embodiment will be described.

[Example of hardware configuration]
FIG. 7 is a diagram showing an example of the hardware configuration of the computer used in the information extraction support device 101 of the first embodiment.

The computer used for the information extraction support device 101 includes a control device 501, a main storage device 502, an auxiliary storage device 503, a display device 504, an input device 505, and a communication device 506. The control device 501, the main storage device 502, the auxiliary storage device 503, the display device 504, the input device 505, and the communication device 506 are connected via the bus 510.

The control device 501 executes the program read from the auxiliary storage device 503 to the main storage device 502. The main storage device 502 is a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage device 503 is an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, or the like.

The display device 504 displays the display information. The display device 504 is, for example, a liquid crystal display or the like. The input device 505 is an interface for operating a computer. The input device 505 is, for example, a keyboard, a mouse, or the like. When the computer is a smart device such as a smartphone or a tablet terminal, the display device 504 and the input device 505 are, for example, a touch panel. The communication device 506 is an interface for communicating with another device.

Programs that run on a computer are recorded in a computer-readable storage medium such as a CD-ROM, memory card, CD-R, or DVD (Digital Versailles Disc) in an installable or executable format file. Provided as a computer program product.

Further, the program executed by the computer may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. Further, the program executed by the computer may be configured to be provided via a network such as the Internet without being downloaded.

Further, the program executed by the computer may be configured to be provided by incorporating it into a ROM or the like in advance.

The program executed by the computer has a module configuration including a functional block that can be realized by the program among the functional configurations (functional blocks) of the above-mentioned information extraction support device 101. As actual hardware, each functional block is loaded on the main storage device 502 by the control device 501 reading a program from the storage medium and executing the program. That is, each of the above functional blocks is generated on the main storage device 502.

It should be noted that a part or all of the above-mentioned functional blocks may not be realized by software, but may be realized by hardware such as an IC (Integrated Circuit).

When each function is realized by using a plurality of processors, each processor may realize one of each function, or may realize two or more of each function.

Further, the operation mode of the computer that realizes the information extraction support device 101 may be arbitrary. For example, the information extraction support device 101 may be realized by one computer. Further, for example, the information extraction support device 101 may be operated as a cloud system on the network.

[Example of device configuration]
FIG. 8 is a diagram showing an example of the device configuration of the information extraction support device 101 of the present embodiment. In the example of FIG. 8, the information extraction support device 101 includes a plurality of client devices 1a to 1z, a network 2, and a server device 3.

When it is not necessary to distinguish between the client devices 1a to 1z, it is simply referred to as the client device 1. The number of client devices 1 in the information extraction support device 101 may be arbitrary. The client device 1 is, for example, a computer such as a personal computer and a smartphone. The plurality of client devices 1a to 1z and the server device 3 are connected to each other via the network 2. The communication method of the network 2 may be a wired method or a wireless method, or both may be combined.

For example, the extraction unit 102, the identification unit 103, the granting unit 104, the learning estimation unit 105, and the output unit 106 of the information extraction support device 101 may be realized by the server device 3 and operated as a cloud system on the network 2. For example, the client device 1 may receive the result of the correctness determination from the user and transmit the result of the correctness determination to the server device 3. Then, the server device 3 may transmit the change method output by the output unit 106 to the client device 1.

Although the embodiments of the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 Client device 2 Network 3 Server device 101 Information extraction support device 102 Extraction unit 103 Specific unit 104 Grant unit 105 Learning estimation unit 106 Output unit

Claims (5)

An extraction unit that extracts the first information about the document data from the document data containing a plurality of words,
A specific part that identifies related information as clue information from the first information,
An assigning unit that assigns a label indicating correctness to the first information based on a predetermined rule,
A learning inference unit that executes learning and inference of rules for extracting new first information based on the first information, a feature amount used for extracting the first information, and the label.
An output unit that outputs at least one method of changing the feature amount and the rule based on the label and the correctness determination that is the result of correctness of the one information.
Information extraction support device with. The information extraction support device according to claim 1, wherein the output unit outputs a combination of the change method, correctness determination of the extraction result of the extraction unit, the clue information, and at least one of the feature amounts. It also has a reception unit that accepts the correctness judgment from the user.
The information extraction support device according to claim 1, wherein when the reception unit receives a content indicating that it is not clear whether the answer is correct or incorrect, the clue information collection also outputs another similar result. A step of extracting the first information about the document data from the document data containing a plurality of words, and
A step of identifying related information as clue information from the first information,
A step of assigning a label indicating correctness to the first information based on a predetermined rule, and
A step of learning and inferring a rule for extracting new first information based on the first information, a feature amount used for extracting the first information, and the label, and the label and the first information. An information extraction support method including a step of outputting the feature amount and at least one change method of the rule based on the correctness judgment which is the result of the correctness of the above. An extraction unit that extracts the first information about the document data from the document data containing a plurality of words on the computer.
A specific part that identifies related information as clue information from the first information,
An assigning unit that assigns a label indicating correctness to the first information based on a predetermined rule,
A learning inference unit that executes learning and inference of rules for extracting new first information based on the first information, a feature amount used for extracting the first information, and the label.
An information extraction support program for functioning as an output unit that outputs at least one change method of the feature amount and the rule based on the correctness determination which is the result of correctness of the label and the one information.