JP2022148430A - Document information extraction system and document information extraction method - Google Patents

Abstract

To provide a document information extraction system and method that can assist in setting feature quantities of a model used when extracting useful information from a document, by a machine learning method.SOLUTION: A document information extraction system manages a document group, extracts information from a document of the document group using a model of machine learning, stores a plurality of feature quantities constituting parameters of the model and action types that are information indicating characteristics of actions performed when extracting the information for each of the feature quantities, displays an evaluation target feature quantity that is a feature quantity to be evaluated among the plurality of feature quantities, at least one sentence of an extraction source of the evaluation target feature quantity, and a weight currently set to the evaluation target feature quantity, receives action type of the evaluation target feature quantity or the setting of the weight, and updates the parameters on the basis of the received action type or weight.SELECTED DRAWING: Figure 1

Description

The present invention relates to a document information extraction system and a document information extraction method.

Patent Literature 1 describes an information processing apparatus that selects a feature amount to be used for classifying input data from a plurality of feature amounts extracted from input data. The information processing device generates a combination of a plurality of feature amounts extracted from the input data, and calculates a first evaluation value for evaluating whether the generated combination is suitable for determining the classification of the input data. Then, generate multiple parameters to be used when calculating the second evaluation value for evaluating the combination of feature amounts, calculate the second evaluation value based on the first evaluation value for each of the multiple parameters, and calculate the second evaluation value for each of the multiple parameters A feature is selected based on the second evaluation value to generate a subset of the feature so that the feature can be selected in the selection of the combination of the features.

Patent Literature 2 describes a feature amount classification system that classifies feature amounts in speech recognition and image recognition. A feature quantity classification system includes a feature quantity converter that converts a given feature quantity and learns discrimination criteria for performing feature quantity conversion using correct class labels as teacher data, and a feature quantity converter. and a classifier for classifying the transformed feature quantity, wherein the feature quantity converter classifies the classifier of the other feature quantity classification system during the learning in order to consider the error tendency of the classifier of the other feature quantity classification system. Using the posterior probability of , a feature quantity conversion different from that of other feature quantity converters is performed.

Japanese Patent Application Laid-Open No. 2017-10318 WO2016/021060

As a method of extracting useful information from a huge amount of documents, there is a technique of extracting words and related terms from documents using a mechanism of machine learning such as named entity extraction. However, in order to extract high-quality information using a machine learning mechanism, it is necessary to appropriately set the feature values of the model. Here, the feature amount used for extracting information from a document is, for example, the property of a sentence such as the final word of a word or words appearing to the left and right of a word. Appropriate setting of such feature values requires not only knowledge of the information to be extracted (domain knowledge), but also knowledge and experience in machine learning. The problem is that the human load is high.

Japanese Patent Laid-Open No. 2002-200003 describes a technique for selecting feature amounts used for classifying input data. However, the technique described in the document relates to a technique for extracting a group of feature values such as the mean and variance of pixel values from an image of an object to be inspected, and does not relate to a technique for extracting useful information from a document. . In addition, in the same document, feature values are selected by a mechanical round-robin method for combinations of multiple feature values extracted from the input data, and it is necessary to prepare a considerable amount of training data in order to obtain extraction accuracy. There is also

Patent Literature 2 describes a technique for classifying feature amounts in speech recognition and image recognition. However, this document merely describes a technique for adjusting a classifier used for classifying given feature amounts, and does not describe a technique for setting feature amounts.

SUMMARY OF THE INVENTION The present invention has been made in view of such a background. , and a document information extraction method.

One of the present inventions for achieving the above object is a document information extraction system, comprising an information processing device, a document management unit for managing a group of documents, and a machine learning model for extracting the documents. a document information extraction unit for extracting information from a group of documents; a plurality of feature quantities constituting parameters of the model; an evaluation target feature quantity that is a feature quantity to be evaluated among the plurality of feature quantities, one or more sentences from which the evaluation target feature quantity is extracted, and the evaluation target a parameter setting reception unit that receives setting of the action type or the weight of the evaluation target feature amount while displaying the weight currently set for the feature amount; and the parameter based on the received action type or the weight and a parameter updating unit that updates the

In addition, the problems disclosed by the present application and their solutions will be clarified by the description of the mode for carrying out the invention and the drawings.

According to the present invention, it is possible to support the setting of the feature amount of a model used when extracting useful information from a document by a machine learning method.

1 is a system flow diagram illustrating the main configuration of a document information extraction system; FIG. 1 is a block diagram illustrating the main configuration of a document information extraction system; FIG. It is a figure explaining the evaluation method of the correctness-likeness of the feature-value used for determination of a priority. It is an example of a feature evaluation table. It is an example of a feature amount frequency table. It is an example of a feature amount similarity table. It is an example of a feature value co-occurrence table. It is an example of a display priority calculation table. It is an example of a feature amount display priority table. It is an example of a parameter setting screen. It is an example of a before-and-after update comparison screen. It is an example of a feature extraction source information table. This is an example of character decoration. It is an example of a mapping target feature amount table. It is an example of a feature evaluation setting information table. It is an example of a feature extraction status display table. FIG. 11 is a flowchart for explaining feature amount evaluation setting information table generation processing; FIG. FIG. 11 is a flowchart for explaining feature extraction status display table generation processing; FIG. It is an example of an information processing device used for realizing a document information extraction system.

Hereinafter, embodiments will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are appropriately omitted or simplified for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following description, the same or similar configurations may be denoted by the same reference numerals, and redundant description may be omitted. Also, in the following description, the letter "S" attached before the reference sign means a processing step. Also, in the following description, various types of information may be described using expressions such as “table” and “information”, but various types of information may be expressed in data structures other than these.

Hereinafter, as one embodiment of the present invention, an information processing system (hereinafter referred to as "document information extraction system 1”) will be described. The document information extraction system 1, for example, extracts useful information for drafting guidelines for research and development of new materials from a huge set of documents (patent documents, various papers, business reports, corporate reports, news, etc.), such as It is used for data-driven material development using so-called MI (Materials Informatics), which extracts correlations between material structures, properties, and functions, and provides the extracted results to users.

FIG. 1 is a system flow diagram for explaining the main configuration of the document information extraction system 1. As shown in FIG. FIG. 2 is a block diagram for explaining the main configuration of the document information extraction system 1. As shown in FIG. As shown in FIG. 1, the document information extraction system 1 has functions of a document information extraction section 170 and a parameter setting section 180 . Among them, the document information extraction unit 170 extracts information from the accumulated document 111 using a machine learning model (learning model) and presents it to the user. In addition, the parameter setting unit 180 sets the parameters (feature amounts, weights of the feature amounts) of the model in an interactive manner with the user via the user interface.

As shown in FIG. 1 or 2, the document information extraction unit 170 includes a word/related word extraction unit 171 and a document information presentation unit 172. FIG. While referring to the initial dictionary 112 and the word/related word dictionary 113, the word/related word extraction unit 171 extracts words, word-related words, word synonyms, etc. (hereinafter referred to as “word/related word 114 ) is extracted from the document 111 . As a mechanism of the machine learning, for example, there is a named entity extraction technology realized by using RNN (Recurrent Neural Network) and LSTM (Long short-term memory). However, the type of machine learning is not necessarily limited.

The model is represented by, for example, a matrix or vector containing information on the feature amount and the weight of each feature amount. The features of the model for extracting information from the document 111 include, for example, the features of peripheral words (left and right words, parts of speech of left and right words, dependencies), the features of the words themselves (prefix, suffix, , number of characters, uppercase and lowercase letters, parts of speech, types of characters (numbers, kanji, etc.), etc.

The document 111 is managed by the document management section 160 shown in FIG. The documents 111 include a large number of documents (for example, documents acquired via the Internet by web scraping or web crawling, or documents registered by users). In the present embodiment, one or more sentences, one or more sentences, one or more clauses, one or more clauses or phrases, two or more words, etc., which are described on a predetermined topic, are grouped together. Objects are called documents. In the following, a case where various types of information processing performed by the document information extraction system 1 are performed on a document-by-document basis will be described as an example, but the processing unit is not necessarily limited. In the following, the document 111 is assumed to be managed as data in text format (text data), but the method of managing the document 111 is not necessarily limited. Documents to be subjected to various types of information processing described below are decomposed into morphemes by, for example, a morphological analysis technique, and each morpheme is converted into distributed representation (word embedding), for example, and handled.

The initial dictionary 112 and the word/related word dictionary 113 referenced by the document information extraction unit 170 are managed by the dictionary management unit 161 shown in FIG. Among these, the initial dictionary 112 is set with, for example, words indicating the directionality of the extraction target by the machine learning. For example, when extracting information about temperature from the document 111 about thermochemistry, words such as "30 degrees" and "10 degrees" are set in the initial dictionary 112, for example. In this case, the document information extraction unit 170 extracts, for example, “30° C.” and “10° C.” as related terms. The word/related word dictionary 113 manages information in which words are associated with related words and synonyms. For example, the contents of the initial dictionary 112 and the word/related word dictionary 113 may be set by the user, or may be set using a corpus or dictionary obtained from the Internet or the like. For example, the user prepares the initial dictionary 112 and the word/related word dictionary 113 whose contents correspond to the technical field from which useful information is to be obtained and the type of information to be extracted.

The document information presenting unit 172 extracts the word/related word 114 extracted by the word/related word extracting unit 171 and information generated based on the extracted word/related word 114 (hereinafter referred to as "document extraction information 115"). ) to the user. The user uses the document extraction information 115 provided by the document information presentation unit 172 via, for example, a user interface provided in the document information extraction system 1 or another information processing device communicably connected to the document information extraction system 1. do.

The parameter setting unit 180 has functions of a feature amount information generation unit 181 , a feature amount display priority setting unit 182 , a parameter setting screen generation unit 183 , a parameter setting reception unit 184 and a parameter update unit 185 .

Of these, the feature amount information generation unit 181 acquires the current model parameters (feature amounts and weights of each feature amount) (hereinafter referred to as "current parameters 155") from the word/related word extraction unit 171, and acquires Feature quantity information 120 (feature quantity evaluation table 121, feature quantity frequency table 122, feature quantity similarity table 123, feature quantity An origination table 124 and a feature extraction source information table 125) are generated. Details of the feature amount information 120 will be described later.

The feature amount display priority setting unit 182 sets the feature amount display priority (hereinafter referred to as The display priority calculation table 131 used for calculating the “feature amount display priority” is generated. The feature amount display priority setting unit 182 obtains the feature amount display priority based on the display priority calculation table 131 and stores the obtained feature amount display priority in the feature amount display priority table 132 .

The parameter setting screen generating unit 183 displays the feature amounts in the order according to the feature amount display priority, and generates a screen (parameter setting screen 1000 shown in FIG. 10) for accepting information from the user regarding the setting of the parameters of the displayed feature amounts. Generate. When generating the parameter setting screen 1000, the parameter setting screen generation unit 183 generates the parameter setting screen information 140 (character decoration definition table 141, mapping target feature amount table 142, feature amount evaluation setting information table 143, and feature amount extraction status display table). 144) is referenced or generated. Details of the parameter setting screen information 140 will be described later.

The parameter setting reception unit 184 displays the parameter setting screen 1000 and receives information regarding parameter settings from the user.

The parameter update unit 185 updates the feature amount evaluation table 121 based on the information received from the user by the parameter setting reception unit 184 (the storage unit 110 stores the contents of both the feature amount evaluation table 121 before and after the update). . In addition, the parameter update unit 185 updates the difference information (hereinafter referred to as “parameter update information 152
”. ), and based on the generated parameter update information 152, the parameters of the model used by the word/related word extraction unit 171 are updated.

Next, a method of calculating the feature amount display priority performed by the feature amount display priority setting unit 182 will be described. The feature amount display priority setting unit 182 gives priority to the parameter setting screen 1000 for the feature amount that the user wants to confirm with priority (the feature amount that is considered to have a large influence on the accuracy of extracting useful information from the document 111). Set the feature amount display priority so that it is displayed as a target. Specifically, the feature quantity display priority setting unit 182 sets a high feature quantity display priority for a feature quantity corresponding to any of the following (1) to (3). In the following description, a feature amount that acts to extract information to be extracted from the document 111 is referred to as a "positive feature amount", and a feature amount that acts not to extract useful information from the document 111. This is called a “negative feature quantity”, and a feature quantity that does not correspond to either a “positive feature quantity” or a “negative feature quantity” is called a “neutral feature quantity”, respectively.
(1) A positive feature amount, but in the current parameters 155, the feature amount weight is set to be near zero or negative (hereinafter referred to as a "misevaluation correctness feature amount").
(2) Negative feature amount, but in the current parameters 155, the feature amount is set to have a large weight (near zero or plus) (hereinafter referred to as "erroneous evaluation negative feature amount").
(3) Neutral feature amount, but in the current parameters 155, the feature amount for which the absolute value of the feature amount is set large (hereinafter referred to as "false evaluation neutral feature amount").

For example, the feature amount display priority setting unit 182 sets an index (hereinafter referred to as "likelihood of correctness") indicating the possibility of corresponding to the false evaluation correctness feature amount of (1) to the feature amount of the evaluation target ( hereinafter referred to as the "relevant feature") and other positive features used in the current model (hereinafter referred to as the "comparative positive feature"), and comparison with the feature Evaluation is performed based on the degree of co-occurrence with the positive feature amount, the weight of the feature amount in the current parameters 155, and the appearance frequency of the feature amount in the document 111. FIG.

Further, for example, the feature amount display priority setting unit 182 sets the index indicating the possibility of falling under (2) above (hereinafter referred to as “negative likelihood”) to the feature amount of the evaluation target (hereinafter referred to as “corresponding feature”) and other negative features used in the current model (hereinafter referred to as “comparative negative feature”), the feature and the comparative negative feature , the weight of the feature quantity in the current parameters 155, and the appearance frequency of the feature quantity in the document 111.

Further, for example, the feature amount display priority setting unit 182 sets the index indicating the possibility of corresponding to the above (3) (hereinafter referred to as “neutrality”) to the feature amount of the evaluation target (hereinafter referred to as “relevant feature") and other neutral features used in the current model (hereinafter referred to as "comparative neutral feature"), the feature and the comparative neutral feature , the absolute value of the weight of the feature quantity in the current parameters 155, and the appearance frequency of the feature quantity in the document 111.

In any of the above cases, the appearance frequency in the document 111 of the feature quantity to be evaluated is used for evaluation because the feature quantity with a low appearance frequency in the document 111 does not affect the extraction accuracy of information from the document 111. This is because it is considered that the effect of the sine wave is small.

FIG. 3 is a diagram showing an example of the concept of evaluation of applicability to (1) above. In the figure, a case of calculating the "likelihood of correctness" of the feature quantity for extracting the word representing "temperature" from the illustrated document 111 is illustrated. As shown in the figure, the initial dictionary 112 is set with "30 degrees", "10 degrees", etc. for learning words indicating temperature in the model. : degree" (a word ending with "degree") and "left word: temperature wa" (a word having a character string of "temperature" on the left side).

In the same figure, "final word: times" (words ending with "times"), "final words: ℃" (words ending with "℃"), and "left word: declining" (words ending with " Words in which the character string of "lowered" is present) are all feature quantities to be evaluated for applicability to the above (1). In this example, it is assumed that each feature amount to be evaluated has a sufficiently high appearance frequency in the document 111 .

In the case of this example, the evaluation target feature "final word: times" has similarity only with the comparative positive feature "final word: degree", and the evaluation target "left word: declining" is similar. The feature quantity has co-occurrence only with the comparative positive feature quantity “left word: temperature”. In addition, the feature quantity “final word: °C” to be evaluated has similarity to the comparative positive feature quantity “final word: degree”, and the comparative positive feature quantity “left word: temperature is ” and co-occurrence. Therefore, in this example, the feature quantity "final word: °C" is more likely to correspond to the above (1) than the other two feature quantities ("final word: times", "left word: lower"). Ethnicity, that is, “authenticity” is highly evaluated.

The feature amount display priority setting unit 182 selects the feature amount used when presenting to the user based on the "positiveness", "negativeness", and "neutrality" evaluated for each feature amount as described above. Get display priority. For example, the feature amount display priority setting unit 182 sets a high feature amount display priority to a feature amount having a high “likelihood of authenticity” (exceeding a preset threshold value). Further, for example, the feature amount display priority setting unit 182 sets a high feature amount display priority to a feature amount having a high “negativeness” (exceeding a preset threshold value). Further, for example, the feature amount display priority setting unit 182 sets a high feature amount display priority to a feature amount having a high “neutrality” (exceeding a preset threshold value). Note that the feature amount display priority setting unit 182, for example, comprehensively evaluates any two or more of "positiveness", "negativeness", and "neutrality" to determine the feature amount of each feature amount. You may make it set a display priority.

Next, a method for calculating the feature amount display priority will be described with reference to specific tables. When calculating the feature amount display priority, the feature amount display priority setting unit 182 sets the feature amount information 120 (the feature amount evaluation table 121, the feature amount frequency table 122, the feature amount similarity table 123, the feature amount co-occurrence table 124). ) is referenced or generated (including the meaning of update).

FIG. 4 shows an example of the feature evaluation table 121. As shown in FIG. The feature evaluation table 121 manages information regarding the weight and evaluation of each feature in the current model or after being changed by the user. As shown in the figure, the feature quantity evaluation table 121 is composed of a plurality of entries (records) having respective items of feature quantity 1211 , weight 1212 and evaluation 1213 . One entry in the feature evaluation table 121 corresponds to one feature.

Among the above items, the feature amount 1211 stores the feature amount. The weight 1212 stores the weight of the feature quantity. In the evaluation 1213, information indicating the type (hereinafter referred to as "action type") corresponding to the action characteristic of the feature amount ("Positive" for positive feature amount, "Positive" for negative feature amount) "negative", and "neutral" if the feature is neutral) is set. Note that “-” is set in the evaluation 1213 for feature amounts for which the action type is not set.

FIG. 5 shows an example of the feature quantity frequency table 122. As shown in FIG. The feature quantity frequency table 122 manages the appearance frequency of each feature quantity used in the current model in the document 111 (document group). As shown in the figure, the feature amount frequency table 122 is composed of a plurality of entries (records) each having items of feature amount 1221 and frequency 1222 . One entry in the feature quantity frequency table 122 corresponds to one feature quantity. Among the above items, the feature amount 1221 stores the feature amount. The frequency 1222 stores the appearance frequency of the feature quantity in the document 111 .

FIG. 6 shows an example of the feature amount similarity table 123. As shown in FIG. The feature amount similarity table 123 manages the similarities between different feature amounts forming the model. As shown in the figure, the feature amount similarity table 123 is composed of a plurality of entries (records) having items of a first feature amount 1231, a second feature amount 1232, and a degree of similarity 1233, respectively. One entry in the feature amount similarity table 123 corresponds to one set of feature amounts.

Among the above items, the first feature amount 1231 stores one feature amount of the above set (hereinafter referred to as "first feature amount"). The second feature amount 1232 stores the other feature amount of the set (hereinafter referred to as "second feature amount"). The degree of similarity 1233 stores the degree of similarity between the first feature amount and the second feature amount. Note that the feature amount information generation unit 181 obtains similarity based on cosine similarity or Levenshtein distance, for example.

FIG. 7 shows an example of the feature value co-occurrence table 124. As shown in FIG. The feature amount co-occurrence degree table 124 manages the co-occurrence degrees of different feature amounts forming the model. As shown in the figure, the feature amount co-occurrence table 124 is composed of a plurality of entries (records) each having a first feature amount 1241 , a second feature amount 1242 , and a co-occurrence degree 1243 . One entry in the feature amount co-occurrence table 124 corresponds to one set of feature amounts.

Among the above items, the first feature amount 1241 stores one feature amount of the above set (hereinafter referred to as "first feature amount"). The second feature amount 1242 stores the other feature amount of the set (hereinafter referred to as "second feature amount"). The degree of co-occurrence 1243 stores the degree of co-occurrence between the first feature amount and the second feature amount. Note that the feature amount information generation unit 181, for example, generates a word distributed representation (for example, a distributed representation generated by word2vec, BERT, etc.) for a word, and uses the generated distributed representation of each word to generate a shared representation of two words. ask for a wake up call.

FIG. 8 shows an example of the display priority calculation table 131. As shown in FIG. The display priority calculation table 131 manages information based on the contents of the feature quantity evaluation table 121 , the feature quantity frequency table 122 , the feature quantity similarity table 123 , and the feature quantity co-occurrence degree table 124 . As shown in the figure, the display priority calculation table 131 includes a first feature amount 1311, a second feature amount 1312, a degree of similarity 1313, a degree of co-occurrence 1314, a first feature amount weight 1315, a second feature amount weight 1316, and a plurality of entries (records) having each item of the second feature quantity frequency 1317 . One entry in the display priority calculation table 131 corresponds to one set of feature amounts.

Among the above items, the first feature amount 1311 stores one feature amount of the above set (hereinafter referred to as "first feature amount"). The second feature amount 1312 stores the other feature amount of the set (hereinafter referred to as "second feature amount"). The degree of similarity 1313 stores the degree of similarity between the first feature amount and the second feature amount. The degree of co-occurrence 1314 stores the degree of co-occurrence between the first feature amount and the second feature amount. The first feature weight 1315 stores the weight of the first feature. The weight of the second feature amount is stored in the second feature amount weight 1316 . The appearance frequency of the second feature amount in the document 111 is stored in the second feature amount frequency 1317 .

FIG. 9 shows an example of the feature amount display priority table 132. As shown in FIG. The feature amount display priority table 132 manages information used for setting the feature amount display priority of each feature amount, which is generated by the feature amount display priority setting unit 182 based on the display priority calculation table 131 . As shown in the figure, the feature quantity display priority table 132 is composed of a plurality of entries (records) having respective items of a feature quantity 1321, positiveness 1322, negativeness 1323, and neutrality 1324. . One entry in the feature quantity display priority table 132 corresponds to one feature quantity.

Among the above items, the feature amount 1321 stores the feature amount.

The legitimacy 1322 stores a value (legitimacy) obtained from the following equation based on the display priority calculation table 131 . In the following equation, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. a, b, c, and d in the following equation are constants.

The negative likelihood 1322 stores a value (negative likelihood) obtained from the following equation based on the display priority calculation table 131 . In the following equation, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. a, b, c, and d in the following equation are constants.

The neutrality 1323 stores a value (neutrality) obtained from the following equation based on the display priority calculation table 131 . In the following equation, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. a, b, c, and d in the following equation are constants.

As described above, the parameter setting screen generation unit 183 displays feature values based on the positiveness 1322, the negativeness 1323, and the neutrality 1324 obtained as described above when setting the parameters through interaction with the user. Ask for priority.

As described above, the document information extraction system 1 obtains at least one of positiveness, negativity, and neutrality for each feature amount, and sets the feature amount display priority based on the obtained value. Therefore, it is possible to preferentially present to the user highly effective feature amounts (feature amounts with high possibility of extracting more appropriate words and related words from the document 111) that can be obtained by adjustment, and the user can efficiently Parameters can be adjusted (tuned).

Next, parameter setting performed by the document information extraction system 1 through interactive processing with the user will be described. For simplicity of explanation, the case of extracting words from the document 111 will be described below as an example.

FIG. 10 is an example of a screen (hereinafter referred to as "parameter setting screen 1000") displayed by the parameter setting screen generation unit 183 when parameters are set through interactive processing with the user. As shown in the figure, the parameter setting screen 1000 has a feature amount evaluation setting field 1010, a feature amount extraction status display field 1020, a before/after update comparison button 1030, and a confirmation button 1040. FIG.

In the feature amount evaluation setting field 1010, the feature amounts for prompting the user to confirm are displayed in descending order of feature amount display priority in the feature amount display priority table 132. FIG. As shown in the figure, the feature quantity evaluation setting field 1010 is composed of a plurality of entries having respective items of a feature quantity 1011, a weight 1012, a sentence 1013, and an evaluation 1014. FIG. One entry in the feature quantity evaluation setting field 1010 corresponds to one feature quantity.

Among the above items, in the feature amount 1011, a feature amount prompting the user to confirm is displayed. The weight 1012 displays the weight set for the feature amount. The user can evaluate the feature amount while confirming the weight set for the feature amount. The value of the weight 1012 can also be manually changed by the user using the screen.

The sentence 1013 displays one or more sentences (sentences that are elements of the document 111) from which words have been extracted based on the feature amount. A portion corresponding to the feature amount of the text 1013 is highlighted (underlined and italicized in this example), so that the user can easily grasp the position of the feature amount in the text.

The evaluation 1014 displays the action type set for the feature amount. As for the feature amount for which the action type has been set in the past, the feature amount already set in the evaluation 1014 is automatically displayed as the default value. For feature amounts for which action types have not been set in the past, for example, the action types determined by the parameter setting unit 180 based on the content of the feature amount display priority table 132 are displayed in the evaluation 1014 as default values. For example, a feature amount whose positivity 1322 exceeds a preset threshold is assigned “positivity,” a feature amount whose negativeness 1323 exceeds a preset threshold is assigned “negative,” and neutral. "Neutral" is displayed in the evaluation 1014 as a default value for the feature amount whose likelihood 1324 exceeds a preset threshold. Note that the method for automatically determining the default value is not necessarily limited. The content of the evaluation 1014 can also be manually changed by the user using the screen. The user changes the contents of the evaluation 1014 by, for example, operating the type of action (“Positive”, “Negative”, and “Neutral”) displayed as a pull-down menu.

The feature quantity extraction status display field 1020 displays information indicating the extraction status (mode) of each feature quantity displayed in the feature quantity evaluation setting field 1010 . As shown in the figure, the feature quantity extraction status display field 1020 is composed of a plurality of entries having respective items of a document ID 1021 , an extracted word 1022 , an extraction source sentence 1023 and an unmapping feature quantity 1024 . One entry in the feature quantity extraction status display field 1020 corresponds to one word extracted from the document 111 .

Among the above items, the document ID 1021 displays the document ID that is the identifier of the extraction source document (the file name of the extraction source document in this example). A word extracted by the feature amount 1011 displayed in the feature amount evaluation setting column 1010 is displayed in the extracted word 1022 .

An extraction source sentence 1023 displays the sentence from which the word is extracted. As shown in the figure, the portion corresponding to the feature quantity of the extraction source sentence 1023 is decorated with characters (solid underline, framed, italic, dashed underline, no decoration) according to the weight of the feature quantity. It is It should be noted that the feature amount to which character decoration is added in the extraction source sentence 1023 is the feature amount defined in the mapping target feature amount table 142, which will be described later. In this example, the difference in the weight (effectiveness) set for each feature amount is distinguished by changing the type of character decoration, but other methods such as color coding may be used for distinction.

In this example, among the character decorations, the solid underline is added to the feature quantity (positive feature quantity) that greatly affects the extraction of the word indicating temperature, and the framed character is used to extract the word indicating temperature. Italic characters are attached to features (negative features) that act not to extract words other than temperature, and dashed underlines is added to the feature quantity (negative feature quantity) that has a large effect on not extracting words that are not temperature. gender feature).

The non-mapping feature amount 1024 displays another feature amount (the number of characters in this example) that contributed to the extraction of the word. Examples of other feature values displayed in the non-mapping feature value 1024 include, for example, "number or letter", "English or Japanese", "romaji or kanji", and "capital letter or small letter".

When the user operates the before/after update comparison button 1030, the parameter setting unit 180 compares the contents of the feature amount evaluation setting field 1010 before the user performs the change operation and the feature amount evaluation setting field 1010 after the user performs the change operation. (hereinafter referred to as "before and after update comparison screen 1100") is displayed.

FIG. 11 shows an example of a before and after update comparison screen 1100 . As shown in the figure, the example before and after update comparison screen 1100 includes a feature amount evaluation setting field (before update) 1111 that displays the content of the feature amount evaluation setting field 1010 before updating, and a feature amount evaluation setting before updating. It also has a feature value evaluation setting column (after update) 1112 in which the contents of the column 1010 are displayed. By referring to the before and after update comparison screen 1100, the user can easily confirm how the evaluation and weighting of which feature amount has been changed by the user. When the user changes the content of the feature amount evaluation 1014, the feature amount display priority setting unit 182 recalculates the feature amount display priority based on the updated evaluation (action type) and weight, and displays the feature amount. The order is also updated.

Returning to FIG. 10, when the user operates the confirm button 1040, the parameter update unit 185 generates the parameter update information 152 based on the setting contents (weight 1012, evaluation 1014) of the parameter setting screen 1000, and the generated parameter update information 152, the parameters of the model used by the word/related word extraction unit 171 are updated. Note that the parameter updating unit 185 may automatically set the weight of the corresponding feature amount based on the evaluation 1014 on the parameter setting screen 1000 . In that case, the weight automatically set for each feature amount is set based on, for example, the positivity 1322 , negativeness 1323 , and neutrality 1324 in the feature amount display priority table 132 .

As described above, since the feature amount 1011 and the text 1013 from which the feature amount is extracted are displayed in the feature amount evaluation setting field 1010 of the parameter setting screen 1000, the user can set an appropriate evaluation 1014 for the feature amount. can do. In addition, in the feature quantity extraction status display field 1020, information (extracted word 1022, extraction source sentence 1023, non-mapping feature quantity 1024) regarding the extraction status of the feature quantity displayed in the feature quantity evaluation setting field 1010 is displayed. Therefore, the user can visually confirm how the feature value works in word extraction, and even users with little knowledge or experience of machine learning can refer to this information. feature quantity can be set efficiently and appropriately. Therefore, for example, it is possible to effectively utilize the abundant knowledge and experience of domain users who are not familiar with machine learning but who are familiar with the target information field.

Next, examples of various tables used to generate the parameter setting screen 1000 and the before/after update comparison screen 1100 are shown.

FIG. 12 is an example of the feature quantity extraction source information table 125 . The feature quantity extraction source information table 125 manages information about the feature quantity extraction sources. As shown in the figure, the feature amount extraction source information table 125 includes a document ID 1251, a word 1252, an extraction source sentence 1253, a word start point 1254, a word end point 1255, a feature amount 1256, a feature amount start point 1257, and a feature amount end point 1258. It consists of multiple entries (records) with each item. One entry in the feature quantity extraction source information table 125 corresponds to one feature quantity.

Among the above items, the document ID 1251 stores the document ID of the document from which the feature amount is extracted. The word 1252 stores the word extracted by the feature amount. The extraction source sentence 1253 stores the sentence from which the word is extracted. The word start point 1254 and the word end point 1255 store the position (start point, end point) of the word in the document, respectively. The feature amount 1256 stores the feature amount. A feature quantity start point 1257 and a feature quantity end point 1258 respectively store positions (start point, end point) at which the feature quantity appears in the extraction source sentence.

FIG. 13 is an example of the character decoration definition table 141. As shown in FIG. The character decoration definition table 141 manages information indicating character decoration corresponding to the weight of the feature quantity. The contents of the character decoration definition table 141 are set by, for example, a system administrator or a user. As shown in the figure, the character decoration definition table 141 is composed of a plurality of records having weight 1411 and character decoration 1412 items. Information indicating the weight range is stored in the weight 1411 . Information indicating the type of character decoration is stored in the character decoration 1412 .

FIG. 14 is an example of the mapping target feature quantity table 142 . In the mapping target feature amount table 142, among the feature amounts displayed in the extraction source sentence 1023 and the non-mapping feature amount 1024 in the feature amount extraction status display field 1020 of the parameter setting screen 1000, the feature amounts to be subjected to character decoration are displayed. (target feature quantity 1421) is managed. The contents of the mapping target feature amount table 142 are set by, for example, a system administrator or a user.

FIG. 15 is an example of the feature evaluation setting information table 143. As shown in FIG. Information to be displayed in the feature amount evaluation setting column 1010 of the parameter setting screen 1000 is managed in the feature amount evaluation setting information table 143 . As shown in the figure, the feature evaluation setting information table 143 includes a plurality of entries ( records). One entry in the feature quantity evaluation setting information table 143 corresponds to one feature quantity.

Among the above items, the feature amount 1431 stores the feature amount to be evaluated. The weight 1432 stores the weight of the feature amount. The extraction source text 1433 stores the text from which the feature amount is extracted. Information (start point, end point) indicating the position of the feature amount in the extraction source sentence 1433 is stored in the feature amount start point 1434 and the feature amount end point 1435, respectively. In the evaluation 1436, the default value described above or the evaluation set by the user (one of “positive”, “negative”, and “neutral”) is set.

FIG. 16 is an example of the feature quantity extraction status display table 144. As shown in FIG. Information displayed in the feature extraction status display column 1020 of the parameter setting screen 1000 is managed in the feature extraction status display table 144 . As shown in the figure, the feature quantity extraction status display table 144 includes document I
D1441, word 1442, extraction source sentence 1443, word start point 1444, word end point 1445, feature amount 1446, feature amount start point 1447, feature amount end point 1448, weight 1449, character decoration 1450, and non-mapping feature amount 1451. Consists of multiple records. One entry in the feature quantity extraction status display table 144 corresponds to one word extracted from the document 111 .

The document ID 1441 stores the document ID of the document 111 from which the word (word 1442) is extracted. The word 1442 stores a word extracted by the feature amount (feature amount 1446). The extraction source sentence 1443 stores the sentence from which the word is extracted. The word start point 1444 and word end point 1445 store the position (start point, end point) of the word in the extraction source sentence 1443 . The feature amount 1446 stores the feature amount used to extract the word. Information (start point, end point) indicating the position of the feature amount in the extraction source sentence 1443 is stored in the feature amount start point 1447 and the feature amount end point 1448, respectively.

The weight 1449 stores the weight set for the feature amount (feature amount 1446). The character decoration 1450 stores information indicating the character decoration to be applied when displaying the feature amount. The non-mapping feature amount 1451 stores other feature amounts that have contributed to the extraction of the word and are not defined in the mapping target feature amount table 142 .

Next, processing performed by the parameter setting unit 180 when generating the feature amount evaluation setting information table 143 and the feature amount extraction status display table 144 will be described.

FIG. 17 is a flowchart for explaining the process performed by the parameter setting unit 180 when generating the feature evaluation setting information table 143 (hereinafter referred to as "feature evaluation setting information table generation process S1700"). The feature evaluation setting information table generation processing S1700 will be described below with reference to FIG.

First, the parameter setting unit 180 reads the feature quantity extraction source information table 125 and the feature quantity evaluation table 121 (S1711 to S1712).

Subsequently, the parameter setting unit 180 selects one entry in the feature evaluation table 121 (S1713).

Next, the parameter setting unit 180 adds to the feature evaluation setting information table an entry in which the content associated with the extraction source sentence 1253 of the feature extraction source information table 125 corresponding to the feature of the entry selected in S1713 is set. (S1714).

Subsequently, the parameter setting unit 180 determines whether or not all entries in the feature evaluation table S1713 have been selected in S1713 (S1715). If all entries have not been selected (S1715: NO), the process returns to S1713, selects an unselected entry, and performs the same process as above. On the other hand, if all entries have been selected (S1715: YES), the feature evaluation setting information table generation processing S1700 ends.

FIG. 18 is a flowchart for explaining the process performed by the parameter setting unit 180 when generating the feature extraction status display table 144 (hereinafter referred to as "feature extraction status display table generation process S1800"). The feature quantity extraction status display table generation processing S1800 will be described below with reference to FIG.

First, the parameter setting unit 180 sets the feature amount extraction source information table 125, the feature amount evaluation table 121, the character decoration definition table 141, and the mapping target feature amount table 142.
is read (S1811 to S1814).

Subsequently, the parameter setting unit 180 selects one entry in the feature quantity extraction source information table 125 and adds one entry to the feature quantity extraction status display table 144 using the content of the selected entry (S1815).

Subsequently, the parameter setting unit 180 stores the corresponding feature amount weight in the feature amount evaluation table 121 in the added entry weight 1449 (S1816).

Subsequently, the parameter setting unit 180 stores the character decoration corresponding to the weight stored in S1816 in the character decoration definition table 141 in the character decoration 1450 of the added entry (S1817).

Subsequently, the parameter setting unit 180 determines whether or not the mapping target feature amount table 142 has the same feature amount as the feature amount 1446 of the entry added in S1815 (S1818). If there is the same feature amount (S1818: YES), the parameter setting unit 180 sets the non-mapping feature amount 1451 of the added entry to blank, and then the process proceeds to S1820. If the same feature quantity does not exist (S1818: NO), the parameter setting unit 180 stores the feature quantity in the non-mapping feature quantity 1451 of the added entry (S1819), and then the process proceeds to S1820.

Subsequently, the parameter setting unit 180 determines whether or not all entries in the feature quantity extraction source information table 125 have been selected in S1815 (S1820). If all entries have not been selected (S1820: NO), the process returns to S1815, selects an unselected entry, and performs the same process as above. On the other hand, if all entries have been selected (S1815: YES), the feature extraction status display table generation processing S1800 ends.

Next, an information processing apparatus (computer) used for the configuration of the document information extraction system 1 described above will be described.

FIG. 19 shows an example of an information processing device used for configuring the document information extraction system 1 . As shown in the figure, the illustrated information processing apparatus 10 includes a processor 11 , a main storage device 12 , an auxiliary storage device 13 , an input device 14 , an output device 15 and a communication device 16 .

The document information extraction system 1 may be implemented using, for example, a plurality of information processing apparatuses 10 that are communicatively connected. Further, the information processing apparatus 10 is a virtual information processing resource provided in whole or in part using virtualization technology, process space separation technology, etc., such as a virtual server provided by a cloud system. may be implemented using Also, all or part of the functions provided by the information processing apparatus 10 may be realized by services provided by the cloud system via an API (Application Programming Interface) or the like, for example.

Further, all or part of the functions provided by the information processing device 10 are realized using, for example, SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), etc. can be anything.

The processor 11 shown in the figure is, for example, a CPU (Central Processing Unit), an MPU
(Micro Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit),
It is configured using an AI (Artificial Intelligence) chip or the like.

The main storage device 12 is a device that stores programs and data, and is, for example, a ROM (Read
Only Memory), RAM (Random Access Memory), nonvolatile memory (NVRAM (Non Volatile RAM)), and the like.

The auxiliary storage device 13 is, for example, an SSD (Solid State Drive), a hard disk drive, an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage system, an IC card, an SD card, or an optical recording device. They are a read/write device for a recording medium such as a medium, a storage area of a cloud server, and the like. Programs and data can be read into the auxiliary storage device 13 via a recording medium reading device or the communication device 16 . Programs and data stored (stored) in the auxiliary storage device 13 are read into the main storage device 12 at any time.

The input device 14 is an interface that receives input from the outside, and includes, for example, a keyboard, mouse, touch panel, card reader, pen-input tablet, voice input device, and the like.

The output device 15 is an interface for outputting various information such as processing progress and processing results. The output device 15 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the above various information, a device (audio output device (speaker, etc.)) that converts the above various information into sound. , a device (printing device, etc.) that converts the above various information into characters. For example, the information processing device 10 may be configured to input and output information with another device via the communication device 16 .

The input device 14 and the output device 15 constitute a user interface that realizes interactive processing (acceptance of information, presentation of information, etc.) with the user.

The communication device 16 is a device that realizes communication with other devices. The communication device 16 implements communication with other devices via a communication network 5 (LAN (Local Area Network), WAN (Wide Area Network), Internet, public communication network, dedicated line, etc.). Alternatively, it is a wireless communication interface, such as a NIC (Network Interface Card), a wireless communication module, a USB module, or the like.

For example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), etc. may be installed in the information processing apparatus 10 . The document information extraction system 1 stores the various types of information (data) described above, for example, as files managed by a database table or a file system.

Various functions provided by the document information extraction system 1 are performed by the processor 11 reading and executing a program stored in the main storage device 12, or by hardware (FPGA, ASIC, AI chip, etc.) itself. Various functions of the document information extraction system 1 include, for example, various known data mining techniques such as text data mining, various known natural language processing techniques (morphological analysis, syntactic analysis, semantic analysis, contextual analysis, feature quantity extraction, , word distributed representation, named entity extraction, text classification, sequence labeling), and various known machine learning techniques (deep learning (DNN (Deep Neural Network), RNN (Recurrent Neural Network), etc.).

As described in detail above, the document information extraction system 1 of the present embodiment, when setting the parameters (feature amounts, weights), sets the evaluation target feature amount and one or more sentences from which the evaluation target feature amount is extracted. , while displaying the weight set for the feature to be evaluated, the setting of the action type or weight is accepted, so the user can confirm the text and weight of the extraction source (what kind of feature to what kind of feature? It is possible to efficiently and appropriately set the evaluation target feature amount while checking whether the setting has been made.

In addition, since the part of the evaluation target feature quantity is highlighted in the text 1013 of the feature quantity evaluation setting column 1010 of the parameter setting screen 1000, the user can easily confirm how the evaluation target feature quantity affects the document. can do.

In addition, in the extraction source sentence 1023 of the feature quantity extraction status display field 1020 of the parameter setting screen 1000, the evaluation target feature quantity portion of the displayed sentence is highlighted by character decoration, color, etc., distinguished for each action type. Therefore, the user can efficiently and appropriately set the evaluation target feature quantity while recognizing the action characteristics of the evaluation target feature quantity for each action type.

In addition, since the words extracted from the document are displayed in the feature quantity extraction status display field 1020 of the parameter setting screen 1000, the user can see how the feature quantity works and what kind of words are extracted. It is possible to efficiently and appropriately set the evaluation target feature amount while recognizing the problem.

In addition, in the feature quantity extraction status display field 1020 of the parameter setting screen 1000, other feature quantity (non-mapping feature quantity 1024) that affected the extraction of information extracted from the document by the evaluation target feature quantity is also displayed. can efficiently and appropriately set the evaluation target feature amount while also referring to other feature amounts.

As described above, according to the document information extraction system 1 of the present embodiment, it is possible to support the setting of the feature amount of the model used when extracting useful information from the document by the machine learning method. Therefore, for example, even a person who has knowledge about the field of information to be extracted (domain knowledge) but does not have knowledge or experience of machine learning can efficiently and appropriately perform feature values, and domain user This knowledge can be effectively used to appropriately set the feature amount.

Although one embodiment of the present invention has been described above, it goes without saying that the present invention is not limited to the above-described embodiment, and can be variously modified without departing from the scope of the invention. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Moreover, it is possible to add, delete, or replace a part of the configuration of the above embodiment with another configuration.

For example, in the above description, various processes are performed on a document basis, but the processing unit is not necessarily limited. The same processing can be performed using sentences consisting of the above sentences as a unit.

In the above description, the document information extraction unit 170 extracts information from the accumulated documents 111 by using a machine learning model (learning model), and presents the extracted information to the system administrator and the user. For example, information may be extracted using an extraction rule defined by a system administrator or user and presented to the system administrator or user. In that case, for example, in the parameter setting screen 1000 shown in FIG. 10, the non-mapping feature amount 1024 may display information indicating whether or not the rule is matched.

Further, each of the above-described components, functional units, processing units, processing means, etc. may be realized by hardware, for example, by designing them in an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that realize each function is stored in recording devices such as memories, hard disks, SSDs (Solid State Drives),
It can be placed in recording media such as IC cards, SD cards, and DVDs.

Moreover, the arrangement form of various functional units, various processing units, and various databases of each information processing apparatus described above is merely an example. The arrangement form of various functional units, various processing units, and various databases can be changed to an optimum arrangement form from the viewpoint of the performance, processing efficiency, communication efficiency, etc. of hardware and software provided in these devices.

Also, the configuration of the database (schema, etc.) that stores the various data described above can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

1 document information extraction system, 110 storage unit, 111 document, 112 initial dictionary, 113
word/related word dictionary, 114 word/related word, 115 document extraction information, 120 feature amount information, 121 feature amount evaluation table, 122 feature amount frequency table, 123 feature amount similarity table, 124 feature amount co-occurrence table, 125 feature quantity extraction source information table 131
display priority calculation table 132 feature amount display priority table 140 parameter setting screen information 141 character decoration definition table 142 mapping target feature amount table 143 feature amount evaluation setting information table 144 feature amount extraction status display table 152 parameter update information, 155 current parameter, 160 document management unit, 161 dictionary management unit, 170 document information extraction unit, 171 word/related word extraction unit, 172 document information presentation unit, 180 parameter setting unit, 181 feature amount information generation unit, 182 Feature value display priority setting unit 183 Parameter setting screen generation unit 184 Parameter setting reception unit 185 Parameter update unit 1000 Parameter setting screen 1100 Before/after update comparison screen S1700 Feature value evaluation setting information table generation process S1800 Features Amount extraction status display table generation processing

Claims (15)

configured using an information processing device,
a document management unit that manages a group of documents;
a document information extraction unit that extracts information from the documents of the document group using a machine learning model;
a storage unit that stores a plurality of feature amounts that constitute the parameters of the model, and action types that are information indicating characteristics of actions when the information is extracted for each of the feature amounts;
An evaluation target feature quantity which is a feature quantity to be evaluated among the plurality of feature quantities, one or more sentences from which the evaluation target feature quantity is extracted, and a weight currently set for the evaluation target feature quantity. and a parameter setting reception unit that receives the setting of the action type or the weight of the evaluation target feature quantity while displaying;
a parameter updating unit that updates the parameter based on the received action type or weight;
A document information extraction system. The document information extraction system according to claim 1,
The action type is positive, which indicates that the feature quantity works to extract information that should be extracted from the document group, and that the feature quantity does not extract information that should not be extracted from the document group. including at least one of the negative indicating that the action of the feature amount is neither the positive nor the negative,
Document information extraction system. The document information extraction system according to claim 1,
The parameter setting reception unit highlights a portion of the text to be displayed that is the evaluation target feature amount.
Document information extraction system. The document information extraction system according to claim 3,
The parameter setting reception unit highlights the portion of the evaluation target feature amount in the displayed sentence, distinguishing it by the action type of the evaluation target feature amount.
Document information extraction system. The document information extraction system according to claim 4,
The parameter setting reception unit distinguishes and emphasizes the part of the evaluation target feature quantity in the text to be displayed by attaching a different character decoration or a different color for each action type of the evaluation target feature quantity,
Document information extraction system. The document information extraction system according to claim 1,
The parameter setting reception unit further displays other feature amounts that affect the extraction of the information extracted from the document by the evaluation target feature amount.
Document information extraction system. The document information extraction system according to claim 1,
The parameter setting reception unit further displays information extracted from the document based on the evaluation target feature amount.
Document information extraction system. The document information extraction system according to claim 1,
For each of the evaluation target feature quantities, the degree of similarity between the evaluation target feature quantity in the document group and the other feature quantity having the action type in common, the degree of co-occurrence with the other feature quantity in the document group, feature quantity display, which is a priority for displaying the evaluation target feature quantity based on at least one of a weight currently set for the evaluation target feature quantity and an appearance frequency of the evaluation target feature quantity in the document group; a feature amount display priority setting unit for obtaining priority;
a parameter setting reception unit that receives setting of the action type or the weight of the evaluation target feature quantity while displaying the evaluation target feature quantity according to the feature quantity display priority;
A document information extraction system, further comprising: The document information extraction system according to claim 8,
The action type is positive, which indicates that the feature quantity works to extract information that should be extracted from the document group, and that the feature quantity does not extract information that should not be extracted from the document group. including at least one of the negative indicating that the action of the feature amount is neither the positive nor the negative,
Document information extraction system. The document information extraction system according to claim 9,
The feature amount display priority setting unit is configured to set the evaluation target feature amount classified into the positive action type with other feature amounts having the same action type as the evaluation target feature amount in the document group. The higher the similarity of, the higher the degree of co-occurrence with the other feature amount, and the smaller the currently set weight, the higher the feature amount display priority is set.
Document information extraction system. The document information extraction system according to claim 9,
The feature amount display priority setting unit is configured to set the evaluation target feature amount classified into the negative action type with other feature amounts having the same action type as the evaluation target feature amount in the document group. The higher the similarity of the feature, the higher the co-occurrence with the other feature, and the higher the currently set weight, the higher the feature display priority is set.
Document information extraction system. The document information extraction system according to claim 9,
The feature amount display priority setting unit is configured to set the evaluation target feature amount classified into the neutral action type with other feature amounts having the same action type as the evaluation target feature amount in the document group. The higher the similarity of the feature, the higher the co-occurrence with the other feature, and the higher the currently set weight, the higher the feature display priority is set.
Document information extraction system. The document information extraction system according to claim 1,
the information is a word or a related word of a word,
Document information extraction system. The information processing device
managing a collection of documents;
extracting information from the documents of the set of documents using a machine learning model;
a step of storing a plurality of feature amounts constituting parameters of the model and an effect type, which is information indicating characteristics of an effect when the information is extracted for each of the feature amounts;
An evaluation target feature quantity which is a feature quantity to be evaluated among the plurality of feature quantities, one or more sentences from which the evaluation target feature quantity is extracted, and a weight currently set for the evaluation target feature quantity. and receiving the setting of the action type or the weight of the evaluation target feature quantity while displaying;
updating the parameter based on the received action type or weight;
A document information extraction method that performs The document information extraction method according to claim 14,
The information processing device
For each of the evaluation target feature quantities, the degree of similarity between the evaluation target feature quantity in the document group and the other feature quantity having the action type in common, the degree of co-occurrence with the other feature quantity in the document group, feature quantity display, which is a priority for displaying the evaluation target feature quantity based on at least one of a weight currently set for the evaluation target feature quantity and an appearance frequency of the evaluation target feature quantity in the document group; determining a priority; and
receiving the setting of the action type or the weight of the evaluation target feature quantity while displaying the evaluation target feature quantity according to the feature quantity display priority;
A method for extracting document information, further comprising:

Images