JP2018180866A - Determination method, determination program and determination device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a determination method and the like to determine a similarity index appropriate to the category of a document.SOLUTION: A computer performs processing of: obtaining documents in which mentions in the documents are associated with entities having category information in a knowledge base; generating document groups in which the obtained documents are classified by each of the associated categories; generating, for each of the generated document groups, document pairs including the documents in which the same mentions are associated; giving to the generated document pairs with labels indicating whether or not the entities match; creating similarity indexes on the basis of the document pairs to which the labels are given, and outputting the created similarity indexes by associating thereof with the categories corresponding to the document groups.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a determination method and the like for determining document contents for clustering.

  A technique for obtaining the degree of similarity between two documents has been proposed (Patent Document 1 etc.). The degree of similarity is determined by using a similarity index in which the degree of similarity of the words included in the document is numerically determined.

JP 2000-155762 A

  However, the similarity index of Patent Document 1 etc. can not change the similarity index depending on the category of the document. Therefore, depending on the category of the document, there is a problem that the accuracy of the similarity index decreases.

  One aspect is to provide a determination method or the like for determining a similarity index suitable for a category of a document.

  The determination method disclosed in the present application is a document in which a computer acquires a document in which a mention in the document is associated with an entity having category information in the knowledge base, and the acquired document is classified for each category associated with the document. A group is generated, and for each generated document group, a document pair including the document to which the same mention is associated is generated, and a label indicating whether the entity matches the generated document pair. A similarity index is created based on the document pair that has been assigned and labeled, and the created similarity index is output in association with the category corresponding to the document group.

  According to one aspect of the present application, it is possible to determine the similarity index suitable for the category of the document.

It is a block diagram which shows the structural example of a similarity score calculation apparatus. It is explanatory drawing which shows the example of a record layout of entity information DB. It is explanatory drawing which shows the example of a record layout of data set DB. It is explanatory drawing which shows the example of a record layout of type-category corresponding | compatible DB. It is explanatory drawing which shows the example of a record of similarity index DB. It is a flowchart which shows the procedure of similarity index production | generation processing. It is a flowchart which shows the procedure of similarity score calculation processing. It is a flowchart which shows the procedure of a clustering process. It is explanatory drawing which shows the example of the document grouping using a similarity index. It is a flowchart which shows the other procedure of a clustering process. It is explanatory drawing which shows the production method of type / category corresponding | compatible DB. It is explanatory drawing which shows the other creation method of type-category corresponding DB. It is a block diagram which shows an example of a function structure of a similarity score calculation apparatus.

  Hereinafter, embodiments will be described with reference to the drawings.

Embodiment 1
FIG. 1 is a block diagram showing a configuration example of the similarity score calculation device 1. The similarity score calculation device (determination device) 1 includes a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a large capacity storage unit 14, a communication unit 15, and an input / output unit 16. And a reading unit 17. Each configuration is connected by a bus B.

  The CPU 11 controls each part of the hardware in accordance with a control program (determination program) 1 P stored in the ROM 12. The RAM 13 is, for example, an SRAM (Static RAM), a DRAM (Dynamic RAM), or a flash memory. The RAM 13 temporarily stores data generated when the CPU 11 executes a program.

  The large-capacity storage unit 14 is, for example, a hard disk or a solid state drive (SSD). The large-capacity storage unit 14 stores various data necessary for the determination process of the similarity index and the similarity score calculation process. The large-capacity storage unit 14 stores a document DB (Data Base) 141, an entity information DB 142, a data set DB 143, a type-category correspondence DB 144, and a similarity index DB 145. Alternatively, the control program 1P may be stored in the large capacity storage unit 14.

  The communication unit 15 communicates with other computers via a network. The input / output unit 16 receives an operation signal from a keyboard or a mouse. The input / output unit 16 also outputs a display image to a display device such as a liquid crystal display device.

  The reading unit 17 reads a portable storage medium 1 a including a CD (Compact Disc) -ROM and a DVD (Digital Versatile Disc) -ROM. The CPU 11 may read the control program 1 P from the portable storage medium 1 a via the reading unit 17 and store the control program 1 P in the large-capacity storage unit 14. Further, the CPU 11 may download the control program 1 P from another computer via a network or the like and store the control program 1 P in the large-capacity storage unit 14. Furthermore, the CPU 11 may read the control program 1P from the semiconductor memory 1b.

  Next, the operation of the similarity score calculation device 1 will be described. The similarity score calculation device 1 has two operation modes. The two operation modes are a similarity index creation mode and a similarity score calculation mode. In the similarity index creation mode, the similarity score calculation device 1 acquires category-based similarity indexes. The similarity score calculation device 1 acquires a similarity index by learning. The similarity score calculation device 1 acquires learning data used for learning using a large scale knowledge base. An example of a large-scale knowledge base is Wikipedia (Wikipedia).

  In the similarity score calculation mode, the similarity score calculation device 1 calculates the similarity score of the document pair using the category-based similarity index acquired in the similarity index creation mode.

  Here, definitions of terms used in the following description will be shown. An mention is a string that points to a specific entity that appears in the document. entity is the entity itself. For example, in the sentence, "Today is the signing meeting of the artist Suzuki Ichiro.", Suzuki Ichiro is a mention. In the same sentence, the person named Suzuki Ichiro is the entity. Also, a category is a classification of the subject. The categories are, for example, persons, companies, and municipalities.

  In addition, in acquiring the similarity index, as described above, a large-scale knowledge base is used. The use of a large-scale knowledge base assumes that the following hypothesis holds. (1) The criteria for determining the degree of identity of an entity appearing in a document differ depending on the attribute (category) of the entity. (2) An attribute (category) can be associated with category information of a large scale knowledge base. By establishing (1) and (2), it becomes possible to obtain learning data for obtaining a category-based similarity index using a large-scale knowledge base.

  Subsequently, a database stored in the large capacity storage unit 14 will be described. The document DB 141 stores various document data. The document data is, for example, a document acquired from a large scale knowledge base used in the similarity index creation mode or a document group to be subjected to similarity score calculation in the similarity score calculation mode. Note that the document used in the similarity index creation mode is not limited to being acquired from the knowledge base as long as it is linked to the knowledge base entity (for example, if there is a link).

  FIG. 2 is an explanatory view showing an example of the record layout of the entity information DB 142. As shown in FIG. The entity information DB 142 stores information about an entity (person, company, etc.) included in the document. The document is, for example, one stored in the document DB 141. The information stored in the entity information DB 142 is one of learning corpuses for creating a similarity index. The entity information DB 142 includes a document sequence, a mention sequence, an entity sequence, and a category sequence. The document sequence stores the content of the document. The mention column stores the mention contained in the document. The entity column stores the entity contained in the document. The category column stores the category to which the entity belongs on the large knowledge base.

  FIG. 3 is an explanatory view showing an example of the record layout of the data set DB 143. As shown in FIG. The data set DB 143 stores for each category whether or not the document pair matches. The information stored in the data set DB 143 is one of learning corpuses for creating a similarity index. The data set DB 143 includes a category column, a first document column, a second document column, and a label column. The category column stores the document category. Each of the first document sequence and the second document sequence stores the content of the document. The label column stores whether the entity of the first document matches the entity of the second document. The data set DB 143 is created based on the contents of the entity information DB 142.

  FIG. 4 is an explanatory view showing an example of the record layout of the type-category correspondence DB 144. As shown in FIG. The type-category correspondence DB 144 includes specific expression type columns and knowledge base category columns. The specific expression type sequence stores specific expression types obtained by specific expression extraction. The knowledge base category column stores large knowledge base categories corresponding to specific expression types. The type-category correspondence DB 144 is manually created in advance. Not limited to that, it may be generated from machine learning.

  FIG. 5 is an explanatory view showing an example of a record of the similarity index DB 145. As shown in FIG. The similarity index DB 145 stores coefficients for calculating the similarity index by category. The calculation formula of the similarity index is, for example, the following formula (1).

Similarity index = a × word similarity + b × proper noun similarity + c × document URL similarity
+ ... + b_w × I (word w matches) + ... (1)

  In FIG. 5, a person / similarity index 1451, a company / similarity index 1452, and a sport / similarity index 1453 are shown.

  Next, processing performed by the similarity score calculation device 1 will be described. FIG. 6 is a flowchart showing the procedure of the similarity index creation process. The similarity index creation process is an operation in the similarity index creation mode. The CPU 11 of the similarity score calculation device 1 acquires a document for creating a similarity index, and stores the document in the document DB 141 (step S1). The document to be acquired is a document in which a link to a large-scale knowledge base is embedded. The document is acquired from another computer via the communication unit 15. The document may be acquired from the portable storage medium 1a storing the document via the reading unit 17. The document may be acquired from the stored semiconductor memory 1b.

  The CPU 11 acquires entity information from each document stored in the document DB 141, and stores the acquired entity information in the entity information DB 142 (step S2). As described above, each document stored in the document DB 141 is embedded with a link to a large scale knowledge base. Here, the portion in which the link is embedded is referred to as a mention, and the link destination is referred to as an entity. Also, a category is a category to which an entity belongs on a large scale knowledge base. The CPU 11 associates the document, the mention, the entity, and the category, and stores them in the entity information DB 142.

  The CPU 11 selects one category to be processed (step S3). The CPU 11 acquires entity information on the selected category from the entity information DB 142 (step S4). The CPU 11 creates a data set from the acquired entity information, and stores the data set in the data set DB 143 (step S5). In the acquired entity information, the CPU 11 creates a document pair consisting of two documents having the same mention. The CPU 11 compares the entities of each of the documents included in the document pair for each of the acquired document pairs. The CPU 11 determines a label to be assigned to the document pair based on the comparison result. If the CPU 11 determines that the two entities match, it labels the document pair as a match. If the CPU 11 determines that the two entities are different, it labels the document pair as non-coincidence. The CPU 11 stores the labeled document pair, that is, the data set in the data set DB 143.

  The CPU 11 creates a similarity index based on the created data set (step S6). The similarity index is determined by, for example, similarity score learning by machine learning using SVM (Support Vetor Machine) or logistic regression. Since the creation of the similarity index by machine learning is a known technique, the details will be omitted.

  The CPU 11 associates the created similarity index with the category, and stores the created similarity index in the similarity index DB 145 (step S7). The CPU 11 determines whether there is an unprocessed category (step S8). Here, unprocessed means that the similarity index is not created. If it is determined that there is an unprocessed category (YES in step S8), the CPU 11 returns the process to step S3 and performs the process on the unprocessed category. If the CPU 11 determines that there is no unprocessed category (NO in step S8), the process ends.

  Subsequently, the operation of the similarity score calculation device 1 in the similarity score calculation mode will be described. In the operation of the similarity score calculation mode, the similarity score calculation device 1 uses a category-based similarity index. Therefore, before the operation in the similarity score calculation mode, it is premised that the similarity index according to category is created by the operation of the similarity index creation mode.

Before describing similarity score calculation, specific expression extraction will be described. Named entity extraction is a well-known technique, so it will be described briefly. Named entity extraction is a technique for extracting, from documents, proper nouns such as persons and companies, and numerical expressions. There are multiple types (herein, referred to as types) of specific expressions obtained for specific expression extraction. The specific expression extraction makes it possible to extract the appearance location of the specific expression included in the document and its type. For example, specific expression extraction is performed on the sentence "Taro Tanaka is a researcher at OX pharmaceutical in Shiodome." The result is “<person> Tanaka Taro </ person> is a researcher at <company> OX pharmaceutical </ company> at <place> Shiodome </ place >>. Here, the underlined part, that is, the part enclosed by tags <...></...> indicates that it is a unique expression. ... in the tag indicates the type. In the above example, it shows that "Taro Tanaka" is a type: specific expression of a person. Indicates that "Shiodome" is a type: a specific expression of a place. Indicates that "OX Pharmaceutical" is a type: a unique expression of a company.

  FIG. 7 is a flowchart showing the procedure of the similarity score calculation process. The CPU 11 acquires a keyword and a document pair (step S11). A keyword is a word that is a criterion for determining the degree of similarity. Also, keywords are premised to be mentions. For example, when it is desired to calculate the similarity score of a document pair written for Tanaka Taro, the keyword is Taro Tanaka. The document pair is a pair of documents for which the similarity score is to be calculated. The CPU 11 extracts a unique expression for each of the documents included in the document pair (step S12). For each of the documents, the unique expression type of the mention corresponding to the keyword is obtained. The CPU 11 determines whether or not the unique expression types of each of the documents in the document match (step S13). If the CPU 11 determines that the specific expression types do not match (NO in step S13), it determines that the document pair does not match, and outputs a predetermined minimum score (step S14). The CPU 11 ends the process. If the CPU 11 determines that the specific expression types acquired from the respective documents match (YES in step S13), the CPU 11 acquires a category corresponding to the matched specific expression type from the type-category correspondence DB 144 (step S15). The CPU 11 acquires a similarity index corresponding to the acquired category from the similarity index DB 145, and calculates a similarity score (step S16). The CPU 11 outputs the calculated score (step S17).

  When there are a plurality of mentions in one document, the first one of them is used as a representative to calculate the score. Alternatively, score calculation is performed for each of the elements, and the average value of all the calculated scores is used as a final score.

  The present embodiment has the following effects. Different document similarity indexes are selected for each target (person, company, municipality) whose similarity is to be determined, and the similarity score is calculated. Therefore, it is possible to acquire the similarity score with high accuracy.

  Subsequently, clustering of document groups using the similarity score will be described. In the following description, as an example, clustering of documents by entity of a person will be described. For example, it is a case where a large number of documents written about a person such as a person biography, a biography, a memoir, etc. are classified according to the person being taken up. FIG. 8 is a flowchart showing the procedure of the clustering process. The CPU 11 acquires the category and the document group (step S21). The category is here a person. The unique expression extraction is performed on each of the documents included in the document group (step S22). The document group is divided for each personal name extracted by the CPU 11 specific expression extraction, and stored in the document DB 141 or the like (step S23). The CPU 11 acquires a similarity index corresponding to the category: person from the similarity index DB 145 (step S24). The CPU 11 calculates the similarity score from the document group divided for each personal name. (Step S25). The CPU 11 uses the similarity score to organize the document group (step S26). The CPU 11 performs steps S25 and S26 for each document group divided for each personal name. The CPU 11 outputs the result (step S27) and ends the process.

  FIG. 9 is an explanatory view showing an example of document grouping using the similarity index. When document grouping is performed, not only the similarity index for the "document-document" pair but also the "document-document group" pair and the "document group-document group" pair are defined. FIG. 9A shows the process for the "document-document" pair. The similarity score is calculated for the document including the document 1 and the document 2. If the score value is larger than a predetermined threshold value, the two documents are made the same document group. If the score value is less than or equal to the threshold, the two documents are different document groups.

  9B and 9C illustrate the process for the "document-document group" pair. In the example shown in FIG. 9B and FIG. 9C, when two groups of group 1 and group 2 have already been created as document groups, processing for determining the group to which a document (new document) to which grouping has not been made belongs Is shown. FIG. 9B shows an example in which a new document is divided into group 1. The similarity score between the new document and group 1 and the similarity score between the new document and group 2 are calculated. As a result of calculation, it is assumed that 3.5 is obtained as the former and 0.5 as the latter. At this time, the new document is grouped into a group having a higher score, that is, Group 1. FIG. 9C shows a case where a new document is not divided into any group, and a new group is created. When any of the calculated similarity scores is less than or equal to a predetermined threshold, a new document is not divided into existing groups, but a new group is created. In the example shown in FIG. 9C, the threshold is defined as zero. Then, the similarity score between the new document and the group 1 is -1.0, and the similarity score between the new document and the group 2 is -0.5. Since any similarity score is less than or equal to 0, new documents are grouped into newly created group 3. The document group can be organized by the process of FIG.

  Subsequently, another example of the clustering process will be described. This is an example of clustering for each entity of a person as described above. An example here is a process of clustering the result of applying a full text search to a document group with a target personal name. This processing is effective when there is another person with the same surname and the same name other than the intended person, and a document related to the other person is also included in the search result.

  FIG. 10 is a flowchart showing another procedure of the clustering process. The CPU 11 acquires a category, a keyword, and a document group (step S31). Here the category is a person. The keyword is, for example, a personal name such as Ichiro Suzuki. The CPU 11 performs a keyword search on the document group (step S32). The CPU 11 stores the document hit in the keyword search in the document DB 141 or the like (step S33). The CPU 11 acquires the similarity index corresponding to the category: person from the similarity index DB 145 (step S34). The CPU 11 calculates the similarity score for the document group stored in step S33 (step S35). The CPU 11 organizes the document group using the calculated score (step S36). The CPU 11 outputs the summarized result (step S37) and ends the process. The calculation of the similarity score and the grouping of the document group are the same as those described above, so the description will be omitted.

  Next, the type-category correspondence DB 144 will be described in detail. The similarity score calculation device 1 realizes improvement in the accuracy of the similarity score by selectively using the similarity index for each category of the document. The category of the document is obtained by converting the specific expression type obtained from the specific expression extraction into a category using the type-category correspondence DB 144. Then, the similarity index corresponding to the category is selected. Therefore, the accuracy of the type-category correspondence DB 144 affects the accuracy of the similarity score.

  As a method of creating the type-category correspondence DB 144, two methods will be described. The first method is a case where the unique expression type and the category are in one-to-one correspondence as described above. The second method is one-to-many correspondence between specific expression types and categories. In the following description, geographical position information is described as an example. In the specific representation type, geographical location information is LOCATION. In the large-scale knowledge-based category, geographical location information has three types: Japanese municipalities, Japanese wards, and Japanese geography.

  FIG. 11 is an explanatory view showing a method of creating the type-category correspondence DB 144. As shown in FIG. FIG. 11 shows the case where the unique expression types and the categories are in one-to-one correspondence. Similarity indicators are created for each category. Therefore, the category: similarity index 145a corresponding to Japanese municipalities, category: similarity index 145b corresponding to Japanese ward, and category: similarity index 145c corresponding to geography of Japan are created by the similarity index creation processing. Be done.

  After creating three similarity indices, the similarity indices are evaluated, and the category corresponding to the index that is evaluated to be the most accurate corresponds to the unique expression LOCATION. After the creation of the similarity index, a data set is generated from a group of documents different from the document used when creating the similarity index. The data set is as described above. The data set includes a document pair and a label indicating whether the entities of the document pair match. The similarity score of the document pair included in each of the generated data sets is calculated for each similarity index. By comparing the similarity score with the value of the label, it is possible to calculate the accuracy of the similarity index. Since the method of calculating the accuracy is a known technique, the description will be omitted. In the example of FIG. 11, the category: the accuracy index corresponding to the municipalities in Japan is the highest, so the specific expression type: the category corresponding to LOCATION stores a record that the municipalities in Japan are in the type-category correspondence DB 144 Do.

  FIG. 12 is an explanatory view showing another method of creating the type-category correspondence DB 144. As shown in FIG. FIG. 12 shows the case where one-to-many correspondence is made between specific expression types and categories. The point of creating three similarity indexes 145a, 145b, 145c and the point of evaluating each of the three similarity indexes 145a, 145b, 145c are the same as in the case of one-to-one correspondence.

  In the case of one-to-many correspondence, the score by the similarity index of a plurality of categories is combined by weighting for one specific expression type. Weighting is determined by the evaluation result of each index. In the example shown in FIG. 12, the weighting is the accuracy of each index. When the specific expression type is LOCATION, the type-category correspondence DB 144 uses the similarity index 145a corresponding to the municipalities of Japan, the similarity index 145b corresponding to the ward of Japan, and the similarity index 145c corresponding to the geography of Japan Is shown. First, a score is calculated using each of the similarity indexes 145a, 145b, and 145c. When the respective scores are SC1, SC2 and SC3, the final score S is calculated by the following equation (2).

  S = 0.8 × SC1 + 0.6 × SC2 + 0.3 × SC3 (2)

  As described above, the accuracy of the similarity score can be improved by creating the type-category correspondence DB 144 using machine learning instead of human hands.

  FIG. 13 is a block diagram showing an example of a functional configuration of the similarity score calculation device 1. The similarity score calculation device 1 includes an acquisition unit 11a, a group generation unit 11b, a document pair generation unit 11c, an assignment unit 11d, a creation unit 11e, and an output unit 11f. These function units are realized by the CPU 11 operating based on the control program 1P.

  The acquisition unit 11a acquires a document in which a mention in the document is associated with an entity having category information in the knowledge base. The group generation unit 11 b generates a document group in which the acquired document is classified into each associated category. The document pair generation unit 11 c generates, for each of the generated document groups, a document pair including a document associated with the same mention. The assignment unit 11d assigns a label indicating whether the entities match, to the generated document pair. The creation unit 11 e creates the similarity index based on the document pair to which the label is attached. The output unit 11 f outputs the created similarity index in association with the category corresponding to the document group.

The technical features (component requirements) described in the respective embodiments can be combined with each other, and by combining, new technical features can be formed.
It should be understood that the embodiments disclosed herein are illustrative in all respects and not restrictive. The scope of the present invention is indicated not by the meaning described above but by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

  Further, the following appendices will be disclosed regarding the above embodiment.

(Supplementary Note 1)
The computer is
Get a document in which a mention in the document is associated with an entity with category information in the knowledge base,
Generating a document group in which the acquired documents are classified according to the categories associated with each other;
Generating a document pair including the document to which the same mention is associated for each of the generated document groups;
Label the generated document pair whether the entity matches or not,
Create a similarity index based on the labeled document pair,
The discrimination | determination method which performs the process which matches the created similarity index with the said category corresponding to the said document group, and outputs it.

(Supplementary Note 2)
The determination method according to appendix 1, wherein the category is defined in the knowledge base and given to each of the entities.

(Supplementary Note 3)
Accept keywords and multiple documents that hit the search using the keywords,
Extract specific names from multiple accepted documents,
Compare specific expression types of mentions corresponding to the keywords for each document,
When a plurality of documents match, the category corresponding to the specific expression type is acquired,
Acquire the similarity index associated with the acquired category,
Find similarity score among multiple documents using the obtained similarity index,
The determination method according to Appendix 1 or 2, wherein the calculated similarity score is output.

(Supplementary Note 4)
Accept the category and multiple documents,
Extract specific names from multiple accepted documents,
Select one of the specific expression types corresponding to the extracted specific expression,
The plurality of documents are divided according to the specific representation of the selected specific representation type,
Acquire a similarity index associated with the category,
Determine the similarity score for each document group obtained by division using the acquired similarity index,
The determination method according to Appendix 1 or 2, wherein the calculated similarity score is output.

(Supplementary Note 5)
Clustering the plurality of documents or document groups according to the similarity score;
The determination method according to appendix 3 or 4, which outputs the result of clustering.

(Supplementary Note 6)
Get a document in which a mention in the document is associated with an entity with category information in the knowledge base,
Generating a document group in which the acquired documents are classified according to the categories associated with each other;
Generating a document pair including the document to which the same mention is associated for each of the generated document groups;
Label the generated document pair whether the entity matches or not,
Create a similarity index based on the labeled document pair,
A determination program that causes a computer to execute a process of outputting the created similarity index in association with the category corresponding to the document group.

(Appendix 7)
An acquisition unit for acquiring a document in which a mention in the document is associated with an entity having category information in the knowledge base;
A group generation unit that generates a document group in which the acquired document is classified according to the categories associated with each other;
A document pair generation unit that generates a document pair including the document to which the same mention is associated for each of the generated document groups;
An assigning unit for assigning a label indicating whether the entity matches the generated document pair;
A creation unit (11e) that creates a similarity index based on the labeled document pair;
An output unit that outputs the created similarity index in association with the category corresponding to the document group.

1 Similarity score calculator 11 CPU
11a acquisition unit 11b group generation unit 11c document pair generation unit 11d addition unit 11e generation unit 11f output unit 12 ROM
13 RAM
14 Mass Storage 141 Document DB
142 Entity information DB
143 Data set DB
144 category corresponding DB
145 Similarity Index DB
15 communication unit 16 input / output unit 17 reading unit 1P control program 1a portable storage medium 1b semiconductor memory B bus

Claims (5)

The computer is
Get a document in which a mention in the document is associated with an entity with category information in the knowledge base,
Generating a document group in which the acquired documents are classified according to the categories associated with each other;
Generating a document pair including the document to which the same mention is associated for each of the generated document groups;
Label the generated document pair whether the entity matches or not,
Create a similarity index based on the labeled document pair,
The discrimination | determination method which performs the process which matches the created similarity index with the said category corresponding to the said document group, and outputs it. The method according to claim 1, wherein the category is defined in the knowledge base and is given to each of the entities. Accept keywords and multiple documents that hit the search using the keywords,
Extract specific names from multiple accepted documents,
Compare specific expression types of mentions corresponding to the keywords for each document,
When a plurality of documents match, the category corresponding to the specific expression type is acquired,
Acquire the similarity index associated with the acquired category,
Find similarity score among multiple documents using the obtained similarity index,
The discrimination | determination method of Claim 1 or Claim 2 which outputs the calculated | required similarity score. Get a document in which a mention in the document is associated with an entity with category information in the knowledge base,
Generating a document group in which the acquired documents are classified according to the categories associated with each other;
Generating a document pair including the document to which the same mention is associated for each of the generated document groups;
Label the generated document pair whether the entity matches or not,
Create a similarity index based on the labeled document pair,
A determination program that causes a computer to execute a process of outputting the created similarity index in association with the category corresponding to the document group. An acquisition unit for acquiring a document in which a mention in the document is associated with an entity having category information in the knowledge base;
A group generation unit that generates a document group in which the acquired document is classified according to the categories associated with each other;
A document pair generation unit that generates a document pair including the document to which the same mention is associated for each of the generated document groups;
An assigning unit for assigning a label indicating whether the entity matches the generated document pair;
A creation unit that creates a similarity index based on the labeled document pair;
An output unit that outputs the created similarity index in association with the category corresponding to the document group.

Images