JP2012168678A - Inter-document similarity calculation device, inter-document similarity calculation method and inter-document similarity calculation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an inter-document similarity calculation device capable of calculating similarity with high accuracy while preventing a burden from being excessive in calculating the similarity between documents.SOLUTION: A device 100 comprises: a unit (101) for, when representing the total number of characters constituting a sentence included in each of multiple documents as N, generating suffix part information that shows a suffix part equivalent to a remaining character string after excluding i characters from the top of the sentence, for each of integers i from 0 to N-1; a unit (102) for selecting a suffix part generated based on the multiple sentences as a reference suffix part from the suffix parts; a unit (103) for, with respect to each of the multiple documents, generating similarity basic information that shows whether or not the document includes the reference suffix part; and a unit (104) for calculating the similarity showing a level of similarity between a first document and a second document based on the similarity basic information generated for the first document and the similarity basic information generated for the second document.

Description

  The present invention relates to an inter-document similarity calculating apparatus that calculates a similarity indicating a degree of similarity between a plurality of documents.

  There is known an inter-document similarity calculation apparatus that calculates a similarity indicating a degree of similarity between a plurality of documents. The inter-document similarity calculation device described in Patent Document 1 as one of this type of inter-document similarity calculation device divides the document into words by performing morphological analysis on each of a plurality of documents. .

  Further, the inter-document similarity calculation apparatus counts the number of occurrences of the word in the document for each word for each document. Then, the inter-document similarity calculation device generates similarity basic information representing a vector having, as a value, the number of occurrences of the word as a component representing the word for each word.

  The inter-document similarity calculation device includes an angle between a vector represented by the similarity basic information generated for the first document and a vector represented by the similarity basic information generated for the second document. The degree of similarity having a value that increases as becomes smaller is calculated.

JP 2002-108894 A

  By the way, the inter-document similarity calculation device performs morphological analysis based on words registered in advance in the dictionary. Therefore, when a word that is not registered in the dictionary is included in the document, the inter-document similarity calculation device may not be able to use an accurate word as a vector component represented by the similarity basic information. This type of problem also occurs in an inter-document similarity calculation apparatus configured to use a feature word preset by a user as a vector component represented by similarity basic information.

  As a result, there is a possibility that the similarity between the documents cannot be calculated with high accuracy.

  It is also conceivable to configure the inter-document similarity calculation apparatus so that a character string as an index generated according to an N-gram method is used as a vector component represented by the similarity basic information. According to this, even when a word that is not registered in the dictionary is included in the document, it is expected that the similarity can be calculated with high accuracy.

  By the way, a part of the first word may coincide with the whole of the second word. For example, it is assumed that the first word is “printer” and the second word is “pudding”. In this case, when the inter-document similarity calculation device generates “pudding” as an index, the similarity between the first document related to “printer” and the second document related to “printing” is excessively large. There is a risk of calculating the value. That is, in this case, there is a possibility that the similarity cannot be calculated with high accuracy.

  In addition, the index is generated including a character string that does not have a specific meaning, for example, “N is a meal”. Therefore, the total number of generated indexes is relatively large. For this reason, the number of vector components (dimensions) represented by the similarity basic information is excessively large. As a result, there is a possibility that the load of the inter-document similarity calculation device when calculating the similarity based on the similarity basic information becomes excessive.

  For this reason, the object of the present invention is the above-described problem that “the similarity between documents cannot be calculated with high accuracy, and that the load when calculating the similarity between documents is excessive. It is an object of the present invention to provide an inter-document similarity calculation apparatus capable of solving the “occurrence of a case where“

In order to achieve such an object, an inter-document similarity calculation apparatus according to an aspect of the present invention includes:
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selecting means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generation means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the similarity basic information generated for the first document of the plurality of documents and the similarity basic information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
Is provided.

In addition, the inter-document similarity calculation method according to another aspect of the present invention is:
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Generate suffix information that is information representing the suffix that is the remaining character string,
From the suffixes represented by the generated suffix information, select a suffix generated based on a plurality of sentences as a reference suffix,
For each of the plurality of documents, generate similarity basic information indicating whether or not the document includes each of the selected reference suffixes,
Based on the similarity basic information generated for the first document of the plurality of documents and the similarity basic information generated for the second document of the plurality of documents. This is a method of calculating a similarity indicating the degree of similarity between the first document and the second document.

Moreover, the similarity calculation program between documents which is the other form of this invention is the following.
In the information processing device,
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selecting means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generation means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the similarity basic information generated for the first document of the plurality of documents and the similarity basic information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
It is a program for realizing.

  According to the present invention configured as described above, it is possible to calculate the similarity with high accuracy while preventing an excessive load when calculating the similarity between documents.

It is a block diagram showing the outline of the inter-document similarity calculation apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing which showed notionally the vector which similarity basic information represents. It is the flowchart which showed the document similarity calculation program which the document similarity calculation apparatus which concerns on 1st Embodiment of this invention performs. It is the flowchart which showed the reference | standard suffix part selection process which the similarity calculation apparatus between documents which concerns on 1st Embodiment of this invention performs. It is the flowchart which showed the similarity basic information generation process which the inter-document similarity calculation apparatus which concerns on 1st Embodiment of this invention performs. It is explanatory drawing which showed notionally the relationship between a sentence, suffix information, and a reference suffix. It is a block diagram showing the outline of the similarity calculation apparatus between documents which concerns on 2nd Embodiment of this invention.

  Hereinafter, embodiments of an inter-document similarity calculation apparatus, an inter-document similarity calculation method, and an inter-document similarity calculation program according to the present invention will be described with reference to FIGS.

<First Embodiment>
(Constitution)
As shown in FIG. 1, the inter-document similarity calculation apparatus 10 according to the first embodiment is an information processing apparatus. The inter-document similarity calculation device 10 is a personal computer, server device, mobile phone terminal, PHS (Personal Handyphone System), PDA (Personal Data Assistant), Personal Digital Assistant, car navigation terminal, game terminal, or the like. There may be.

  The inter-document similarity calculation device 10 includes a central processing unit (CPU) (not shown) and a storage device (memory and hard disk drive (HDD)). The inter-document similarity calculation device 10 is configured to realize functions to be described later when a CPU executes a program stored in a storage device.

(function)
FIG. 1 is a block diagram showing the function of the inter-document similarity calculation apparatus 10 configured as described above.
The functions of the inter-document similarity calculation device 10 include a document information storage unit 11, a suffix information generation unit (suffix information generation unit) 12, a reference suffix selection unit (reference suffix selection unit) 13, and a similarity. A basic information generation unit (similarity basic information generation unit) 14 and a similarity calculation unit (similarity calculation unit) 15 are included.

  The document information storage unit 11 stores a plurality of document information. Document information is information representing a document. The document includes at least one sentence. A sentence is composed of a character string composed of a plurality of characters. The document information stored in the document information storage unit 11 may be information input by a user or information received from another information processing apparatus.

  The suffix information generation unit 12 generates suffix information for each sentence included in each of a plurality of documents represented by a plurality of document information stored in the document information storage unit 11. The suffix information is a remaining character string excluding i characters from the head of the sentence for each of the integers i from 0 to N−1, where N represents the total number of characters constituting the sentence. Information indicating a suffix (that is, N suffixes).

  In this example, the suffix information is information representing a suffix array (suffix array). The suffix array is an array in which suffixes are rearranged in dictionary order.

For example, when the sentence is “BANANA”, the suffix information generation unit 12 includes six suffixes including “BANANA”, “ANANA”, “NANA”, “ANA”, “NA”, and “A”. Suffix information representing an array of “A”, “ANA”, “ANANA”, “BANANA”, “NA”, and “NANA” is generated by rearranging the parts in dictionary order.
Note that the suffix information may be information representing a suffix tree (suffix tree).

  The reference suffix selection unit 13 selects a suffix generated based on a plurality of sentences as a reference suffix from the suffixes represented by the suffix information generated by the suffix information generation unit 12. The reference suffix selecting unit 13 generates a suffix generated based on a sentence having a number larger than a preset threshold number (an integer of 2 or more) from the suffixes represented by the generated suffix information. May be selected as the reference suffix.

  The similarity basic information generation unit 14 generates similarity basic information for each of a plurality of documents represented by a plurality of document information stored in the document information storage unit 11. The similarity basic information is information indicating whether or not the document includes each of the reference suffixes selected by the reference suffix selection unit 13.

  In this example, the similarity basic information is information representing a vector having components representing each of the reference suffixes selected by the reference suffix selection unit 13.

  Specifically, the similarity basic information generation unit 14 sets each value of the vector component represented by the similarity basic information to a positive value when the document includes the reference suffix represented by the component, On the other hand, the reference suffix represented by the component is set to 0 when the document is not included.

  Further, in this example, the similarity basic information generation unit 14 includes each value of the vector component represented by the similarity basic information and the number of the document including the reference suffix represented by the component (that is, the document) The number of times the reference suffix appears) is multiplied by the increment value.

  That is, the increment value is a value representing an increment that increases the value as a component representing the reference suffix each time the number of the reference suffix included in the document increases by one. In this example, the similarity basic information generation unit 14 sets the increment value to a value that increases as the total number of characters constituting the reference suffix increases (for example, a value that is directly proportional to the total number of characters).

  In this way, the similarity basic information generation unit 14 converts information representing a vector having a value that increases as a component representing the reference suffix as the number of reference suffixes included in the document increases. It can be said that it is generated as.

  The similarity calculation unit 15 calculates a similarity indicating the degree to which any two documents among a plurality of documents represented by a plurality of document information stored in the document information storage unit 11 are similar. The similarity calculation unit 15 applies the similarity basic information generated by the similarity basic information generation unit 14 to one of the two documents (first document) and the other of the two documents (second document). The similarity is calculated based on the similarity basic information generated by the similarity basic information generation unit 14 for the document.

  Specifically, the similarity calculation unit 15 includes a vector represented by the similarity basic information generated for the first document, a vector represented by the similarity basic information generated for the second document, A value that increases as the angle between and decreases (in this example, the cosine of the angle formed by the two vectors) is calculated as the similarity.

  FIG. 2 conceptually shows a vector V1 for the first document and a vector V2 for the second document when “cat”, “white”, and “鼠” are selected as reference suffixes. It is explanatory drawing shown in.

  In this example, the similarity calculation unit 15 calculates a similarity for each of all combinations of a plurality of documents represented by a plurality of document information stored in the document information storage unit 11. The similarity calculation unit 15 calculates the similarity only for a specific combination (for example, designated by the user) of a plurality of documents represented by a plurality of document information stored in the document information storage unit 11. It may be configured as follows.

(Operation)
Next, the operation of the above-described inter-document similarity calculation apparatus 10 will be described.
The CPU of the inter-document similarity calculation apparatus 10 is configured to execute the inter-document similarity calculation program shown by the flowcharts in FIGS.

  More specifically, when the inter-document similarity calculation apparatus 10 starts processing of the inter-document similarity calculation program, first, each sentence (a plurality of document information represented by a plurality of document information stored in the document information storage unit 11) is described. Suffix information is generated for each of the sentences included in each document (step S101).

  For example, as shown in FIGS. 6A and 6B, the first document includes “a black cat and a white cat rang” as sentence # 1, and the second document is sentence # 2. Assuming that "my white cat sang" as well. In this case, the inter-document similarity calculation apparatus 10 generates information representing the suffix array shown in FIG. 6A as the suffix information for the sentence # 1. Further, the inter-document similarity calculation apparatus 10 generates information representing the suffix array shown in FIG. 6B as the suffix information for the sentence # 2.

  Then, the inter-document similarity calculation apparatus 10 performs a reference suffix selection process for each of the suffixes (each suffix) indicated by the generated suffix information (step S102). Specifically, the inter-document similarity calculation apparatus 10 performs the reference suffix selection process illustrated in FIG. 4 for each suffix.

  That is, first, the inter-document similarity calculation apparatus 10 obtains the number (basic sentence number) of sentences (that is, including the suffix) from which a suffix that is a target of the reference suffix selection process is acquired. Obtain (step S201).

Next, the inter-document similarity calculation device 10 determines whether or not the acquired number of basic sentences is larger than a preset threshold number (1 in this example) (step S202).
When the number of basic sentences is larger than the threshold number, the inter-document similarity calculation apparatus 10 determines “Yes”, proceeds to step S203, and selects a suffix that is a target of the reference suffix selection process as a reference suffix. . Then, the inter-document similarity calculation apparatus 10 ends the reference suffix selection process.

  On the other hand, when the number of basic sentences is equal to or less than the threshold number, the inter-document similarity calculation apparatus 10 determines “No” and does not select the suffix that is the target of the reference suffix selection process as the reference suffix. Then, the reference suffix selection process is terminated.

  For example, as shown in FIGS. 6A and 6B, the first document includes “a black cat and a white cat rang” as sentence # 1, and the second document is sentence # 2. Assuming that "my white cat sang" as well. In this case, the inter-document similarity calculation apparatus 10 selects the suffix shown in (C) of FIG. 6 as the reference suffix.

  Next, the inter-document similarity calculation apparatus 10 executes similarity basic information generation processing for each document (each of a plurality of documents represented by a plurality of document information stored in the document information storage unit 11) (step S1). S103).

  Specifically, the inter-document similarity calculation apparatus 10 executes the similarity basic information generation process shown in FIG. 5 for each document.

  That is, the inter-document similarity calculation apparatus 10 executes a loop process (steps S301 to S305) in which each selected reference suffix is sequentially processed.

  In the loop processing, first, the inter-document similarity calculation apparatus 10 acquires the total number of characters (number of characters of the reference suffix) constituting the reference suffix to be processed (step S302). Next, the inter-document similarity calculation apparatus 10 acquires the number of reference suffixes (number of reference suffixes) to be processed included in the document to be processed by the similarity basic information generation process (step S303).

  The inter-document similarity calculation device 10 then multiplies the acquired number of reference suffixes by the number of characters (increment value) of the acquired reference suffix, and the value of the component representing the reference suffix to be processed ( (Component value) is calculated (step S304).

  Then, the inter-document similarity calculation apparatus 10 performs the loop process (steps S301 to S305) for all the reference suffixes selected in step S203, and then proceeds to step S306.

  Then, the inter-document similarity calculation apparatus 10 generates similarity basic information representing a vector having components representing each of the selected reference suffixes (step S306). Thereafter, the inter-document similarity calculation apparatus 10 ends the similarity basic information generation process.

Next, the inter-document similarity calculation device 10 calculates the similarity for each of all combinations of the plurality of documents represented by the plurality of document information stored in the document information storage unit 11 (step S104). Specifically, the inter-document similarity calculation device 10 represents the vector represented by the similarity basic information generated for the first document and the similarity basic information generated for the second document. The cosine of the angle formed by the vector is calculated as the similarity.
Thereafter, the inter-document similarity calculation apparatus 10 ends the processing of the inter-document similarity calculation program.

  As described above, according to the inter-document similarity calculation apparatus 10 according to the first embodiment, it is possible to prevent an excessive load on the inter-document similarity calculation apparatus 10 when calculating the similarity between documents. However, the similarity between documents can be calculated with high accuracy.

  Further, the inter-document similarity calculation apparatus 10 according to the first embodiment calculates an increment value that increases a value as a component representing the reference suffix each time the number of reference suffixes included in the document increases by 1. It is configured to increase as the total number of characters constituting the suffix increases.

  By the way, as the total number of characters constituting the suffix portion increases, the suffix portion better represents the characteristics of the document including the suffix portion. Therefore, according to the inter-document similarity calculation apparatus 10 configured as described above, it is possible to calculate the similarity between documents with higher accuracy.

  The inter-document similarity calculation apparatus 10 according to the modification of the first embodiment uses a suffix generated based on a plurality of documents as a reference suffix from the suffixes represented by the generated suffix information. Configured to select as.

  By the way, the greater the number of documents including the same suffix, the better the feature of the document including the suffix. Therefore, by configuring the inter-document similarity calculation apparatus 10 in this way, it is possible to calculate the similarity between documents with higher accuracy.

  Further, the inter-document similarity calculation device 10 is configured to store the generated suffix information in association with the document identification information for identifying the document that is the basis for generating the suffix information. May be. Further, the inter-document similarity calculation device 10 stores the generated similarity basic information and the document identification information for identifying the document that is the basis for generating the similarity basic information in association with each other. It may be configured.

Second Embodiment
Next, an inter-document similarity calculation apparatus according to a second embodiment of the present invention will be described with reference to FIG.
The inter-document similarity calculation apparatus 100 according to the second embodiment
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. A suffix information generating unit (suffix information generating means) 101 that generates suffix information that is information indicating a suffix that is a residual character string;
A reference suffix selecting unit (reference suffix selecting means) 102 for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
A similarity basic information generation unit (similarity basic information generation means) that generates, for each of the plurality of documents, similarity basic information indicating whether or not the document includes each of the selected reference suffixes. 103,
Based on the similarity basic information generated for the first document of the plurality of documents and the similarity basic information generated for the second document of the plurality of documents. A similarity calculation unit (similarity calculation means) 104 for calculating a similarity indicating the degree of similarity between the first document and the second document;
Is provided.

  Accordingly, it is possible to calculate the similarity between documents with high accuracy while preventing an excessive load on the inter-document similarity calculation apparatus 100 when calculating the similarity between documents.

  Although the present invention has been described with reference to the above embodiment, the present invention is not limited to the above-described embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  For example, the inter-document similarity calculation apparatus may be configured to classify (for example, cluster) a plurality of documents based on the calculated similarity.

  In the above embodiments, each function of the inter-document similarity calculation device is realized by the CPU executing a program (software), but may be realized by hardware such as a circuit.

  In each of the above embodiments, the program is stored in the storage device, but may be stored in a computer-readable recording medium. For example, the recording medium is a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

  In addition, as another modified example of the above-described embodiment, any combination of the above-described embodiments and modified examples may be employed.

<Appendix>
A part or all of the above embodiment can be described as the following supplementary notes, but is not limited thereto.

(Appendix 1)
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selection means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generating means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
An inter-document similarity calculation apparatus.

  According to this, it is possible to calculate the similarity between documents with high accuracy while preventing an excessive load on the inter-document similarity calculation apparatus when calculating the similarity between documents.

(Appendix 2)
An inter-document similarity calculation apparatus according to appendix 1,
The similarity basic information generation means has a positive value as a component representing the reference suffix when the document includes the reference suffix for each of the selected reference suffixes, And, when the document does not include the reference suffix, information representing a vector having 0 as the component is generated as the similarity basic information,
The similarity calculation means includes a vector represented by the similarity basic information generated for the first document and a vector represented by the similarity basic information generated for the second document. An inter-document similarity calculation device configured to calculate a value that increases as the angle between them decreases as the similarity.

(Appendix 3)
An inter-document similarity calculation apparatus according to appendix 2,
The similarity basic information generation means uses, as the similarity basic information, information representing the vector having a value that increases as a component representing the reference suffix as the number of the reference suffixes included in the document increases. An inter-document similarity calculation device configured to generate.

(Appendix 4)
An inter-document similarity calculation apparatus according to appendix 3,
The similarity basic information generation means, each time the number of the reference suffixes included in the document increases by 1, increases an increment value that increases a value as a component representing the reference suffix, the characters constituting the reference suffix The inter-document similarity calculation device is configured to increase as the total number increases.

  By the way, as the total number of characters constituting the suffix portion increases, the suffix portion better represents the characteristics of the document including the suffix portion. Therefore, by configuring the inter-document similarity calculation apparatus as described above, it is possible to calculate the similarity between documents with higher accuracy.

(Appendix 5)
The inter-document similarity calculation device according to any one of appendix 1 to appendix 4,
The reference suffix selecting means is configured to select a suffix generated based on a plurality of documents from the suffixes represented by the generated suffix information as the reference suffix. Similarity calculation device.

  By the way, the greater the number of documents including the same suffix, the better the feature of the document including the suffix. Therefore, by configuring the inter-document similarity calculation apparatus as described above, it is possible to calculate the similarity between documents with higher accuracy.

(Appendix 6)
The inter-document similarity calculation device according to any one of appendix 1 to appendix 5,
The inter-document similarity calculation device, wherein the suffix information is information indicating a suffix tree or a suffix array.

(Appendix 7)
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Generate suffix information that is information representing the suffix that is the remaining character string,
From the suffixes represented by the generated suffix information, select a suffix generated based on a plurality of sentences as a reference suffix,
For each of the plurality of documents, generate similarity basic information indicating whether or not the document includes each of the selected reference suffixes,
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. An inter-document similarity calculation method for calculating a similarity indicating a degree of similarity between the first document and the second document.

(Appendix 8)
The method for calculating the similarity between documents according to appendix 7,
For each of the selected reference suffixes, the document has a positive value as a component representing the reference suffix when the document includes the reference suffix, while the document has the reference suffix When not included, information representing a vector having 0 as the component is generated as the similarity basic information,
The smaller the angle between the vector represented by the similarity basic information generated for the first document and the vector represented by the similarity basic information generated for the second document, the larger the smaller the angle is. The inter-document similarity calculation method for calculating a value obtained as the similarity.

(Appendix 9)
An inter-document similarity calculation method according to appendix 8,
The inter-document similarity calculation method for generating, as the similarity basic information, information representing the vector having a value that increases as a component representing the reference suffix as the number of the reference suffix included in the document increases. .

(Appendix 10)
In the information processing device,
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selection means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generating means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
Inter-document similarity calculation program for realizing

(Appendix 11)
An inter-document similarity calculation program according to attachment 10, wherein
The similarity basic information generation means has a positive value as a component representing the reference suffix when the document includes the reference suffix for each of the selected reference suffixes, And, when the document does not include the reference suffix, information representing a vector having 0 as the component is generated as the similarity basic information,
The similarity calculation means includes a vector represented by the similarity basic information generated for the first document and a vector represented by the similarity basic information generated for the second document. An inter-document similarity calculation program configured to calculate, as the similarity, a value that increases as the angle between them decreases.

(Appendix 12)
An inter-document similarity calculation program according to attachment 11, wherein
The similarity basic information generation means uses, as the similarity basic information, information representing the vector having a value that increases as a component representing the reference suffix as the number of the reference suffixes included in the document increases. An inter-document similarity calculation program configured to generate.

  The present invention can be applied to an inter-document similarity calculation device that calculates a degree of similarity indicating a degree of similarity between a plurality of documents, a document classification device that classifies a plurality of documents, and the like.

DESCRIPTION OF SYMBOLS 10 Inter-document similarity calculation apparatus 11 Document information storage part 12 Suffix information generation part 13 Reference suffix part selection part 14 Similarity basic information generation part 15 Similarity calculation part 100 Inter-document similarity calculation apparatus 101 Suffix information generation part 102 Reference suffix selection unit 103 Similarity basic information generation unit 104 Similarity calculation unit

Claims (10)

For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selection means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generating means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
An inter-document similarity calculation apparatus. The inter-document similarity calculation apparatus according to claim 1,
The similarity basic information generation means has a positive value as a component representing the reference suffix when the document includes the reference suffix for each of the selected reference suffixes, And, when the document does not include the reference suffix, information representing a vector having 0 as the component is generated as the similarity basic information,
The similarity calculation means includes a vector represented by the similarity basic information generated for the first document and a vector represented by the similarity basic information generated for the second document. An inter-document similarity calculation device configured to calculate a value that increases as the angle between them decreases as the similarity. The inter-document similarity calculation apparatus according to claim 2,
The similarity basic information generation means uses, as the similarity basic information, information representing the vector having a value that increases as a component representing the reference suffix as the number of the reference suffixes included in the document increases. An inter-document similarity calculation device configured to generate. The inter-document similarity calculation apparatus according to claim 3,
The similarity basic information generation means, each time the number of the reference suffixes included in the document increases by 1, increases an increment value that increases a value as a component representing the reference suffix, the characters constituting the reference suffix The inter-document similarity calculation device is configured to increase as the total number increases. The inter-document similarity calculation device according to any one of claims 1 to 4,
The reference suffix selecting means is configured to select a suffix generated based on a plurality of documents from the suffixes represented by the generated suffix information as the reference suffix. Similarity calculation device. The inter-document similarity calculation apparatus according to any one of claims 1 to 5,
The inter-document similarity calculation device, wherein the suffix information is information indicating a suffix tree or a suffix array. For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Generate suffix information that is information representing the suffix that is the remaining character string,
From the suffixes represented by the generated suffix information, select a suffix generated based on a plurality of sentences as a reference suffix,
For each of the plurality of documents, generate similarity basic information indicating whether or not the document includes each of the selected reference suffixes,
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. An inter-document similarity calculation method for calculating a similarity indicating a degree of similarity between the first document and the second document. The inter-document similarity calculation method according to claim 7,
For each of the selected reference suffixes, the document has a positive value as a component representing the reference suffix when the document includes the reference suffix, while the document has the reference suffix When not included, information representing a vector having 0 as the component is generated as the similarity basic information,
The smaller the angle between the vector represented by the similarity basic information generated for the first document and the vector represented by the similarity basic information generated for the second document, the larger the smaller the angle is. The inter-document similarity calculation method for calculating a value obtained as the similarity. The inter-document similarity calculation method according to claim 8,
The inter-document similarity calculation method for generating, as the similarity basic information, information representing the vector having a value that increases as a component representing the reference suffix as the number of the reference suffix included in the document increases. . In the information processing device,
For each sentence included in each of a plurality of documents, when the total number of characters constituting the sentence is represented by N, i characters are excluded from the head of the sentence for each integer i from 0 to N-1. Suffix information generating means for generating suffix information which is information representing a suffix that is a residual character string;
Reference suffix selection means for selecting, as a reference suffix, a suffix generated based on a plurality of sentences from the suffixes represented by the generated suffix information;
For each of the plurality of documents, similarity basic information generating means for generating similarity basic information indicating whether or not the document includes each of the selected reference suffixes;
Based on the basic similarity information generated for the first document of the plurality of documents and the basic similarity information generated for the second document of the plurality of documents. A similarity calculating means for calculating a similarity indicating the degree of similarity between the first document and the second document;
Inter-document similarity calculation program for realizing

Images