JP2001155027A - Method, system and device for calculating similarity between documents, and recording medium recorded with program for similarity calculation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system and a method for retrieving similar document, by which much deeper meaning processing is made possible and the dealing of compound word or phrase can be facilitated. SOLUTION: This device is provided with a first vector generating means for counting the word frequency of a word string decomposed by a word retrieval means and generating a first vector based on the word dictionary of an input document, a first similarity calculating means for comparing the first vector with a vector stored in a first vector storage means and calculating similarity, a document pattern generating means for defining the arrangement of words as a document pattern, pattern retrieving means for scanning the document pattern on the basis of patterns stored in a pattern dictionary storage means and decomposing the document pattern into the arrangements of patterns and words a second vector generating means for counting the pattern frequency in the arrangements of patterns and words and generating a second vector based on the pattern dictionary of the input document, a second similarity calculating means for comparing the second vector with a vector stored in a second vector storage means and calculating similarity, and similarity merging means for merging and outputting the outputs of the first and second similarity calculating means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a similarity calculation technique, and more particularly to a similar document search system, a similar document search method, and a recording medium storing a similar document search program which are suitable for use in search and classification. .

[0002]

2. Description of the Related Art As a conventional similar document search method in the field of information search, for example, reference 1 (G. Salton, M. McGill,
Introduction to Modern Information Retrieval, NewY
As described in Ork, McGraw-Hill, 1983), a method of calculating the distance or similarity of a document based on the frequency of occurrence of a word in the document is known. In this conventional similar document search method, a frequency vector of a word in each document is obtained, each vector is weighted by TF / IDF, and a cosine value of an angle formed by the two vectors (cos θ = (x · y ) / | X || y | where (x
y) is the inner product of two vectors x and y, and | x | and | y | are absolute values) as the similarity between documents. In the TF / IDF method, the importance of a word is defined using an appearance frequency tf and the reciprocal idf of the number of occurrences df, and the word in the category Ci
The importance (weight) W (tk, Ci) of tk is W (tk, Ci) = tf (tk, C
i) log (Li / df (tk, Ci) +1), tf (tk, Ci) is the frequency of occurrence of word tk in category Ci, and df (tk, Ci) is
Number of occurrences of word tk in category Ci, Li is category Ci
Represents the total number of texts in.

In calculating the similarity between two documents, for example, Japanese Patent Application Laid-Open No. 11-134359 discloses a morphological analysis of two sets of input documents and a summary to extract words and include the words in the summary. A method has been proposed in which a word to be weighted is calculated and the similarity between two documents is calculated based on the word included in each document.

[0004]

However, the above-mentioned conventional method has the following problems.

A first problem is that the accuracy of similarity between documents is low.

The reason is that only shallow semantic processing is performed based on a word vector.

[0007] The second problem is that it is difficult to handle compound words.

The reason is that an attempt was made to represent a document with one vector.

Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide a similar document search system and method capable of performing deeper semantic processing, and a recording medium.

Another object of the present invention is to provide a similar document search system and method and a recording medium which can easily handle compound words and collocations.

[0011]

A similarity search system according to the present invention that achieves the above object comprises a first similarity calculator based on a word dictionary, a second similarity calculator based on a pattern dictionary, and two similarity calculators. Means for calculating one similarity from the similarity.

Further, the present invention provides a first similarity calculating means based on a word dictionary, a second similarity calculating means based on a first pattern dictionary, and a third similarity calculating based on a second pattern dictionary. Means and means for calculating one similarity from the three similarities.

[0013]

Embodiments of the present invention will be described. In a preferred embodiment of the apparatus of the present invention, referring to FIG. 1, a first vector storage means (31) for storing a first vector according to a word dictionary of a stored document, and storing the word dictionary. Word dictionary storage means (3
2) a second vector storage unit (33) for storing a second vector of the stored document according to the pattern dictionary, and a pattern dictionary storage unit (34) for storing the pattern dictionary (3) And a document input means (1) for inputting a document, wherein the data processing device is configured to decompose the input document into a word string based on the words stored in the word dictionary storage means (34). 21), first vector generation means (22) for counting the word frequency of the decomposed word string, and generating a first vector of the input document based on the word dictionary, and the first vector generation means ( First similarity calculating means (23) for comparing the first vector generated in 22) with the first vector stored in the first vector storing means (31) and calculating the similarity thereof; When,
A document pattern is scanned by a document pattern generation unit (24) that uses a sequence of words decomposed by the word search unit (21) as a document pattern, and a pattern stored in the pattern dictionary storage unit (34). And a pattern search means (25) for decomposing into a sequence of words and a second vector generating means for counting a pattern frequency of the sequence of patterns and words and generating a second vector of the input document by the pattern dictionary (26) and the second vector generation means (2
A second similarity calculating means (2) for comparing the second vector generated in 6) with the second vector stored in the second vector storing means (33) and calculating the similarity thereof.
7) and similarity integration means (20) for integrating the output of the first similarity calculation means (23) and the output of the second similarity calculation means (27) and outputting the same as one similarity. And means (4) for outputting the similarity output from the similarity integrating means (20).

The present invention, in another embodiment, is shown in FIG.
, A storage device (3) includes first vector storage means (31) for storing a first vector of a stored document in a word dictionary, word dictionary storage means (32) for storing a word dictionary, A second vector storage means (3) for storing a second vector of the stored document according to the pattern dictionary.
3), pattern dictionary storage means (34) for storing the first pattern dictionary, and third vector storage means (35) for storing a third vector of the stored document according to the second pattern dictionary. And a second pattern dictionary storage means (36) for storing a second pattern dictionary, wherein the data processing device is configured to:
Word search means (2) for decomposing an input document into word strings
1), a first vector generating means (22) for counting the word frequency of the decomposed word string and generating a vector of the input document according to the word dictionary, and a first vector generating means (22). First similarity calculating means (23) for comparing the generated first vector with the first vector stored in the first vector storage means (31) and calculating the similarity; The document pattern is scanned by a document pattern generation unit (24) that uses the arrangement of words decomposed by the word search unit (21) as a document pattern and a pattern stored in the first pattern dictionary storage unit (34). First pattern retrieval means for decomposing into patterns and word arrays (25)
A second vector generating means (26) for counting a pattern frequency of a pattern and a sequence of words and generating a second vector of the input document based on the first pattern dictionary.
And the second vector generated by the second vector generating means (26).
A second similarity calculating means (27) for comparing the vector of the second vector with the second vector stored in the second vector storing means (33) and calculating the similarity thereof; A second step of scanning the document pattern by the pattern stored in the means (36) and decomposing it into an array of patterns and words;
And a pattern search means (28) for counting the pattern frequency of the pattern and the sequence of words, and calculating the second
A third vector generating means (29) for generating a third vector based on the pattern dictionary of (3), and a third vector generated by the third vector generating means (29) and a third vector storing means (35). A third similarity calculating means (2) for comparing the stored third vector with the third vector and calculating the similarity.
A) and similarity integrating means (20) for integrating the outputs of the first to third similarity calculating means and outputting as one similarity
And

In the embodiment of the present invention, the processing and functions of each means of the data processing device are realized by a program executed by the data processing device. In this case, the present invention can be implemented by loading an executable program from a recording medium on which the program is recorded via a predetermined reading device into a main storage of the data processing device and executing the program.

[0016]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of the present invention; FIG. 1 is a diagram showing the configuration of the first exemplary embodiment of the present invention. Referring to FIG. 1, a first embodiment of the present invention is shown.
The embodiment includes a document input unit 1, a data processing device 2 operated by program control, a storage device 3, and a similarity output unit 4.

Referring to FIG. 1, the data processing device 2
A similarity integration means 20, a word search means 21, a first vector generation means 22, a first similarity calculation means 23,
Document pattern generation means 24, pattern search means 25, second vector generation means 26, second similarity calculation means 2
7 is included.

The storage device 3 comprises a first vector storage means 3
1, a word dictionary storage means 32, a second vector storage means 33, and a pattern dictionary storage means 34.

Each of these means operates as follows.

The document input means 1 inputs a document to the data processing device 2.

The similarity output means 4 outputs the similarity between the stored document and the input document.

The similarity integrating means 20 integrates the output of the first similarity calculating means 23 and the output of the second similarity calculating means 27 and outputs the result as one similarity.

The word search means 21 is a word dictionary storage means 3
2 is decomposed into word strings by the words stored in the second word.

The first vector generation means 22 counts the word frequencies of the decomposed word strings, and generates a vector of the input document based on the word dictionary.

The first similarity calculating means 23 compares the vector generated by the first vector generating means 22 with the vector stored in the first vector storing means 31, and calculates the similarity.

The document pattern generation means 24 uses the arrangement of words decomposed by the word search means 21 as a document pattern.

The pattern search means 25 scans the document pattern by using the pattern stored in the pattern dictionary storage means 34 and decomposes the document pattern into an array of patterns and words.

The second vector generation means 26 counts the pattern frequency of the pattern and the arrangement of words, and generates a vector of the input document by the pattern dictionary.

The second similarity calculation means 27 compares the vector generated by the second vector generation means 26 with the vector stored in the second vector storage means 33, and calculates the similarity.

The first vector storage means 31 stores a vector of the stored document according to the word dictionary.

The word dictionary storage means 32 stores the word search means 2
The word dictionary used in step 1 is stored.

The second vector storage means 33 stores a vector of the stored document according to the pattern dictionary.

The pattern dictionary storage means 34 stores a pattern dictionary used by the pattern search means 25.

FIG. 2 is a flowchart showing a processing procedure according to the first embodiment of the present invention. The overall operation of the first embodiment of the present invention will be described in detail with reference to FIGS.

First, a document is input to the data processing device 2 (step S1 in FIG. 2).

Next, a word search is performed to replace the document with a word string (step S2).

Further, the frequency of words is counted to generate a first vector (step S3).

Further, the first vector is stored between the generated vector and the vector stored in the first vector storage means 31.
Is calculated (step S4).

Next, the word string is regarded as a document pattern (step S5), and a pattern search is performed on the document pattern to replace the document with the pattern string (step S6).

The frequency of the pattern is counted to generate a second vector (step S7).

The second similarity is calculated between the generated vector and the vector stored in the second vector storage means 33 (step S8).

Finally, an integrated similarity of the two obtained similarities is calculated and output (step S9).

Next, the operation and effect of the first embodiment of the present invention will be described.

In the first embodiment of the present invention, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity based on shallow semantic processing using a word dictionary and the similarity based on deeper semantic processing using a pattern dictionary are used. The degree is configured to be used effectively at the same time. Therefore, deeper semantic processing can be performed than in the conventional method.

Next, a first embodiment of the present invention will be described with reference to a specific example.

As shown in FIGS. 6 to 8, it is assumed that a document having the content "Please send material." 6, the word dictionary of FIG. 6 is decomposed into "material", "wo", "send", "do", "please", "ne", ".", And "material", "wo", "Send", "Yes", "Send", "Ne", "." Are converted to normal form

From now on, as the word frequency vector (first vector), ("material" 1, "wo" 1, "send" 1, "do" 1, "give" 1, "ne" 1, "." 1 ) Is obtained. Based on this, the similarity is calculated using a known TF / IDF method or the like.

Next, as shown in the example of the document pattern in FIG. 8, a document (original document) having the content "Please send the material." (The document pattern example 1) is converted into (in this case, the symbol represents the end of the sentence).

When an unnecessary word dictionary that makes unnecessary words such as “do”, “ne”, “.”, Etc. is used, it is also converted into the form “send material * please * ＄” (document pattern example 2). You.

When such a document pattern is used by using a pattern dictionary as shown in FIG. 7, it is converted into a form of "material transmission request". From this, ("material" 1) is converted into a vector (second vector). , "Request to send" 1) is obtained.

As described above, by generating vectors corresponding to a plurality of abstractions, it is possible to select an expression of the abstraction according to the purpose.

As in the previous case, for example, a well-known TF
The similarity is calculated by the IDF method or the like.

The obtained two similarities are converted into one similarity by a simple product or a weighted sum and output.

In the first embodiment of the present invention, the obtained similarities may be sourced in ascending or descending order.

Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 3 is a diagram showing the configuration of the second embodiment of the present invention. Referring to FIG. 3, a second embodiment of the present invention comprises a document input unit 1, a data processing device 2 operated under program control, a storage device 3, and a similarity output unit 4
And

The data processing device 2 includes a similarity integration means 20
, Word search means 21 and first vector generation means 22
A first similarity calculating means 23, a document pattern generating means 24, a first pattern searching means 25, a second vector generating means 26, a second similarity calculating means 27, a second Pattern search means 28 and third vector generation means 29
And third similarity calculating means 2A.

The storage device 3 comprises a first vector storage means 3
1, a word dictionary storage unit 32, a second vector storage unit 33, a first pattern dictionary storage unit 34, a third vector storage unit 35, and a second pattern dictionary storage unit 36.
And

Each of these means has the following functions.

The document input unit 1 inputs a document to the data processing device 2.

The similarity output means 4 outputs the similarity between the stored document and the input document.

The similarity integrating means 20 integrates the output of the first similarity calculating means 23 and the output of the second similarity calculating means 27 and outputs the result as one similarity.

The word search means 21 is used for the word dictionary storage means 3
2 is decomposed into word strings by the words stored in the second word.

The first vector generation means 22 counts the word frequencies of the decomposed word strings, and generates a vector of the input document based on the word dictionary.

The first similarity calculating means 23 compares the vector generated by the first vector generating means 22 with the vector stored in the first vector storing means 31, and calculates the similarity.

The document pattern generation means 24 uses the word array decomposed by the word search means 21 as a document pattern.

The first pattern search means 25 scans the document pattern using the pattern stored in the first pattern dictionary storage means 34 and decomposes the document pattern into an array of patterns and words.

The second vector generation means 26 counts the pattern frequency of the pattern and the arrangement of the words, and generates a vector of the input document by the pattern dictionary.

The second similarity calculating means 27 compares the vector generated by the second vector generating means 26 with the vector stored in the second vector storing means 33, and calculates the similarity.

The second pattern search means 28 scans the document pattern using the pattern stored in the second pattern dictionary storage means 36 and decomposes it into an array of patterns and words.

The third vector generation means 29 counts the pattern frequency of the pattern and the arrangement of words, and generates a vector of the input document by the pattern dictionary.

The third similarity calculating means 2A compares the vector generated by the third vector generating means 29 with the vector stored in the third vector storing means 35, and calculates the similarity.

The first vector storage means 31 stores a vector of the stored document according to the word dictionary.

The word dictionary storage means 32 stores the word search means 2
The word dictionary used in step 1 is stored.

The second vector storage means 33 stores a vector of the stored document according to the pattern dictionary.

The pattern dictionary storage means 34 stores a pattern dictionary used by the pattern search means 25.

Next, FIG. 4 is a flowchart showing a processing procedure of the second embodiment of the present invention. Referring to FIG. 3 and FIG.
The overall operation of the second embodiment of the present invention will be described in detail. Steps S1 to S8 in FIG. 4 are substantially the same as the processing shown in FIG.

First, a document is input to the data processing device (step S1 in FIG. 4).

Next, a word search is performed to replace the document with a word string (step S2).

Further, the frequency of words is counted to generate a first vector (step S3).

Further, between the generated vector and the vector stored in the first vector storage means 31,
Is calculated (step S4).

Next, the word string is regarded as a document pattern (step S5), a pattern search is performed on the document pattern using the first pattern dictionary, and the document is replaced with the pattern string (step S6).

Further, the frequency of the pattern is counted to generate a second vector (step S7).

Further, the second vector between the generated vector and the vector stored in the second vector
Is calculated (step S8).

A pattern search is performed on the document pattern using the second pattern dictionary, and the document is replaced with a pattern sequence (step S10).

Further, the frequency of the pattern is counted to generate a third vector (step S11).

Further, the generated third vector and the third vector
A third similarity calculation is performed between the vector and the vector stored in the vector storage means 35 (step S12).

Finally, the integrated similarity of the two obtained similarities is calculated and output (step S13).

Next, the operation and effect of the second embodiment of the present invention will be described.

In this embodiment, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity obtained by shallow semantic processing using a word dictionary and the similarity obtained by deeper semantic processing using a pattern dictionary can be simultaneously enabled. Since it is configured so that it can be used, deeper semantic processing than before can be performed.

Next, a second embodiment of the present invention will be described with reference to a specific example.

As shown in FIGS. 6 to 8, if a document with the content “Please send the material” is input, “material”, “A”, “Send” ",""","","","","." To the normal form.

[0093] From now on, as word frequency vectors, ("material" 1, "wo" 1, "send" 1, "do" 1, "given"
1, “Ne” 1, “.” 1) are obtained, and based on this, the similarity is calculated by, for example, the known TF / IDF method.

Next, as shown in the example of the document pattern in FIG. 8, a document having the content "Please send the material." Is in the form of "Please send the material." It is converted (however, ＄ is a symbol indicating the end of the sentence).

Further, by using an unnecessary word dictionary that makes unnecessary words such as “do”, “ne” and “.”, It is possible to write in the form of “send material * please * ＄”.

When such a document pattern is used by using a pattern dictionary as shown in FIG. 7, it is converted into a form of "material transmission request", and is converted into a vector ("material").
1, "Request to send" 1) is obtained.

As described above, by generating vectors corresponding to a plurality of abstractions, it is possible to select an expression of the abstraction according to the purpose. Then, as in the previous case, for example, the known TF
The similarity is calculated by the IDF method or the like.

Similarly, when another pattern dictionary is used, another similarity is calculated.

The obtained three similarities are converted into one similarity by a simple product or a weighted sum and output.

In the first embodiment of the present invention, the obtained similarities may be sourced in ascending or descending order.

Next, a third embodiment of the present invention will be described in detail with reference to the drawings. FIG. 5 is a diagram showing the configuration of the third embodiment of the present invention. Referring to FIG. 5, the third embodiment of the present invention includes a recording medium 5 on which a similarity calculation program is recorded. Examples of the medium include a magnetic disk such as an FD (floppy disk), a semiconductor memory, and a CD.
-ROM, DVD (digital versatile disk), MT
Other recording media may be used. Further, the data processing device 2 may implement the present invention by downloading a similarity calculation program from a storage medium of another data processing device such as a server device via a communication unit. The medium 5 includes a communication medium.

The similarity calculation program is read into the data processing device 2 from the recording medium 5 and controls the operation of the computer. The computer performs the following processing under the control of the similarity calculation program, that is, the first embodiment described above,
Or processing by the data processing device 2 in the second example,
That is, the same processing as the processing specified in the flowcharts of FIGS. 2 and 4 is executed.

That is, referring to FIG. 1, a document input means 1, a data processing device 2 operated by program control, a storage device 3, and a similarity output means 4 are provided. Integrating means 20, word searching means 21, first vector generating means 22, first similarity calculating means 23, document pattern generating means 24, pattern searching means 25, and second vector generating means 26 And second similarity calculating means 27.

The storage device 3 comprises a first vector storage means 3
1, a word dictionary storage means 32, a second vector storage means 33, and a pattern dictionary storage means 34.

Alternatively, referring to FIG. 3, a document input means 1, a data processing device 2 operated by program control, a storage device 3, and a similarity output means 4 are provided. Integrating means 20, word searching means 21, first vector generating means 22, first similarity calculating means 23, document pattern generating means 24, first pattern searching means 25, second vector Generating means 26;
Second similarity calculating means 27 and second pattern searching means 2
8, a third vector generating means 29, and a third similarity calculating means 2A.

The storage device 3 comprises a first vector storage means 3
1, a word dictionary storage unit 32, a second vector storage unit 33, a first pattern dictionary storage unit 34, a third vector storage unit 35, and a second pattern dictionary storage unit 36.
And

[0107]

As described above, according to the present invention, the following effects can be obtained.

The first effect of the present invention is that deeper semantic processing can be performed than in the conventional method.

The reason for this is that, in the present invention, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity based on shallow semantic processing using a word dictionary is similar to the similarity based on deeper semantic processing using a pattern dictionary. The reason is that the degree and the time can be used effectively at the same time.

A second effect of the present invention is that handling of compound words and collocation words is facilitated.

The reason for this is that, in the present invention, these can be registered as patterns in the pattern dictionary.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the first embodiment of the present invention.

FIG. 3 is a diagram showing a configuration of a second exemplary embodiment of the present invention.

FIG. 4 is a flowchart showing the operation of the second exemplary embodiment of the present invention.

FIG. 5 is a diagram showing a configuration of a third exemplary embodiment of the present invention.

FIG. 6 is a diagram for explaining an embodiment of the present invention, and is a diagram showing a specific example of a word dictionary.

FIG. 7 is a diagram for explaining an embodiment of the present invention, and is a diagram showing a specific example of a pattern dictionary.

FIG. 8 is a diagram for explaining an embodiment of the present invention, and is a diagram showing a specific example of a document pattern.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Document input means 2 Data processing apparatus 20 Similarity integration means 21 Word search means 22 First vector generation means 23 First similarity calculation means 24 Document pattern generation means 25 First pattern search means 26 Second vector generation Means 27 Second similarity calculating means 28 Second pattern searching means 29 Third vector generating means 2A Third similarity calculating means 3 Storage device 31 First vector storing means 32 Word dictionary storing means 33 Second Vector storage means 34 Pattern dictionary storage means 4 Similarity output means 5 Recording medium

Claims (20)

[Claims] A first similarity calculating means for calculating a first similarity based on a word dictionary; a second similarity calculating means for calculating a second similarity based on a pattern dictionary; And a similarity integrating means for calculating one similarity from the second similarity. 2. A first similarity calculating means for calculating a first similarity based on a word dictionary, a second similarity calculating means for calculating a second similarity based on a first pattern dictionary, A third similarity calculating unit that calculates a third similarity based on the second pattern dictionary; and a similarity integrating unit that calculates one similarity from the first, second, and third similarities. A similarity search system comprising: 3. The similarity calculating means includes a plurality of similarity calculating means based on a plurality of dictionaries, and a means for integrating similarities output from the plurality of similarity calculating means. The similarity search system according to claim 1. 4. The similarity search system according to claim 1, further comprising means for sorting in order of similarity. 5. The first similarity calculating means calculates the first similarity by comparing vectors generated based on word frequency information of a document, and the second similarity calculating means calculates Calculating the second similarity by comparing vectors generated based on the pattern frequency information of the document pattern using the pattern dictionary storing the pattern information for extracting the meaning information of the document pattern. The similarity search system according to claim 1. 6. The first similarity calculating means calculates the first similarity by comparing vectors generated based on word frequency information of a document. Calculating the second similarity by comparing the vectors generated based on the pattern frequency information of the document pattern using the first pattern dictionary storing the pattern information for extracting the semantic information of the The third similarity calculating means compares the vector generated based on the pattern frequency information of the document pattern using the second pattern dictionary storing the pattern information for extracting the semantic information of the document, thereby obtaining a third similarity. The similarity search system according to claim 2, wherein the similarity is calculated. 7. A step of calculating a first similarity based on a word dictionary, a step of calculating a second similarity based on a pattern dictionary, and one similarity based on the first and second similarities. Calculating a degree, comprising: calculating the degree of similarity. 8. A step of calculating a first similarity based on the word dictionary, a step of calculating a second similarity based on the first pattern dictionary, and a step of calculating a third similarity based on the second pattern dictionary. And calculating one similarity from the first to third similarities. A method for calculating a similarity, comprising: 9. The similarity search method according to claim 7, further comprising a step of sorting in order of similarity. 10. A pattern for extracting semantic information is provided as a pattern dictionary, and a first pattern obtained by using a word dictionary is provided.
And outputting a similarity obtained by integrating a second similarity obtained by the semantic processing using the pattern dictionary. 11. A first similarity calculation process for calculating a first similarity based on a word dictionary, a second similarity calculation process for calculating a second similarity based on a pattern dictionary, And a similarity integration process for calculating one similarity from the first and second similarities; and a recording medium storing a program for causing a computer to execute the similarity integration process. 12. A first similarity calculation process for calculating a first similarity based on a word dictionary, and a second similarity calculation process for calculating a second similarity based on a first pattern dictionary A third similarity calculation process for calculating a third similarity based on a second pattern dictionary; a similarity integration process for calculating one similarity from the first to third similarities; Recording a program for causing a computer to execute the program. 13. The recording medium according to claim 11, wherein a program for causing said computer to further execute a process of sorting in order of similarity is recorded. 14. A word search means for decomposing an input document into a word string; a means for decomposing the input document into a word string based on a word dictionary; Means for generating a first vector based on the word dictionary of the input document; comparing the generated first vector with a first vector based on the word dictionary of a pre-stored document;
Means for calculating the degree of similarity of a document, means for using the array of the decomposed words as a document pattern, and scanning a document pattern based on a pattern of a pattern dictionary for extracting semantic information of the document. Means for decomposing into a sequence of patterns, means for counting the pattern frequency of patterns and patterns of words, and means for generating a second vector based on the pattern dictionary of the input document; and the generated second vector, Means for comparing a document stored in advance with a second vector based on the pattern dictionary to calculate a second similarity; and integrating the first similarity and the second similarity into one similarity. Means for outputting as a degree, and a similarity calculation apparatus between documents. 15. A word dictionary storage means for storing a word dictionary, a first vector storage means for storing a first vector of a stored document according to the word dictionary, and a pattern for extracting document semantic information. Pattern dictionary storage means for storing a dictionary; second vector storage means for storing a second vector of the stored document according to the pattern dictionary; document input means for inputting a document; storage in the word dictionary storage means Word search means for decomposing a document input from the document input means into a word string based on the input word information; counting the word frequency of the decomposed word string; and the word dictionary of the input document A first vector generating means for generating a first vector according to the following: and a first vector generated by the first vector generating means and stored in the first vector storing means. The first
A first similarity calculating means for comparing the vector of the word and calculating the similarity thereof; a document pattern generating means for setting an array of words decomposed by the word searching means to a document pattern; and a pattern dictionary storing means. A pattern search unit that scans a document pattern using the stored pattern and decomposes the document pattern into an array of patterns and words; counts a pattern frequency of the array of patterns and words; A second vector generating means for generating a vector of the second vector, and a second vector generated by the second vector generating means and a second vector stored in the second vector storing means.
A second similarity calculating means for comparing the similarity and the similarity outputted from the first similarity calculating means with the similarity outputted from the first similarity calculating means. An inter-document similarity calculating apparatus, comprising: a similarity integrating unit that integrates degrees and outputs a single similarity; and a unit that outputs the similarity output from the similarity integrating unit. 16. A word dictionary storage means for storing a word dictionary; a first vector storage means for storing a first vector of the stored document according to the word dictionary; A first pattern dictionary storage unit for storing one pattern dictionary; a second vector storage unit for storing a second vector of the stored document according to the first pattern dictionary; and extracting semantic information of the document. Pattern dictionary storage means for storing a second pattern dictionary for storing, a third vector storage means for storing a third vector of the stored document according to the second pattern dictionary, and inputting the document Document input means; word search means for decomposing a document input from the document input means into word strings based on word information stored in the word dictionary storage means; A first vector generation unit that counts a word frequency and generates a vector of an input document according to a word dictionary; and a first vector generated by the first vector generation unit and the first vector storage unit. A first similarity calculating unit that compares the stored first vector and calculates a similarity thereof; a document pattern generating unit that uses a sequence of words decomposed by the word searching unit as a document pattern; A first pattern search unit that scans a document pattern based on the pattern stored in the first pattern dictionary storage unit and decomposes the document pattern into an array of patterns and words; and counts a pattern frequency of the array of patterns and words. A second vector generation unit that generates a second vector of the input document based on the first pattern dictionary; and a second vector generated by the second vector generation unit. A second similarity calculating means for comparing the second vector stored in the second vector storing means and calculating a similarity thereof; and a pattern stored in the second pattern dictionary storing means. A second pattern search means for scanning a document pattern and decomposing it into an array of patterns and words; counting a pattern frequency of the array of patterns and words; and obtaining a third pattern of the input document by using the second pattern dictionary. A third vector generating means for generating a vector of the third vector, and comparing a third vector generated by the third vector generating means with a third vector stored in the third vector storing means, Third similarity calculating means for calculating similarity, similarity integrating means for integrating similarities output from the first to third similarity calculating means and outputting the same as one similarity, Kind Means for outputting the similarity output from the similarity integrating means, and an inter-document similarity calculation apparatus. 17. A first step of inputting a document, a second step of performing a word search and replacing the document with a word string, counting the frequency of words, and using a word dictionary for the input document in a first step. A third step of generating a vector, and calculating a first similarity between the generated first vector and a vector based on the word dictionary stored in advance in the first vector storage means. Step 4, a word string is regarded as a document pattern, a pattern search is performed on the document pattern using a pattern dictionary, and the document is replaced with the pattern string. The frequency of the pattern is counted. A sixth step of generating a second vector based on a pattern dictionary for the generated document, and generating the second vector and the pattern stored in a second vector storage means in advance. A seventh step of calculating a second similarity between a vector based on a binary dictionary, and an eighth step of calculating and outputting an integrated similarity of the first similarity and the second similarity A method for calculating the degree of similarity between documents, comprising: 18. A first step of inputting a document, a second step of performing a word search and replacing the document with a word string, counting the frequency of words, and using a word dictionary for the input document in a first step. A third step of generating a vector, and a fourth step of performing a first similarity calculation between the generated first vector and the first vector stored in the first vector storage means in advance. A step of considering the word string as a document pattern, performing a pattern search on the document pattern using the first pattern dictionary, and replacing the document with the pattern string; counting the frequency of the pattern; A sixth step of generating a second vector, and a second step of performing a second similarity calculation between the generated second vector and a second vector stored in the second vector storage means in advance. 7 of An eighth step of performing a pattern search on the document pattern using the second pattern dictionary and replacing the document with a pattern sequence, and a ninth step of counting the frequency of the pattern and generating a third vector A tenth step of performing a third similarity calculation between the generated third vector and a vector previously stored in the third vector storage means; and the first to third similarities And calculating and outputting an integrated similarity of the two documents. 19. Word dictionary storage means for storing a word dictionary; first vector storage means for storing a first vector of a stored document according to the word dictionary; and a pattern for extracting document semantic information. A storage device comprising: a pattern dictionary storage device for storing a dictionary; a second vector storage device for storing a second vector of the stored document according to the pattern dictionary; and an input device for inputting a document. An information processing apparatus comprising: an output device that outputs a degree; and a data processing device that is controlled by a program. (B) counting the frequency of words in the decomposed word string and generating a first vector based on a word dictionary of the input document;
And (c) comparing the first vector generated in the first vector generation process with the first vector stored in the first vector storage means, and calculating the similarity between the first vector and the first vector. A first similarity calculation process for calculating; (d) a document pattern generation process in which an array of words decomposed by the word search process is used as a document pattern; and (e) a pattern stored in the pattern dictionary storage unit. And (f) counting the pattern frequency of the pattern and word sequence by scanning the document pattern and decomposing it into a pattern and word sequence.
A second vector generation process for generating a second vector of the input document based on the pattern dictionary; and (g) storing the second vector generated in the second vector generation process and the second vector. A second similarity calculation process for comparing the second vector stored in the means and calculating the similarity, and (h) a similarity output from the first similarity calculation and the second similarity calculation. A similarity integration process of integrating similarities output from the similarity calculation process and outputting the same as one similarity; and causing the data processing device to execute each of the processes (a) to (h). A recording medium on which a program is recorded. 20. A word dictionary storage means for storing a word dictionary; a first vector storage means for storing a first vector of the stored document according to the word dictionary; and a second vector storage means for extracting semantic information of the document. A first pattern dictionary storage unit for storing one pattern dictionary; a second vector storage unit for storing a second vector of the stored document according to the first pattern dictionary; and extracting semantic information of the document. A second pattern dictionary storing means for storing a second pattern dictionary for storing a second vector, and a third vector storing means for storing a third vector of the stored document according to the second pattern dictionary. An information processing apparatus comprising: a device; an input device for inputting a document; an output device for outputting a degree of similarity; and a data processing device controlled by a program. A word search process for decomposing the input document into word strings based on the stored word information; and (b) counting the word frequencies of the decomposed word strings, and using the word dictionary of the input document as a vector. And (c) comparing the first vector generated in the first vector generation process with the first vector stored in the first vector storage means. A first similarity calculation process for calculating the similarity, (d) a document pattern generation process for using a sequence of words decomposed by the word search process as a document pattern, and (e) the first pattern dictionary. A first pattern search process for scanning a document pattern using the pattern stored in the storage means and decomposing the document pattern into an array of patterns and words; (f) counting a pattern frequency of the array of patterns and words;
A second of the input document according to the first pattern dictionary
(G) a second vector generation process for generating the vector of the second vector generation process, and a second vector generated in the second vector generation process and a second vector stored in the second vector storage unit. (H) scanning the document pattern by using the pattern stored in the second pattern dictionary storage means and decomposing the document pattern into an array of patterns and words; And (i) counting the pattern frequency of the pattern and the sequence of words,
A third of the input document according to the second pattern dictionary
A third vector generation process for generating a vector of the third vector, and (j) a third vector generated in the third vector generation process and a third vector stored in the third vector storage unit. A third similarity calculation process for comparing and calculating the similarity; and (k) integrating the similarities output from the first to third similarity calculation processes and outputting as one similarity. A recording medium on which a program for causing the data processing device to execute each of the above-described processes (a) to (k) is described.