JP3690216B2 - Document similarity calculation method, system and apparatus, and recording medium recording similarity calculation program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a similarity calculation technique, and more particularly to a similar document search system, a similar document search method, and a recording medium on which a similar document search program is recorded that is preferably used for searching and classification.
[0002]
[Prior art]
As a conventional similar document retrieval method in the field of information retrieval, for example, as described in Reference 1 (G. Salton, M. McGill, Introduction to Modern Information Retrieval, New York, McGraw-Hill, 1983) A method of calculating the distance or similarity of documents based on the appearance frequency of words is known. In this conventional similar document search method, a word frequency vector in each document is obtained, each vector is subjected to weighting called TF / IDF, and a cosine value (cos θ = (x · y) between two vectors is formed. ) / | X || y |, where (x · y) is the inner product of two vectors x and y, and | x | and | y | are absolute values). In the TF / IDF method, the importance of a word is defined using the appearance frequency tf and the reciprocal idf of the number of occurrences df, and the importance (weight) W (tk, Ci) of the word tk in the category Ci is W (tk, Ci). tk, Ci) = tf (tk, Ci) log (Li / df (tk, Ci) +1), where tf (tk, Ci) is the frequency of occurrence of the word tk in category Ci, df (tk, Ci) Represents the number of occurrences of the word tk in category Ci, and Li represents the total number of texts in category Ci.
[0003]
Also, for example, in Japanese Patent Application Laid-Open No. 11-134359, etc., when calculating the similarity between two documents, a word is extracted by performing morphological analysis of the two sets of input documents and the summary, and the words included in the summary are extracted. A method has been proposed in which the similarity between two documents is calculated based on the words included in each document.
[0004]
[Problems to be solved by the invention]
However, the conventional method described above has the following problems.
[0005]
The first problem is that the accuracy of similarity between documents is low.
[0006]
This is because only shallow semantic processing based on word vectors is performed.
[0007]
The second problem is that it is difficult to handle compound words.
[0008]
The reason is that the document is expressed by one vector.
[0009]
Accordingly, the present invention has been made in view of the above problems, and an object thereof is to provide a similar document search system and method, and a recording medium, which can perform deeper semantic processing.
[0010]
Another object of the present invention is to provide a similar document search system and method, and a recording medium that can easily handle compound words and collocations.
[0011]
[Means for Solving the Problems]
The similarity search system of the present invention that achieves the above object includes a first similarity calculation means based on a word dictionary, a second similarity calculation means based on a pattern dictionary, and one similarity calculated from two similarities. Means.
[0012]
The present invention also includes a first similarity calculation unit based on a word dictionary, a second similarity calculation unit based on a first pattern dictionary, a third similarity calculation unit based on a second pattern dictionary, Means for calculating one similarity from the three similarities.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described. In a preferred embodiment of the apparatus of the present invention, referring to FIG. 1, a first vector storage means (31) for storing a first vector according to a word dictionary of a stored document and a word dictionary are stored. A word dictionary storage means (32) for storing, a second vector storage means (33) for storing a second vector based on a pattern dictionary of the stored document, a pattern dictionary storage means (34) for storing a pattern dictionary, The storage device (3) includes a document input means (1) for inputting a document, and the data processing device converts the input document into a word string using words stored in the word dictionary storage means (34). A word search means (21) for decomposing, a first vector generating means (22) for counting the word frequency of the decomposed word string, and generating a first vector based on the word dictionary of the input document; 1's A first similarity calculation for comparing the first vector generated by the vector generation means (22) and the first vector stored in the first vector storage means (31) and calculating the similarity. A document pattern generation means (24) having a word pattern decomposed by the word search means (21) as a document pattern, and a pattern stored in the pattern dictionary storage means (34). A pattern search means (25) that scans the pattern and decomposes it into a pattern and word arrangement, counts the pattern frequency of the pattern and word arrangement, and generates a second vector by the pattern dictionary of the input document. Second vector generation means (26), second vector generated by second vector generation means (26), and second vector stored in second vector storage means (33) , The second similarity calculation means (27) for calculating the similarity, and the output of the first similarity calculation means (23) and the output of the second similarity calculation means (27) are integrated. And a similarity integrating means (20) for outputting as one similarity, and further comprising means (4) for outputting the similarity output from the similarity integrating means (20).
[0014]
In another embodiment of the present invention, referring to FIG. 3, the storage device (3) includes a first vector storage means (31) for storing a first vector according to a word dictionary of a stored document; Word dictionary storage means (32) for storing a word dictionary, second vector storage means (33) for storing a second vector based on a pattern dictionary of the stored document, and a pattern dictionary for storing the first pattern dictionary Storage means (34), third vector storage means (35) for storing a third vector in the second pattern dictionary of the stored document, and second pattern dictionary storage for storing the second pattern dictionary Means (36), wherein the data processing device uses the words stored in the word dictionary storage means to decompose the input document into word strings, and the decomposed word strings Word frequency The first vector generating means (22) for counting and generating a vector based on the word dictionary of the input document, the first vector generated by the first vector generating means (22), and the first vector A first similarity calculation means (23) for comparing the first vector stored in the storage means (31) and calculating the similarity, and an array of words decomposed by the word search means (21) A first pattern search that scans a document pattern by a pattern stored in the document pattern generation means (24) having the document pattern as a document pattern and the first pattern dictionary storage means (34), and decomposes the pattern into a pattern of words and words. Means (25); second vector generation means (26) for counting pattern frequencies of patterns and word arrangements and generating a second vector by the first pattern dictionary of the input document; A second vector generated by the second vector generation means (26) and the second vector stored in the second vector storage means (33), and the similarity is calculated. Degree calculation means (27), second pattern search means (28) that scans the document pattern using the patterns stored in the second pattern dictionary storage means (36), and decomposes the document pattern into patterns and word arrays; A third vector generating means (29) for counting the pattern frequency of the pattern and word arrangement and generating a third vector by the second pattern dictionary of the input document; and a third vector generating means ( A third similarity calculation means (2A) for comparing the third vector generated in 29) with the third vector stored in the third vector storage means (35) and calculating the similarity; , First to third similarities Similarity integration means (20) for integrating the outputs of the degree calculation means and outputting them as one similarity.
[0015]
In the embodiment of the present invention, each means included in the data processing device has its processing / function realized by a program executed by the data processing device. In this case, the present invention can be implemented by loading and executing a program in an execution format from the recording medium on which the program is recorded via a predetermined reading device to the main memory of the data processing device.
[0016]
【Example】
In order to describe the above-described embodiment of the present invention in more detail, examples of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing the configuration of the first exemplary embodiment of the present invention. Referring to FIG. 1, the first embodiment of the present invention includes a document input unit 1, a data processing device 2 that operates under program control, a storage device 3, and a similarity output unit 4.
[0017]
Referring to FIG. 1, the data processing apparatus 2 includes a similarity integration unit 20, a word search unit 21, a first vector generation unit 22, a first similarity calculation unit 23, and a document pattern generation unit 24. Pattern search means 25, second vector generation means 26, and second similarity calculation means 27.
[0018]
The storage device 3 includes first vector storage means 31, word dictionary storage means 32, second vector storage means 33, and pattern dictionary storage means 34.
[0019]
Each of these means generally operates as follows.
[0020]
The document input unit 1 inputs a document to the data processing device 2.
[0021]
The similarity output unit 4 outputs the similarity between the stored document and the input document.
[0022]
The similarity integration unit 20 integrates the output of the first similarity calculation unit 23 and the output of the second similarity calculation unit 27 and outputs it as one similarity.
[0023]
The word search means 21 decomposes the document input by the words stored in the word dictionary storage means 32 into word strings.
[0024]
The first vector generation means 22 counts the word frequency of the decomposed word string, and generates a vector based on the word dictionary of the input document.
[0025]
The first similarity calculation unit 23 compares the vector generated by the first vector generation unit 22 with the vector stored in the first vector storage unit 31 and calculates the similarity.
[0026]
The document pattern generation unit 24 uses the word sequence decomposed by the word search unit 21 as a document pattern.
[0027]
The pattern search means 25 scans the document pattern based on the patterns stored in the pattern dictionary storage means 34 and decomposes it into patterns and word arrays.
[0028]
The second vector generation means 26 counts the pattern frequency of the pattern and word arrangement, and generates a vector based on the pattern dictionary of the input document.
[0029]
The second similarity calculation means 27 compares the vector generated by the second vector generation means 26 with the vector stored in the second vector storage means 33, and calculates the similarity.
[0030]
The first vector storage means 31 stores a vector of a stored document according to a word dictionary.
[0031]
The word dictionary storage unit 32 stores a word dictionary used by the word search unit 21.
[0032]
The second vector storage means 33 stores a vector of a stored document according to a pattern dictionary.
[0033]
The pattern dictionary storage unit 34 stores a pattern dictionary used by the pattern search unit 25.
[0034]
Next, FIG. 2 is a flowchart showing the processing procedure of the first embodiment of the present invention. The overall operation of the first exemplary embodiment of the present invention will be described in detail with reference to FIGS.
[0035]
First, a document is input to the data processing device 2 (step S1 in FIG. 2).
[0036]
Next, a word search is performed and the document is replaced with a word string (step S2).
[0037]
Further, the frequency of words is counted to generate a first vector (step S3).
[0038]
Further, a first similarity is calculated between the generated vector and the vector stored in the first vector storage means 31 (step S4).
[0039]
Next, the word string is regarded as a document pattern (step S5), a pattern search is performed on the document pattern, and the document is replaced with the pattern string (step S6).
[0040]
The frequency of the pattern is counted to generate a second vector (step S7).
[0041]
A second similarity is calculated between the generated vector and the vector stored in the second vector storage means 33 (step S8).
[0042]
Finally, the integrated similarity of the two obtained similarities is calculated and output (step S9).
[0043]
Next, the function and effect of the first embodiment of the present invention will be described.
[0044]
In the first embodiment of the present invention, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity by shallow semantic processing using a word dictionary and the similarity by deeper semantic processing using a pattern dictionary are simultaneously displayed. It is configured to be used effectively. For this reason, semantic processing deeper than the conventional method can be performed.
[0045]
Next, the first embodiment of the present invention will be described based on a specific example.
[0046]
As shown in FIG. 6 to FIG. 8, it is assumed that a document with the content “Please send us materials” is input. The word dictionary of FIG.
"Document", "O", "Send", "Do", "Please", "Ne", "."
And then,
"Document", "O", "Send", "Yes", "Give me", "Ne", "."
And is converted to normal form.
[0047]
From now on, as the word frequency vector (first vector),
("Document" 1, "O" 1, "Send" 1, "Yes" 1, "Please" 1, "Ne" 1, "." 1)
Is obtained. Based on this, the similarity is calculated using a known TF / IDF method or the like.
[0048]
Next, as shown in the example document pattern in FIG.
For example, a document (original document) with the content “Please send us your materials.”
"Please send us your materials. $"
(Document pattern example 1) (where $ is a symbol representing the end of a sentence).
[0049]
Also, if you use an unnecessary word dictionary with unnecessary words such as “Yes”, “Ne”, “.”,
“Send materials ** $”
Is also converted into a form (document pattern example 2).
[0050]
If a pattern dictionary such as that shown in FIG. 7 is used for such a document pattern, the document pattern is converted into a form “request for material delivery”. From now on, as a vector (second vector)
("Document" 1, "Request to send" 1)
Is obtained.
[0051]
In this way, by generating a vector corresponding to a plurality of abstraction levels, it is possible to select an expression of abstraction level corresponding to the purpose.
[0052]
Similarly to the previous time, the similarity is calculated by a known TF / IDF method, for example.
[0053]
The obtained two similarities are converted into one similarity by a simple product or a weighted sum and output.
[0054]
In the first embodiment of the present invention, the obtained similarities may be sourced in ascending order or descending order.
[0055]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.
[0056]
FIG. 3 is a diagram showing the configuration of the second exemplary embodiment of the present invention. Referring to FIG. 3, the second embodiment of the present invention comprises a document input means 1, a data processing device 2 operated by program control, a storage device 3, and a similarity output means 4.
[0057]
The data processing apparatus 2 includes a similarity integration unit 20, a word search unit 21, a first vector generation unit 22, a first similarity calculation unit 23, a document pattern generation unit 24, and a first pattern search. Means 25, second vector generation means 26, second similarity calculation means 27, second pattern search means 28, third vector generation means 29, third similarity calculation means 2A, including.
[0058]
The storage device 3 includes a first vector storage unit 31, a word dictionary storage unit 32, a second vector storage unit 33, a first pattern dictionary storage unit 34, a third vector storage unit 35, 2 pattern dictionary storage means 36.
[0059]
Each of these means generally has the following functions.
[0060]
The document input unit 1 inputs a document to the data processing device 2.
[0061]
The similarity output unit 4 outputs the similarity between the stored document and the input document.
[0062]
The similarity integration unit 20 integrates the output of the first similarity calculation unit 23 and the output of the second similarity calculation unit 27 and outputs it as one similarity.
[0063]
The word search means 21 decomposes the document input by the words stored in the word dictionary storage means 32 into word strings.
[0064]
The first vector generation means 22 counts the word frequency of the decomposed word string, and generates a vector based on the word dictionary of the input document.
[0065]
The first similarity calculation unit 23 compares the vector generated by the first vector generation unit 22 with the vector stored in the first vector storage unit 31 and calculates the similarity.
[0066]
The document pattern generation unit 24 uses the word sequence decomposed by the word search unit 21 as a document pattern.
[0067]
The first pattern search means 25 scans the document pattern with the patterns stored in the first pattern dictionary storage means 34 and decomposes it into patterns and word arrays.
[0068]
The second vector generation means 26 counts the pattern frequency of the pattern and word arrangement, and generates a vector based on the pattern dictionary of the input document.
[0069]
The second similarity calculation means 27 compares the vector generated by the second vector generation means 26 with the vector stored in the second vector storage means 33, and calculates the similarity.
[0070]
The second pattern search means 28 scans the document pattern based on the pattern stored in the second pattern dictionary storage means 36, and decomposes it into an arrangement of patterns and words.
[0071]
The third vector generation means 29 counts the pattern frequency of the pattern and word arrangement, and generates a vector based on the pattern dictionary of the input document.
[0072]
The third similarity calculation means 2A compares the vector generated by the third vector generation means 29 with the vector stored in the third vector storage means 35, and calculates the similarity.
[0073]
The first vector storage means 31 stores a vector of a stored document according to a word dictionary.
[0074]
The word dictionary storage unit 32 stores a word dictionary used by the word search unit 21.
[0075]
The second vector storage means 33 stores a vector of a stored document according to a pattern dictionary.
[0076]
The pattern dictionary storage unit 34 stores a pattern dictionary used by the pattern search unit 25.
[0077]
Next, FIG. 4 is a flowchart showing the processing procedure of the second embodiment of the present invention. The overall operation of the second exemplary embodiment of the present invention will be described in detail with reference to FIGS. Note that steps S1 to S8 in FIG. 4 are substantially the same as the processing shown in FIG.
[0078]
First, a document is input to the data processing apparatus (step S1 in FIG. 4).
[0079]
Next, a word search is performed and the document is replaced with a word string (step S2).
[0080]
Further, the frequency of words is counted to generate a first vector (step S3).
[0081]
Further, a first similarity calculation is performed between the generated vector and the vector stored in the first vector storage unit 31 (step S4).
[0082]
Next, the word string is regarded as a document pattern (step S5), a pattern search is performed on the document pattern using the first pattern dictionary, and the document is replaced with the pattern string (step S6).
[0083]
Further, the frequency of the pattern is counted to generate a second vector (step S7).
[0084]
Further, a second similarity calculation is performed between the generated vector and the vector stored in the second vector storage means 33 (step S8).
[0085]
A pattern search is performed on the document pattern using the second pattern dictionary, and the document is replaced with a pattern string (step S10).
[0086]
Further, the pattern frequency is counted to generate a third vector (step S11).
[0087]
Further, a third similarity calculation is performed between the generated third vector and the vector stored in the third vector storage means 35 (step S12).
[0088]
Finally, the integrated similarity of the two obtained similarities is calculated and output (step S13).
[0089]
Next, the function and effect of the second embodiment of the present invention will be described.
[0090]
In this embodiment, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity by shallow semantic processing using a word dictionary and the similarity by deeper semantic processing using a pattern dictionary can be used effectively at the same time. Therefore, deeper semantic processing can be performed.
[0091]
Next, a second embodiment of the present invention will be described based on a specific example.
[0092]
As shown in FIG. 6 to FIG. 8, if a document with the content “Please send materials” is input, the word dictionary of FIG.
"Document", "O", "Send", "Do", "Please", "Ne", "."
And then,
"Document", "O", "Send", "Yes", "Give me", "Ne", "."
And is converted to normal form.
[0093]
From now on, as the word frequency vector,
("Document" 1, "O" 1, "Send" 1, "Yes" 1, "Please" 1, "Ne" 1, "." 1)
Based on this, the similarity is calculated by, for example, a known TF / IDF method.
[0094]
Next, as shown in the example of the document pattern in FIG. 8, a document with the content “Please send materials” is, for example,
"Please send me the materials. $"
(Where $ is a symbol representing the end of a sentence).
[0095]
Also, if you use an unnecessary word dictionary with unnecessary words such as “Yes”, “Ne”, “.”, Etc.,
“Send materials ** $”
It can also be written in the form of
[0096]
For such a document pattern, if a pattern dictionary as shown in FIG. 7 is used,
It will be converted into the form of “Request to send materials” and from now on as a vector,
("Document" 1, "Request to send" 1)
Is obtained.
[0097]
Thus, by generating a vector corresponding to a plurality of abstraction levels, an expression of abstraction level corresponding to the purpose can be selected. Similar to the previous time, the similarity is calculated by, for example, a known TF / IDF method.
[0098]
Similarly, when another pattern dictionary is used, another similarity is calculated.
[0099]
The obtained three similarities are converted into one similarity by a simple product or a weighted sum and output.
[0100]
In the first embodiment of the present invention, the obtained similarities may be sourced in ascending order or descending order.
[0101]
Next, a third embodiment of the present invention will be described in detail with reference to the drawings. FIG. 5 is a diagram showing the configuration of the third exemplary embodiment of the present invention. Referring to FIG. 5, the third embodiment of the present invention includes a recording medium 5 on which a similarity calculation program is recorded. The medium may be a magnetic disk such as an FD ( flexible disk), a semiconductor memory, a CD-ROM, a DVD (digital versatile disk), an MT, or other recording medium. Further, the present invention may be implemented by the data processing device 2 downloading a similarity calculation program from a storage medium of another data processing device such as a server device via a communication means.
[0102]
The similarity calculation program is read from the recording medium 5 into the data processing device 2 and controls the operation of the computer. The computer controls the following processing under the control of the similarity calculation program, that is, the processing by the data processing device 2 in the first embodiment or the second example, that is, the processing defined by the flowcharts of FIGS. Execute the same process.
[0103]
That is, referring to FIG. 1, a document input unit 1, a data processing device 2 that operates under program control, a storage device 3, and a similarity output unit 4 are provided. The data processing device 2 includes a similarity integration unit 20. A word search means 21, a first vector generation means 22, a first similarity calculation means 23, a document pattern generation means 24, a pattern search means 25, a second vector generation means 26, 2 similarity calculation means 27.
[0104]
The storage device 3 includes first vector storage means 31, word dictionary storage means 32, second vector storage means 33, and pattern dictionary storage means 34.
[0105]
Alternatively, referring to FIG. 3, it includes a document input unit 1, a data processing device 2 that operates under program control, a storage device 3, and a similarity output unit 4, and the data processing device 2 includes a similarity integration unit 20. The word search means 21, the first vector generation means 22, the first similarity calculation means 23, the document pattern generation means 24, the first pattern search means 25, and the second vector generation means 26. And second similarity calculation means 27, second pattern search means 28, third vector generation means 29, and third similarity calculation means 2A.
[0106]
The storage device 3 includes a first vector storage unit 31, a word dictionary storage unit 32, a second vector storage unit 33, a first pattern dictionary storage unit 34, a third vector storage unit 35, 2 pattern dictionary storage means 36.
[0107]
【The invention's effect】
As described above, the present invention has the following effects.
[0108]
The first effect of the present invention is that semantic processing deeper than the conventional method can be performed.
[0109]
The reason for this is that in the present invention, a pattern for extracting semantic information is provided as a pattern dictionary, and the similarity based on shallow semantic processing using a word dictionary and the similarity based on deeper semantic processing using a pattern dictionary. This is because it can be used effectively at the same time.
[0110]
The second effect of the present invention is to facilitate the handling of compound words and collocations.
[0111]
This is because in the present invention, these can be registered as patterns in the pattern dictionary.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a first exemplary embodiment of the present invention.
FIG. 2 is a flowchart showing the operation of the first exemplary embodiment of the present invention.
FIG. 3 is a diagram showing a configuration of a second exemplary embodiment of the present invention.
FIG. 4 is a flowchart showing the operation of the second exemplary embodiment of the present invention.
FIG. 5 is a diagram showing a configuration of a third exemplary embodiment of the present invention.
FIG. 6 is a diagram for explaining an embodiment of the present invention, and is a diagram showing a specific example of a word dictionary.
FIG. 7 is a diagram for explaining an example of the present invention, and is a diagram showing a specific example of a pattern dictionary.
FIG. 8 is a diagram for explaining an embodiment of the present invention, and is a diagram showing a specific example of a document pattern.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Document input means 2 Data processing apparatus 20 Similarity integration means 21 Word search means 22 First vector generation means 23 First similarity calculation means 24 Document pattern generation means 25 First pattern search means 26 Second vector generation Means 27 Second similarity calculation means 28 Second pattern search means 29 Third vector generation means 2A Third similarity calculation means 3 Storage device 31 First vector storage means 32 Word dictionary storage means 33 Second Vector storage means 34 Pattern dictionary storage means 4 Similarity output means 5 Recording medium

Claims (7)

Word search means for decomposing the input document into word strings;
Means for decomposing the input document into word strings based on a word dictionary;
Means for counting a word frequency of a word string and generating a vector of word frequencies (referred to as a “first vector”) based on the word dictionary of the input document;
Means for comparing the generated first vector with a first vector of the word dictionary of a document stored in advance to calculate a first similarity;
Means for using the decomposed word array as a document pattern;
Means for scanning the document pattern based on the pattern dictionary pattern for extracting the semantic information of the document, and decomposing the pattern into word and word sequences;
Means for counting a pattern frequency of a pattern and a word sequence, and generating a vector of pattern and word sequence pattern frequencies (referred to as a “second vector”) according to the pattern dictionary of the input document;
Means for comparing the generated second vector with a second vector stored in advance in the pattern dictionary of the document and calculating a second similarity;
Means for integrating the first similarity and the second similarity by a simple product or a weighted sum and outputting as one similarity;
An inter-document similarity calculation device characterized by comprising: Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
Pattern dictionary storage means for storing a pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
A document input means for inputting a document;
Word search means for decomposing a document input from the document input means into a word string based on word information stored in the word dictionary storage means;
First vector generation means for counting word frequencies of the decomposed word string and generating a first vector of the input document;
First similarity calculation means for comparing the first vector generated by the first vector generation means with the first vector stored in the first vector storage means and calculating the similarity. When,
A document pattern generation unit that uses an array of words decomposed by the word search unit as a document pattern;
A pattern search unit that scans a document pattern according to a pattern stored in the pattern dictionary storage unit and decomposes it into a pattern and a word arrangement;
Second vector generation means for counting pattern frequencies of patterns and word sequences and generating a second vector of the input document;
Second similarity calculation means for comparing the second vector generated by the second vector generation means with the second vector stored in the second vector storage means and calculating the similarity When,
Similarity integration means for integrating the similarity output from the first similarity calculation means and the similarity output from the second similarity calculation means by a simple product or a weighted sum and outputting it as one similarity When,
Means for outputting the similarity output from the similarity integration means;
An inter-document similarity calculation device characterized by comprising: Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
First pattern dictionary storage means for storing a first pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the first pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
Second pattern dictionary storage means for storing a second pattern dictionary for extracting semantic information of the document;
Third vector storage means for storing a frequency vector (referred to as “third vector”) of a pattern of the stored document according to the second pattern dictionary;
A document input means for inputting a document;
Word search means for decomposing a document input from the document input means into a word string based on word information stored in the word dictionary storage means;
First vector generation means for counting the word frequency of the decomposed word string and generating a first vector by the word dictionary of the input document;
First similarity calculation means for comparing the first vector generated by the first vector generation means with the first vector stored in the first vector storage means and calculating the similarity; ,
A document pattern generation unit that uses an array of words decomposed by the word search unit as a document pattern;
First pattern search means for scanning a document pattern according to a pattern stored in the first pattern dictionary storage means and for decomposing the pattern into a pattern and a word arrangement;
Second vector generation means for counting pattern frequencies of patterns and word arrangements and generating a second vector of the input document by the first pattern dictionary;
Second similarity calculation means for comparing the second vector generated by the second vector generation means with the second vector stored in the second vector storage means and calculating the similarity; ,
A second pattern search unit that scans a document pattern according to a pattern stored in the second pattern dictionary storage unit and decomposes the document pattern into a pattern and a word arrangement;
A third vector generating means for counting a pattern frequency of patterns and word arrangements and generating a third vector by the second pattern dictionary of the input document;
Third similarity calculation means for comparing the third vector generated by the third vector generation means with the third vector stored in the third vector storage means and calculating the similarity; ,
Similarity integration means for integrating the similarities respectively output from the first to third similarity calculation means by a simple product or a weighted sum and outputting them as one similarity;
Means for outputting the similarity output from the similarity integration means;
An inter-document similarity calculation device characterized by comprising: Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
Pattern dictionary storage means for storing a pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
A method for calculating similarity between documents by an information processing apparatus including:
A first step of inputting a document from the document input means;
A second step of searching word information stored in the word dictionary storage means and replacing the document input from the document input means with a word string;
A third step of counting word frequencies and generating a first vector by a word dictionary for the input document;
Calculation of the first similarity between the generated first vector and the first vector stored in advance in the first vector storage means and read out from the first vector storage means A fourth step of performing
A fifth step that regards a word string as a document pattern, performs a pattern search on the document pattern using the pattern dictionary stored in the pattern dictionary storage unit, and replaces the document with a pattern string;
A sixth step of counting a pattern frequency and generating a second vector by a pattern dictionary for the input document;
A second similarity between the generated second vector and the second vector stored in advance in the second vector storage means and read from the second vector storage means by the pattern dictionary; A seventh step of calculating the degree;
An eighth step of calculating and outputting an integrated similarity of the first similarity and the second similarity by a simple product or a weighted sum ;
A method for calculating similarity between documents, including: Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
Pattern dictionary storage means for storing a pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
Second pattern dictionary storage means for storing a second pattern dictionary for extracting semantic information of the document;
Third vector storage means for storing a frequency vector (referred to as “third vector”) of a pattern of the stored document according to the second pattern dictionary;
A method for calculating similarity between documents by an information processing apparatus including:
A first step of inputting a document from the document input means;
A second step of searching word information stored in the word dictionary storage means and replacing the document input from the document input means with a word string;
A third step of counting word frequencies and generating a first vector by a word dictionary for the input document;
A first similarity calculation is performed between the generated first vector and the first vector stored in advance in the first vector storage means and read out from the first vector storage means. A fourth step to perform;
A word string is regarded as a document pattern, a pattern search is performed on the document pattern using the first pattern dictionary stored in advance in the first pattern dictionary storage means, and the document is replaced with a pattern string. 5 steps,
A sixth step of counting a pattern frequency and generating a second vector;
A second similarity calculation is performed between the generated second vector and the second vector stored in advance in the second vector storage means and read out from the second vector storage means. A seventh step to perform;
An eighth step of performing a pattern search on the document pattern using the second pattern dictionary and replacing the document with a pattern sequence;
A ninth step of counting the frequency of patterns and generating a third vector by the second pattern dictionary;
A third similarity calculation is performed between the generated third vector and the third vector stored in advance in the third vector storage means and read out from the third vector storage means. A tenth step;
An eleventh step of calculating and outputting an integrated similarity of the first to third similarities by a simple product or a weighted sum ;
A method for calculating similarity between documents, including: Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
Pattern dictionary storage means for storing a pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
A storage device comprising:
An input device for inputting a document;
An output device for outputting the similarity,
In an information processing device comprising a program-controlled data processing device,
(A) a word search process for decomposing an input document into a word string based on word information stored in the word dictionary storage means;
(B) counting a word frequency of the decomposed word string, and generating a first vector based on a word dictionary of the input document;
(C) A first similarity for comparing the first vector generated in the first vector generation process with the first vector stored in the first vector storage means and calculating the similarity Degree calculation processing,
(D) a document pattern generation process in which an array of words decomposed by the word search process is used as a document pattern;
(E) a pattern search process in which a document pattern is scanned by a pattern stored in the pattern dictionary storage means and decomposed into a pattern and a word arrangement;
(F) a second vector generation process for counting pattern frequencies of patterns and word sequences and generating a second vector by the pattern dictionary of the input document;
(G) a second similarity degree for comparing the second vector generated in the second vector generation process with the second vector stored in the second vector storage means and calculating the degree of similarity Calculation processing,
(H) Similarity in which the similarity output from the first similarity calculation process and the similarity output from the second similarity calculation process are integrated by a simple product or weighted sum and output as one similarity Degree integration processing,
A recording medium on which a program for causing the data processing apparatus to execute each of the processes (a) to (h) is recorded. Word dictionary storage means for storing a word dictionary;
First vector storage means for storing a vector of word frequencies (referred to as “first vector”) according to the word dictionary of a stored document;
Pattern dictionary storage means for storing a pattern dictionary for extracting semantic information of a document;
Second vector storage means for storing a pattern of the stored document according to the pattern dictionary and a vector of word pattern frequency (referred to as “second vector”);
Second pattern dictionary storage means for storing a second pattern dictionary for extracting semantic information of the document;
Third vector storage means for storing a frequency vector (referred to as “third vector”) of a pattern of the stored document according to the second pattern dictionary;
A storage device comprising:
An input device for inputting a document;
An output device for outputting the similarity,
In an information processing device comprising a program-controlled data processing device,
(A) a word search process for decomposing an input document into word strings based on word information stored in the word dictionary storage means;
(B) counting a word frequency of the decomposed word string and generating a vector based on a word dictionary of the input document;
(C) A first similarity for comparing the first vector generated in the first vector generation process with the first vector stored in the first vector storage means and calculating the similarity Degree calculation processing,
(D) a document pattern generation process in which an array of words decomposed by the word search process is used as a document pattern;
(E) a first pattern search process that scans a document pattern with a pattern stored in the first pattern dictionary storage means and decomposes the document pattern into a pattern and a word arrangement;
(F) a second vector generation process for counting a pattern frequency of patterns and word sequences and generating a second vector by the first pattern dictionary of the input document;
(G) a second similarity in which the second vector generated in the second vector generation process is compared with the second vector stored in the second vector storage means, and the similarity is calculated. Degree calculation processing,
(H) a second pattern search process in which a document pattern is scanned according to a pattern stored in the second pattern dictionary storage means and decomposed into a pattern and a word arrangement;
(I) a third vector generation process for counting pattern frequencies of patterns and word arrangements and generating a third vector by the second pattern dictionary of the input document;
(J) Third similarity for comparing the third vector generated by the third vector generation processing with the third vector stored in the third vector storage means and calculating the similarity Degree calculation processing,
(K) Similarity integration processing for integrating the similarities respectively output from the first to third similarity calculation processing and outputting as a single similarity by a simple product or a weighted sum ;
A recording medium on which a program for causing the data processing apparatus to execute each of the processes (a) to (k) is recorded.

Images