JP2001067378A - Calculation method and device for similarity of character string and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate the similarity of character strings with emphasis put on words and to retrieve a document without analyzing a morpheme. SOLUTION: In this calculation method of similarity of character strings, an input character string and a document of a document data base are defined as two character strings and the similarity of both character strings is calculated by a similarity calculation part 14. A coincident character string similarity calculation part 21 of the part 14 calculates the character string score to a partial character string that is common to both character strings and adds this score to the similarity of the remaining partial strings. An optional character string similarity calculation part 22 shifts the correspondence relation of both character strings to calculates a larger degree of similarity, and a maximum value selection part 23 selects the larger degree of similarity. These processes are repeated to totalize the score of partial character strings adaptive to the sequences of two character strings, i.e., the partial character strings which are common to each other and to calculate the final similarity. A retrieval result output part 13 selects a document of a high degree of similarity out of a document data base as a retrieval result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the field of information retrieval, and is particularly suitable for use in determining similarity between an input character string and a document registered in a database.

[0002]

2. Description of the Related Art Information retrieval for extracting a desired document (document) from a database of documents expressed using words is often performed. In such information retrieval, words are treated as character strings obtained by combining words composed of a plurality of characters. Then, the character strings are compared with each other to calculate the similarity, thereby obtaining the degree of conformity with the desired content, or selecting one character string having the highest similarity or selecting some character strings having the highest similarity. The information is searched by doing. Below,
Unless otherwise specified, the value indicating the similarity is 0 or more,
The larger the value, the higher the similarity.

[0003] The similarity between character strings is roughly classified into two types: a method using morphological analysis and a method using n-grams.

A method using morphological analysis is described, for example, in Gera
rd Salton and Christopher Buckley, Term-Weighting
Approaches in Automatic Text Retrieval, Informatio
n Proceeding and Management, 24, pp.513-523, 1988.
Is disclosed. In order to determine the similarity between two character strings by this method, first, both character strings are decomposed into word strings (word strings) by morphological analysis using a dictionary and grammatical knowledge. Next, both word strings are compared to find a matching word. Then, weights are set for the matching words. This weight is then added for all matching words. The sum obtained as a result of this addition is the similarity obtained by morphological analysis.

[0005] The method using n-grams is described in, for example, Yasushi Og
awa and Toru Matsuda, Overlapping statistical word
indexing: A new indexing method for Japanese tex
t, Inproceeding of SIGIR'97, Philadelphia PA, USA,
pp.226-234, 1997. This method does not use morphological analysis. To determine the similarity between character strings by this method, first, a matching part of both character strings is determined. Then, a weight is set for the matching part. This weight is then added for all matching parts. The sum obtained as a result of this addition is
It is a similarity based on n-grams.

[0006] The above-described method based on morphological analysis and the method based on n-grams are additions regarding keywords constituting a document, and this is common in information retrieval. However, in these additions, the order in which the keywords appear is not reflected on the similarity.

[0007] In Japanese, a document is decomposed into a word string in the first step of information retrieval, but it is not easy to correctly perform this. Incorrect word segmentation means that keyword segmentation for effective information retrieval has failed, resulting in a decrease in information retrieval performance.

On the other hand, it is more difficult to specify a portion useful for information retrieval for a pattern having no word boundary. A character string can be regarded as a pattern in which characters are connected one-dimensionally. However, even for a one-dimensional pattern, it is difficult to apply the information retrieval methodology due to one of the difficulties in dividing the pattern.

[0009] From such a background, there is a method of calculating the similarity by summation by dynamic programming (hereinafter also referred to as DP). This includes
There are two methods, a simple addition method and a method of adding weight to a character.

A simple addition method is described, for example, by Robert
R. Korfhage, Information Storage and Retrieval, pp3
00-304, Wiley Computer Publishing, 1997. The method of calculating the similarity between character strings (hereinafter referred to as added DP similarity) by this method is as follows. Here, α and β are character strings, and x and y are character strings having different lengths 1 (characters are also referred to as character strings of length 1 for convenience of description). "" Is a character string of length 0 (hereinafter, an empty character string). The function MAX is a function that calculates a real number from a real number argument, and is a function that calculates the maximum value of the arguments. (The function MAX is also used to calculate other similarities unless otherwise specified.) The same function).

The additional DP similarity SIM ₁ is obtained by recursively applying the following equation according to the matching pattern for the character part of the argument. First, when both arguments are empty strings, SIM ₁ ("", "") = 0.0 (1). For a different character string having a length of one character or less, SIM ₁ (x, y) = 0.0 (2). If the first character is the same, SIM ₁ (xα, xβ) = MAX (SIM ₁ (α, xβ), SIM ₁ (xα, β), 1.0 + SIM ₁ (α, β)) (3) If the first character is different, SIM ₁ (xα, yβ) = MAX (SIM ₁ (α, yβ), SIM ₁ (xα, β), SIM ₁ (α, β)) (4). Equations (3) and (4) are efficiently calculated by the known DP method.

A method of adding weight to a character is described in, for example, H. Berg
hel and D. Roach, An Extension of Ukkonen's Enhanced
Dynamic Programming ASM Algorithm, ACM Transactio
nson Information Systems, Vol.14, No.1, pp.94-106,
January 1996. The method of calculating the similarity between character strings (hereinafter, character weight DP similarity) by this method is as follows. Here, α and β are character strings,
Let x and y be different character strings of length 1. "" Is an empty string. Score (x) is a function for obtaining a positive real value from the character string x.

The character weight DP similarity SIM ₂ is obtained by recursively applying the following equation according to the matching pattern for the character part of the argument. First, when both arguments are empty strings, SIM ₂ ("", "") = 0.0 (5). If the character strings have different lengths, SIM ₂ (x, y) = 0.0 (6). When the first character is the same, SIM ₂ (xα, xβ) = MAX (SIM ₂ (α, xβ), SIM ₂ (xα, β), Score (x) + SIM ₂ (α, β)) (7) And If the first character is different, SIM ₂ (xα, yβ) = MAX (SIM ₂ (α, yβ), SIM ₂ (xα, β), SIM ₂ (α, β)) (8). Equations (7) and (8) can be efficiently calculated by the known DP method, as in the case of the added DP similarity.

The Score (x) function used here gives a weight of a character string (character) x having a length of one. As can be seen by comparing Equations (3) and (7), the point of giving weight to a character using the Score (x) function is a feature of the character weight DP similarity.

[0015]

The method using morphological analysis has a problem that information retrieval cannot be performed well when frequently used words are used. This is because such words are commonly included in many character strings, and the presence or absence of such words alone is not a deciding factor in selecting a document. Another problem is that the order of appearance of words is not reflected in information retrieval.

Further, since morphological analysis is used,
If this morphological analysis is unsuccessful, there is an inherent restriction that information retrieval will end abnormally. In order to increase the accuracy of morphological analysis, a word dictionary and grammar rules must be large-scale, and it is difficult to use information retrieval easily. In addition, for documents in which buzzwords, coined words, and technical terms used only in limited fields appear, the burden of maintaining a word dictionary is a large burden.

On the other hand, the method using the n-gram does not require morphological analysis, and can be easily used. But,
Since the search does not focus on the word, it is difficult to perform a search that accurately grasps the content without considering a slight mismatch of character strings, for example, a difference in the inflected form of the word as a match.

On the other hand, of the two methods listed as the method based on dynamic programming, the method based on simple addition compares character strings one character at a time, and if one character matches, one is added as a score. In addition, a method of adding weight to a character is to compare the character strings one by one, and if one character matches, add a score, but the weight of the character, not 1, is added. This method can be said to be an extension of the simple addition method.

In either method, a value is added to the similarity every time one character matches. However, it is not considered whether the characters before and after also match. Therefore, the continuity of the matched characters is not taken into account, and in some cases, a completely different character string may be determined as a completely matched character string.

The present invention has been made to solve these problems, and has as its object to solve these problems by giving similarities not to characters but to character strings.

[0021]

According to a first aspect of the present invention, there is provided a method for calculating a similarity between two character strings, wherein the partial character string conforming to the order of each of the two character strings. And common to the two strings
A character string similarity calculation method, wherein a plurality of partial character strings are obtained, a weight is determined for each of the plurality of obtained partial character strings, and a similarity is calculated by summing the weights.

In a second aspect based on the first aspect, the weight of the partial character string is heavier than the sum of the weights of the partial character strings obtained by dividing the partial character string into two or more. There is a character string similarity calculation method characterized by the following.

In a third aspect based on the first or second aspect, one of the two character strings is selected from a document database, and the weight is the information amount of the partial character string in the document database. Is a character string similarity calculation method.

According to a fourth aspect of the present invention, any one of the first to third aspects is provided.
In the invention, the weight is stored in the document database.
Of substring information and the concentration of occurrence of substrings
Character string similarity calculation method
is there.

According to a fifth aspect of the present invention, in any one of the first to fourth aspects, a plurality of character string similarities are obtained without dividing the partial character string so that the similarity is the highest. This is a calculation method.

According to a sixth aspect of the present invention, in the method for calculating the similarity between two character strings, a partial character string conforming to the order of each of the two character strings is included in an element of the synonym dictionary. Character string similarity, wherein a plurality of character strings are obtained, a weight is determined for each of the elements of the synonym dictionary corresponding to the plurality of obtained partial character strings, and a similarity is calculated by summing up the weights. This is a calculation method.

According to a seventh aspect , in the sixth aspect , one of the two character strings is selected from a document database, and the weight is determined by an information amount of a synonym dictionary element in the document database. This is a character string similarity calculation method that is characterized by being compatible.

In an eighth aspect based on the sixth or seventh aspect , the two character strings are expressed in different languages, and the elements of the synonym dictionary include synonyms of the different languages. This is a character string similarity calculation method as a feature.

The ninth invention is directed to the first to fourth and sixth to fourth embodiments .
8. A character string similarity calculating method according to any one of the eighth to eighth aspects, wherein a plurality of partial character strings are allowed to be divided so as to obtain the highest similarity.

According to a tenth aspect , in the character string similarity calculating apparatus for calculating the similarity between two character strings, a character string score calculating unit for determining a weight of a partial character string common to the two character strings; And a matching character string similarity calculating unit that calculates the similarity by adding the similarity of the remaining partial character strings, and the similarity of the character strings reduced by one character in one or both of the two character strings. A character string similarity calculating apparatus, comprising: an arbitrary character string similarity calculating unit that obtains the highest similarity; and a selecting unit that selects the highest similarity among the obtained similarities.

According to an eleventh aspect , in the tenth aspect ,
A character string similarity calculating apparatus, wherein one of the two character strings is selected from a document database, and the weight corresponds to an information amount of a partial character string in the document database.

According to a twelfth aspect, in the eleventh aspect,
The weight is a partial character string in the document database.
Information volume and the degree of concentration of substrings
And a character string similarity calculation device.

According to a thirteenth aspect , in the character string similarity calculating apparatus for calculating the similarity between two character strings, a partial character string of the two character strings is included in an element of the synonym dictionary.
A synonym score calculating unit that determines the weight of an element of the synonym dictionary corresponding to the partial character string, and a matching character string similarity calculating unit that calculates the similarity by adding the similarity of the remaining partial character strings to the weight. An arbitrary character string similarity calculating unit for obtaining the highest similarity among the character strings reduced by one character in one or both of the two character strings, and the highest similarity among the obtained similarities And a selecting unit for selecting a character string similarity.

According to a fourteenth aspect , in the thirteenth aspect , one of the two character strings is selected from a document database, and the weight corresponds to an information amount of a synonym dictionary element in the document database. This is a character string similarity calculation device characterized in that:

A fifteenth invention is a text search apparatus for selecting a document similar to a search text from a document database, wherein the search text and the document in the document database are converted into two character strings in a tenth character string . A similarity calculated by the character string similarity calculating apparatus according to the fourteenth aspect of the present invention,
A document search apparatus characterized in that a document having a high degree of similarity is selected from a document database.

A sixteenth invention is a character string similarity calculation program for calculating the similarity between two character strings, wherein a comparison setting step of sequentially setting a part for comparing the two character strings is performed. A partial character string starting from a part, a common partial character string specifying step of specifying a partial character string common to the two character strings, a character string score setting step of determining a weight of the specified partial character string, A computer readable recording of a character string similarity calculation program for causing a computer to execute a character string score adding step of adding weight to the similarity and a progress processing step of advancing these steps to increase the similarity. It is a recording medium.

According to a seventeenth aspect , in the character string similarity calculation program according to the sixteenth aspect , one of the two character strings is selected from a document database, and the weight is set in the document database. This is a computer-readable recording medium that records a character string similarity calculation program, which corresponds to the information amount of a partial character string.

According to an eighteenth aspect, a sentence according to the seventeenth aspect is provided.
In the character string similarity calculation program, the weight may be
The amount of substring information in the document database, and
Sentence characterized by corresponding to the occurrence concentration of substring
Computer reading recorded character string similarity calculation program
It is a removable recording medium.

A nineteenth invention is directed to the character string similarity calculation program according to the seventeenth or eighteenth invention, wherein
The character string score setting step is a computer readable recording medium storing a character string similarity calculation program, wherein the information amount is obtained by using a suffix file.

A twentieth aspect of the present invention is a character string similarity calculation program for calculating the similarity between two character strings, wherein a comparison setting step for sequentially setting a part for comparing the two character strings is performed. A synonym specifying step of specifying a substring that is a substring starting from a part and included in an element of the synonym dictionary, and synonym determining a weight of an element of the synonym dictionary corresponding to the specified substring. A character string similarity calculation program for causing a computer to execute a word score setting step, a synonym score adding step of adding the weight to the similarity, and a progress processing step of advancing these steps to increase the similarity. Is a computer-readable recording medium having recorded thereon.

[0041] 21 invention, in string similarity calculation program according to the twentieth aspect of the present invention, which one of the two strings is selected from the document database, the weight is in the document database A computer-readable storage medium storing a character string similarity calculation program, which corresponds to the information amount of elements of a synonym dictionary.

The twenty-second aspect of the present invention, the string similarity calculation program according to the twenty-first aspect, the synonym score
A computer-readable recording medium recording a character string similarity calculation program, wherein the setting step obtains the information amount using a suffix file.

The twenty-third invention, in the string similarity calculation program according sixteenth to any one of aspects 22, string similarity calculation program the traveling process is characterized in that with dynamic programming techniques Is a computer-readable recording medium having recorded thereon.

A twenty-fourth aspect of the present invention is a text search program for selecting a document similar to a search text from a document database, wherein the search text and the document in the text database are converted into two character strings as a sixteenth character string . A document search program for causing a computer to calculate similarity by the character string similarity calculation program according to any one of the twenty-third to twenty-third aspects, and to select, from a document database, a document having the high calculated similarity. Is a computer-readable recording medium having recorded thereon.

[0045]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment)

First, an expression for obtaining a first similarity (hereinafter, referred to as a character string weight DP similarity) according to the present invention will be described.

Α, β, γ, and δ are character strings having a length of 0 or more, ξ is a character string having a length of 1 or more, and “” is an empty character string. Also, let x, y and z be character strings of length 1. A character string (for example, ξ) connecting a plurality of character strings (for example, γ and δ)
Is indicated by writing a symbol of a character string to be an element successively (for example, ξ = γδ).

The character string weight DP similarity SIM ₃ is obtained by recursively applying the following expression according to the matching pattern for the character part of the argument. First, when both are empty characters, SIM ₃ ("", "") = Score ("") (9). Otherwise, SIM ₃ (α, β) = MAX (SIM _3s (α, β), SIM _3g (α, β)) (10).

Here, SIM _3s is defined as follows: SIM _3s (ξα, ξβ) = MAX (Score (γ) + SIM ₃ (δα, δβ)) (11) It is determined to hold for all γ and δ that become γδ, and if there is no such character string ξ, SIM _3s (α, β) = 0.0 (12).

Further, SIM _3g is obtained by calculating SIM _3g (xα, yβ) = MAX (SIM ₃ (α, yβ), SIM ₃ (xα, β), SIM ₃ (α, β)) for an arbitrary character string. ). This expression is the similarity between the character strings (α and yβ, xα and β, and α and β) corresponding to the rest of one or both of the two character strings (xα and yβ) reduced by one character. This means that the highest similarity is adopted.

By applying the above expression recursively, a plurality of common partial character strings conforming to the respective orders of the two character strings are obtained, and the similarity is maximized. At this time, the formula
As can be seen from (11), the common partial character string is allowed to be further divided. The character string weight DP similarity shown in the present embodiment indicates a character string weight (hereinafter, character string score) as compared with the character weight DP similarity shown in the related art.
The feature is that the domain of the Score function is not a character (a character string of length 1) but a character string of length 1 or more.

It is assumed that the total number of documents is N, df (ξ) is the number of documents including the character string 、, and L is the length of the character string ξ. The Score (ξ) function that weights the character string is as follows: Score (“”) = 0.0 (14) when ξ is an empty character, and Score (ξ) = L × idf (ξ) when ξ is not an empty character. = L × log ₂ (N / df (ξ)) (15).

Here, idf (ξ) is a value corresponding to the information amount of ξ in information theory. That is, it corresponds to the number of bits log ₂ (N / df (ξ)) required to specify df (ξ) documents from among the documents having the total number N. This reflects the property that character strings appearing in many documents are not useful for searching, and conversely, character strings appearing in few documents are useful for searching.

The Score function is not limited to the above formula, but is a non-negative function relating to a character string. The function can be implemented as long as the character string has a tendency to increase in value as compared to simple addition.
That is, it is sufficient that Score (zγ)> Score (z) + Score (γ) (16) holds for zγ connecting two character strings z and γ. That is, the weight of the partial character string (for example, Score (zγ)) is the sum of the weights of the partial character strings obtained by dividing the partial character string into two or more (for example, Score (z) + Score (γ)). By the inequality of the expression (16) being satisfied, the character string weight DP similarity has a property that the similarity is higher as the match is made with a longer character string.

More specifically, the Score function calculates the character string weight D
Used many times to determine the P similarity. The present invention can be implemented if Expression (16) is established as an overall property of the Score function that has contributed to the final character string weight DP similarity. Therefore, a case where the inequality expression (16) does not hold may be included.

For example, in general, the character strings z and γ, which are shorter than the long character string zγ, appear in the document more frequently, so that Expression (16) holds. However, the character strings z and γ are both zγ
If the same number of times appears in the document, Score (zγ) = Score (z) + Score (γ) (17)

When a long character string zγ rarely appears (for example, once) in a document, it is not a word, but
May not be useful for search. In such cases, Score
Setting (zγ) = 0.0 is useful for searching. However, if the character string z and the character string γ frequently appear in the document, Score (zγ) <Score (z) + Score (γ) (18).

As described above, even when the cases where the expressions (17) and (18) are satisfied, since the expression (16) is generally satisfied, the present invention can be implemented by the expression (15). .

The Score function calculates the information amount of a character string and
Or the degree of appearance concentration. For example, data
Degree of occurrence of the character string concentrated in a specific document in the base
As a value indicating the degree of appearance, the degree of appearance concentration W (ξ) is defined as W (ξ) = (df ₂ (ξ) / df ₁ (ξ)) / (df ₁ (ξ) / N) (36) . However, when df ₁ (ξ) = 0, W (ξ) = 0.0 (37) . Then, instead of Score (ξ) in equation (15), W (ξ)
If ≧ K, Score (ξ) = idf (ξ) (38) , and if W (ξ) <K, Score (ξ) = 0.0 (39) , the present invention can be implemented. Where K is positive
Df ₁ (ξ) is the same as df (ξ).
The number of documents containing the character string ξ, df ₂ (ξ)
Indicates the number of documents containing two or more columns ξ.

The degree of appearance concentration is based on a document having a certain character string.
The degree of concentration that appears in
If the character string is, the value of Expression (36) is a value close to 1.0. When
Rolls are valid strings for search (keywords,
Text, or technical terminology for technical discussions)
When that character string appears, it appears twice or more in the same document.
High probability of appearing. Therefore, the value of equation (36) is less than 1.0.
It has the property that the value becomes larger. Therefore, the appearance collection
By switching between Equation (38) and Equation (39),
Search to provide more effective strings to Score
And the search accuracy can be improved. This
This is because the Score function is a sub-sentence in the document database.
It can be obtained by adding the amount of information according to the appearance concentration of the character string
This is an example.

Correspondence of Information Amount and Appearance Concentration to Score Function
The method of applying is not limited to the above example, and various modifications are possible.
You. For example, as another embodiment using the degree of occurrence concentration, the expression (3
8) and Equation (39), when W (ξ) ≧ 1, Score (ξ) = log ₂ (W (ξ)) × idf (ξ) (40) When W (ξ) <1 Can be set as Score (ξ) = 0.0 (41) . Thus, the values of idf (ξ) and W (ξ)
Score function is determined correspondingly. As a result,
More effective strings can contribute to Score,
Search accuracy can be improved.

In the above description, the value indicating the similarity is 0 or more, and the larger the value, the higher the similarity. If the correspondence between the value of the similarity and the height of the similarity is changed, the Score function is not limited to a non-negative function. For example, if it is assumed that the smaller the value of the similarity is, the higher the similarity is, it can be implemented by a Score function in which the sign of Expression (15) is inverted. In this case, the sign of equation (16) is reversed, and instead of the MAX function, the function MIN that calculates the minimum value of the arguments is used.
use. As described above, although the embodiment differs depending on the nature of the value indicating the similarity, the following description will be made on the assumption that the value indicating the similarity is 0 or more, and the larger the value, the higher the similarity.

Next, FIG. 1 shows an embodiment of a document search apparatus for searching for a document having the highest similarity to an input character string by using a character string weight DP similarity. This document search device includes a document database 10, a character string input unit 11, a search control unit 1,
2. It comprises a search result output unit 13, a similarity calculation unit 14, and a recursive execution control unit 15.

In the document database 10, a plurality of documents 10a, 10b,..., 10c to be searched are registered. For the search, a keyword, a word, a phrase, a sentence, a sentence, and the like are input (hereinafter, referred to as a search sentence).
The character string input unit 11 gives the search sentence to the similarity calculation unit 14 as a character string a. The search control unit 12 includes a document 10a,
.., 10c are taken out in order, and as a character string b,
This is given to the similarity calculation unit 14. The search result output unit 13 selects a document having the highest similarity obtained from the similarity calculation unit 14 and outputs it as a search result. The search result output unit 13 selects and outputs not only one document having the highest similarity but also all the documents whose similarity is equal to or more than a predetermined value, or a predetermined number of documents selected in descending order of similarity. You can also write a document.

The similarity calculator 14 calculates the similarity based on the equation (9) or (10). In the process of calculating the similarity, it is necessary to similarly calculate the similarity for a part of the character string. This is performed by the recursive execution control unit 15 using the similarity calculation unit 14 repeatedly. Expressions (10) to (1
In order to explain according to the notation of the symbol of 3), there is a portion in FIG. 1 where different symbols are reassigned to the same character string. For example, the character strings a and b received first by the similarity calculation unit 14 are represented by α and β in the similarity calculation unit 14 (the same applies hereinafter). As the character strings are separated, the similarity calculation unit 14 is used repeatedly, but the character strings α and β received at this time are different from the character strings a and b received first.

The similarity calculating section 14 is implemented by a matching character string similarity calculating section 21, an arbitrary character string similarity calculating section 22, and a maximum value selecting section 23. Matching character string similarity calculator 2
1 calculates SIM _3s (α, β) in equation (10). The arbitrary character string similarity calculation unit 22 calculates SIM _3g (α, β) in Expression (10).
The maximum value selection unit 23 calculates SIM ₃ (α, β) in Expression (10) by performing the function MAX on these. When both of the character strings α and β received by the similarity calculation unit 14 are empty characters, the recursive execution control unit 15 sets SIM ₃ (α, β) =
Set to 0.0. At this time, the matching character string similarity calculating unit 21, the arbitrary character string similarity calculating unit 22, and the maximum value selecting unit 23 are not operated. Needless to say, this value of SIM ₃ (α, β) = 0.0 implements equations (9) and (14).

The matching character string similarity calculation unit 21 is implemented by a character string separation control unit 31, a character string separation similarity calculation unit 32, and a maximum value selection unit 33, and SIM _3s (ξα,
ξβ) is calculated. The character string separation similarity calculation unit 32 is implemented by a character string separation unit 41, a character string score calculation unit 42, a similarity calculation unit 43, and an addition unit 44.

First, the character string separation control unit 31 determines if there is no matching character string に お い て in the character strings ξα and ξβ received by the matching character string similarity calculating unit 21, that is, if the matching character string ξ is an empty character string In the case of, according to equation (12), SIM
_3s (ξα, ξβ) = 0.0. In this case, the character string separation similarity calculation unit 32 and the maximum value selection unit 33 are not operated.

Next, if there is a matching character string 、 in the character strings ξα and ξβ received by the matching character string similarity calculating section 21, the character string separation control section 31 sets all γs in which ξ = γδ. And δ, the character string separation similarity calculation unit 32 is operated, and Score (γ) + SIM ₃ (δα,
δβ). Then, the largest value is selected by the maximum value selector 33. Thereby, SIM _3s (ξα, ξβ) shown in Expression (11) is obtained.

The character string separating section 41 separates the character string に into γ and δ, gives γ to the character string score calculating section 42, and gives δα and δβ to the similarity calculating section 43. The character string score calculation unit 42 calculates Score (γ) of Expression (11) with reference to the document database 10 based on Expression (15). Similarity calculator 43
Calculates SIM ₃ (δα, δβ) in equation (11). The similarity calculation unit 43 is actually implemented by the recursion execution control unit 15 applying the similarity calculation unit 14 to δα and δβ. The adding unit 44 performs addition of Expression (11).

The arbitrary character string similarity calculating section 22 is implemented by the similarity calculating sections 51 to 53 and the maximum value selecting section 54, and calculates SIM _3g (xα, yβ) of the equation (13). Corresponding to each case regarding the presence or absence of the first character x, y of the received character strings xα, yβ, the similarity calculation units 51, 52, and 53 respectively calculate SIM ₃ (α, yβ), SIM ₃ (xα, β), SIM ₃ (α, β)
Ask for. Actually, the similarity calculation units 51 to 53 use the recursion execution control unit 15 to set the similarity calculation unit 14 to α and y
This is implemented by applying to each of β, xα and β, and α and β. The maximum value selection unit 54 calculates the function MAX of Expression (13).
Is carried out.

Note that the character string score calculation unit 42 calculates the expression (15)
Instead of (36) to (39), or (36) and (40)
By calculating Score (γ) using (41) from
Ming can be implemented.

(Second Embodiment)

In the first embodiment, the function MA for finding the maximum value
Although X is used, it can be transformed into a function that performs some other representation. For example, if Score (ξ) has a property that the contribution of the character string having the maximum length determines the overall value, the present invention can be implemented by recursively applying the following expression to obtain the similarity. The Score function shown in Expression (15) has this property, and therefore can be applied to the present embodiment.
Expression (21) below eliminates the function MAX by representing Score (γ) in Expression (11) with Score (ξ) relating to the matching maximum length character string ξ. This example is
The feature is that if there is a continuously matching character string ξ, the weight Score (ξ) is given without further dividing the character string ξ. Therefore, this embodiment requires a smaller amount of calculation than the first embodiment, and the similarity SIM _3a is the same as that of the first embodiment.
IM ₃ approximation. That is, the similarity SIM _3a is the SIM ₃
Compared to, there is an advantage that the amount of calculation can be greatly reduced, though some errors are involved.

First, when both are empty characters, SIM _3a ("", "") = Score ("") (19) Otherwise, SIM _3a (α, β) = MAX (SIM _3as (α, β), SIM _3ag (α, β)) (20).

Here, SIM _{3as finds the} maximum matching character string as ξ, and _obtains SIM _3as (ξα, ξβ) = Score (ξ) + SIM _3a (α, β) (21). If the character string _存在 does not exist, SIM _3as (α, β) = 0.0 (22).

Further, SIM _3ag is _{calculated as} follows: SIM _3ag (xα, yβ) = MAX (SIM _3a (α, yβ), SIM _3a (xα, β), SIM _3a (α, β)) (23) ). By applying the above expression recursively, a plurality of common partial character strings conforming to the respective orders of the two character strings are obtained, and the similarity is maximized. However, unlike the first embodiment, as can be seen from Expression (21), in this embodiment, the common partial character string is not further divided.

FIG. 2 shows an embodiment of a document search apparatus for searching for a document having the highest similarity to the input character string by using the similarity SIM _3a . This document search device includes a document database 10, a character string input unit 11, a search control unit 12, a search result output unit 13, a similarity calculation unit 16, and a recursion execution control unit 17. In the figure, portions denoted by reference numerals 10 to 13 have the same functions and configurations as the portions denoted by the same reference numerals in FIG.

The similarity calculator 16 calculates the similarity based on the equation (19) or (20). In the process of calculating the similarity, it is necessary to similarly calculate the similarity for a part of the character string. This is performed by the recursive execution control unit 17 using the similarity calculation unit 16 repeatedly. Equations (20) to (2
In order to explain according to the notation of the symbol of 3), the same character string is reassigned with a different symbol in FIG. 2 as in FIG.

The similarity calculating section 16 is implemented by a matching character string similarity calculating section 24, an arbitrary character string similarity calculating section 25, and a maximum value selecting section 26. Matching character string similarity calculator 2
4 calculates SIM _3as (α, β) in equation (20). The arbitrary character string similarity calculation unit 25 calculates SIM _3ag (α, β) in Expression (20). The maximum value selection unit 26 calculates SIM _3a (α, β) in Expression (20) by performing the function MAX on these. If both of the character strings α and β received by the similarity calculation unit 16 are null characters, the recursive execution control unit 17
_{3 Let a} (α, β) = 0.0. At this time, the matching character string similarity calculating unit 24, the arbitrary character string similarity calculating unit 25, and the maximum value selecting unit 26 are not operated. Needless to say, this SIM _3a (α,
The value β) = 0.0 implements equations (19) and (14).

The matching character string similarity calculating section 24 is implemented by a matching character string determining section 34, a character string score calculating section 45, a similarity calculating section 46, and an adding section 44.

First, the matching character string determining unit 34 determines that there is no matching character string に お い て in the character strings ξα and ξβ received by the matching character string similarity calculating unit 24, that is, the matching character string ξ is an empty character string. In the case of, according to equation (22), SIM _3as
(ξα, ξβ) = 0.0. At this time, the character string score calculator 45, the similarity calculator 46, and the adder 47 are not operated.

Next, if there is a matching character string in the character strings {α and {β ”received by the matching character string similarity calculating section 24, the matching character string determining section The degree calculating unit 46 and the adding unit 47 are operated to calculate Score (ξ) + SIM _3a (α, β) in Expression (21).
Specifically, the character string score calculation unit 45 refers to the document database 10 based on Expression (15), and calculates the Score of Expression (11).
Calculate (ξ). The similarity calculation unit 46 is actually implemented by the recursion execution control unit 17 applying the similarity calculation unit 16 to α and β. The addition unit performs addition of Expression (21). Thus, SIM _3as (ξα, ξβ) shown in equation (21)
Is found.

The arbitrary character string similarity calculating section 25 is implemented by the similarity calculating sections 55 to 57 and the maximum value selecting section 58, and calculates SIM _3ag (xα, yβ) of the equation (23). Corresponding to each case regarding the presence or absence of the first character x, y of the received character strings xα, yβ, the similarity calculation units 55, 56, and 57 respectively calculate SIM _3a (α, yβ), SIM _3a (xα, β), SIM
_{3a Find} (α, β). Actually, the recursive execution control unit 17 causes the similarity calculating units 55 to 57 to
Is applied to each of α and yβ, xα and β, and α and β. The maximum value selection unit 58 implements the function MAX of Expression (23).

Note that the character string score calculation unit 45 calculates the expression (15)
Instead of (36) to (39), or (36) and (40)
By calculating Score (ξ) using (41) from
Ming can be implemented.

(Third Embodiment)

Next, an embodiment of the present invention using synonym dictionary information will be described. Character strings belonging to a certain set of character strings described in the synonym dictionary are assumed to be synonymous. In addition, as a notation, {, η, ...} is a synonymous character string having a length of 1 or more. That is, it is assumed that a set T of a certain character string exists in the elements of the synonym dictionary D, and ξ, η,. Expressed as an equation, D = {{ξ, η,...}} (24)

Α and β are character strings, and “” is an empty character string. Also, x and y are character strings of length 1. T is a set of character strings that are elements of the synonym dictionary D.

The similarity (hereinafter, referred to as DP similarity having a character string weight having synonym dictionary information) SIM ₄ according to the present embodiment is obtained by recursively applying the following equation.

First, in the case of an empty character string, SIM ₄ ("", "") = 0.0 (25). Otherwise, SIM ₄ (α, β) = MAX (SIM _4s (α, β), SIM _4g (α, β)) (26).

Here, SIM _4s (α, β) means that when words ξ and η having the same meaning are at the head, SIM _4s (α, β) = MAX (SynonymScore (T) + SIM ₄ (γ, δ)) (27) is determined to hold for all ξ and η such that α = ξγ, β = ηδ, ξ∈T, η 、 T, and when there is no such ξ and η, SIM _4s (α, β) = Set to 0.0 (28).

Further, SIM _4g (xα, yβ) is calculated as follows: SIM _4g (xα, yβ) = MAX (SIM _4g (α, yβ), SIM _4g (xα, β), SIM _4g (α, β)) (29). By applying the above formula recursively, a plurality of partial character strings that match the order of each of the two character strings and are synonymous are obtained, and the similarity is maximized.

A function SynonymScore for giving a weight (hereinafter, synonym score) considering the synonym dictionary is df (ξ, η,...)
Is the number of documents containing ξ, η or ζ, and ξ, η,… ζ is T
Let L be the length of the target string, SynonymScore (T) = L × idf (ξ, η,...) = L × log ₂ (N / df (ξ, η,...)) ) (30). In addition, it can also be implemented by a function relating to a set of character strings.

By using the synonym dictionary in this way,
Searches that take into account the correspondence between codes such as full-width and half-width,
You can read names and search for information with aliases.

Next, an embodiment of a document search apparatus for searching for a document having the highest similarity to an input character string by using a DP similarity SIM _{4 having} a character string weight with synonym dictionary information is shown in FIG.
Shown in This document search device includes a document database 10,
Character string input unit 11, search control unit 12, search result output unit 1
3, a similarity calculation unit 18, a recursive execution control unit 19, and a synonym dictionary 20. In the figure, portions denoted by reference numerals 10 to 13 have the same functions and configurations as the portions denoted by the same reference numerals in FIG.

The similarity calculating section 18 calculates the similarity based on the equation (25) or (26). In the process of calculating the similarity, it is necessary to similarly calculate the similarity for a part of the character string. This is performed by the recursive execution control unit 19 using the similarity calculation unit 18 repeatedly. In order to explain according to the notation of the expression symbol, there is a portion in FIG. 3 where different symbols are reassigned to the same character string in FIG.
Is the same as

The synonym dictionary 20 registers synonyms. For example, in Japanese, "US", "US", "USA", "USA" and "MERIKEN" are synonymous.
= Register {United States, United States, United States, United States, Meriken}. In order to cope with variations in notation and common errors, it is also possible to add the hiragana “beikoku” or the erroneous notation “United States of America” as an element. Similarly, for example, in English, "United States""United States of Americ"
"a", "America", "USA", and "US" are synonymous, so if you search in English, the element of the synonym dictionary 20 will be
tes, United States of America, America, USA, U.
S.} is registered. To cope with the difference in character codes, for example, "US" represented by one-byte characters (half-width characters)
A. ”is preferably added to the element in addition to“ USA ”represented by two-byte characters (full-width characters).

Depending on the purpose of the search, the meaning of the word can be widened or narrowed. For example, "Teito", "Tokyo", "Keihin", and "Kanto" are words that indicate the Tokyo area, although they are wide and narrow. In this case, T = {Teito, Tokyo, Keihin, Kanto} may be registered as an element of the synonym dictionary 20. As described above, it is preferable to register many synonyms of useful words (keywords) as clues to be searched in the synonym dictionary 20, because omission of search is reduced.

The similarity calculator 18 includes a synonym similarity calculator 28, an arbitrary character string similarity calculator 29, and a maximum value selector 30.
Has been implemented. The synonym similarity calculation unit 28 calculates the expression
The SIM _4s (α, β) of (26) is calculated. The arbitrary character string similarity calculation unit 29 calculates SIM _4g (α, β) in Expression (26). By performing the function MAX on the maximum value selection unit 30 and these,
Calculate SIM ₄ (α, β) in equation (26). When both the character strings α and β received by the similarity calculation unit 18 are empty characters, the recursive execution control unit 19 sets SIM ₄ (α, β) = 0.0 to execute the equation (25). . At this time, the synonym similarity calculator 28, the arbitrary character string similarity calculator 29, and the maximum value selector 30 are not operated.

The synonym similarity calculating section 28 is implemented by a synonym separation controlling section 61, a synonym separating similarity calculating section 62, and a maximum value selecting section 63. The SIM _4s (α, β) ) Is calculated. The synonym separation similarity calculation section 62 is implemented by a synonym separation section 71, a synonym score calculation section 72, a similarity calculation section 73, and an addition section 74.

First, the synonym separation section 71 sets the synonym dictionary 2
0, and the character strings α and β received by the synonym similarity calculation unit 28 are calculated using α and η included in the element T as α = β.
Try to separate 分離 γ, β = ηδ. If such separation is not possible, the synonym separation control unit 61
SIM _4s (α, β) = 0.0. In this case, the synonym separation control unit 61 does not operate the character string separation similarity calculation unit 32 and the maximum value selection unit 33 any more.

On the other hand, if such separation can be performed, the synonym separation control unit 61 determines that all the ξ that satisfy α = ξγ and β = ηδ
With respect to and η, the synonym separation similarity calculation unit 62 is operated to calculate SynonymScore (T) + SIM ₄ (γ, δ) included in Expression (27). Then, the largest value is set to the maximum value selection unit 63.
Select by. This allows the SIM shown in equation (27)
_{4 s} (α, β) is obtained.

The synonym separation section 71 converts the character strings α and β into
Separately into ξ, γ, η, δ, synonym dictionary to which ξ and γ belong
The element T of D is provided to the synonym score calculation unit 72, and γ and δ are provided to the similarity calculation unit 73. The synonym score calculation unit 72
Based on the expression (30), referring to the document database 10, the expression (2
7) SynonymScore (T) is calculated. Similarity calculator 73
Calculates SIM ₄ (γ, δ) in equation (27). Similarity calculator 7
3 is actually implemented by the recursion execution control unit 19 applying the similarity calculation unit 18 to γ and δ.
The adding unit 74 performs addition of Expression (27).

The arbitrary character string similarity calculating section 29 is implemented by the similarity calculating sections 81 to 83 and the maximum value selecting section 84, and calculates SIM _4g (xα, yβ) of the equation (29). In each case regarding the presence or absence of the first character x, y of the received character strings xα, yβ, the similarity calculation units 81, 82, 83 respectively calculate SIM ₄ (α, yβ), SIM ₄ (xα, β), SIM ₄ (α, β)
Ask for. Actually, the similarity calculation units 81 to 83 use the recursive execution control unit 19 to set the similarity calculation unit 18 to α and y
This is implemented by applying to each of β, xα and β, and α and β. The maximum value selecting unit 84 calculates the function MAX of Expression (29).
Is carried out.

(Fourth embodiment)

Character string weighted D with synonym dictionary information
In P similarity, if a bilingual dictionary is used as a synonym dictionary, the similarity between documents in different languages can be obtained.
This allows you to search for documents in a language different from the search sentence,
Cross-lingual information retrieval (Cross Lingual Informatio
n Retrieval) is shown in FIG.

This cross-language information search device includes a document database 10, a character string input unit 11, a search control unit 12, a search result output unit 13, a similarity calculation unit 18, and a recursive execution control unit 1.
9, a bilingual dictionary 91. 10 to 1 in the figure
3 have the same functions and configurations as the parts with the same reference numerals in FIG. 1, and the parts with the reference numerals 18 and 19 in the figure are the same as the parts with the same reference numerals in FIG. Since it has a function and a configuration, the description is omitted.

This cross-language information search device searches the document database 10 for a document having the highest similarity to the search sentence input from the character string input unit 11 and outputs it. The search text and the document in the document database 10 are described in different languages. For example, the former is written in Japanese and the latter is written in English. Then, by inputting a Japanese document from the character string input unit 11, the most similar English document can be searched from the document database 10.

The bilingual dictionary 91 is a synonym dictionary 2 shown in FIG.
It has the same function as 0, but differs in that registered synonyms are combinations of different languages. It goes without saying that each language matches the language of the search sentence input from the character string input unit 11 and the language of the document in the document database 10. For example, "American" in Japanese
Is synonymous with "American" in English, so T = {American,
American} is registered as a bilingual dictionary element. Of course, including multiple synonyms, for example, T = {American,
It is more preferable to set "American" as the number of search omissions is reduced.

(Fifth Embodiment)

Next, an example in which the text search according to the present invention is implemented by software will be described. FIG. 5 shows an example of a computer system used for executing software. This computer system includes a display 101, a printer 102, a keyboard 103, a floppy (registered trademark) disk device 104, a CD-ROM (Compact Disk-Read Only Mem).
ory) device 105, read only memory (Read Only Mem)
ory. ROM) 106, a readable / writable random access memory (Random Access Memory; hereinafter, RA)
M) 107, magnetic disk device 108, central processing unit (Central Processing Unit; hereinafter, CPU), communication interface 110, and bus 1 connecting these.
11. Floppy disk drive 10
4 reads and writes the floppy disk 112, and
The ROM device 104 reads the CD-ROM 113; The computer system is connected to a communication network 114 via a communication interface 110.

A sentence search program for implementing the present invention is:
It is stored in the ROM 106. Alternatively, a text search program is stored in the floppy disk 112, the CR-ROM 113, or the magnetic disk device 108, and the
After the transfer to the AM 107, the CPU 109 may execute the processing. The CPU 109 executes the text search program using the RAM 107 as a work area. If necessary, the magnetic disk device 108 may be used as a work area. An instruction to execute the text search program is issued from the keyboard 103, and the execution result is output to the display 101 or the printer 102. It goes without saying that the execution of the text search program may be instructed from the floppy disk 112 and the execution result may be written to the floppy disk 112.

The document database and the synonym dictionary (if necessary) are stored on the floppy disk 112, CD-ROM 1
13 or on the magnetic disk 108. The data may be transferred to the RAM 107 for high-speed access. When transferring the data to the RAM 107, the data may be converted into a format that can be easily processed. In addition, a text search program, a document database, a synonym dictionary (if necessary), or an execution instruction is input to the computer system via the network 114, and an execution result is output from the computer system via the network 114. What you can do is
Of course.

The present invention is not limited to those shown in the figures, and various recording media, input means, and output means can be used to input and output to the computer system, and can be modified into various embodiments. Needless to say. These recording media,
It goes without saying that the input means and the output means may be those directly accessed by the computer system, or those accessed via a communication network.

FIGS. 6 to 10 show the processing flow of the document search program using the DP similarity SIM _{3 having} a character string weight. This program inputs a search sentence, searches a sentence database, and outputs a plurality of documents having high similarity.

FIG. 6 shows a flow of processing for searching a document database based on a search sentence, selecting and outputting a document having a high degree of similarity. First, step S11 (hereinafter, S
11), a suffix file (Suffix File) is created by combining all documents included in the document database in order to efficiently calculate the number of appearances of a certain character string. For details on how to create and use suffix files, see Mikio Yamamoto and Kenneth W. Church, Usi
ng Suffix Arrays to Compute Term Frequency and Doc
ument Frequency for All Substrings in a Corpus, In
proceeding of 6th Workshop on Very Large Corpora,
Ed. Eugene Charniak, Motreal, pp 28-37, 1998.

By using a suffix file, the number of times a character string appears in a document can be obtained at high speed.
The suffix file is implemented by sorting partial character strings that can occur in all documents according to character codes (sorting them in the order of character codes) and adding serial numbers (suffixes). The number of times a character string appears in a document can be obtained by calculating the number of character strings that match the character string in the suffix file.

Specifically, first, a matching character string is searched from the head of the suffix file, and the suffix mi is searched.
Find n. If there is no matching string, the number of occurrences in the document is zero. Next, search for a matching string from the end of the suffix file,
Find x. Then, the number of times the character string appears in the document tf
Is obtained by tf = max−min + 1 (31). The documents in the document database are distinguished from each other by a document number (Document ID). The substring registered in the suffix file contains
This document number is assigned so that the document from which the partial character string is derived can be known.

Next, in step S12, the search text is read into the character string X. In S13, a certain document is selected from the documents in the document database and read into the character string Y. Next, S14
Calculates the similarity between the character strings X and Y. The processing performed in S14 will be described later with reference to FIG. In S15, the obtained similarity and the document number are registered as a set in the document management table. In S16, it is determined whether or not the similarity has been calculated for all the documents included in the document database. if,
If the calculation has not been performed for all the documents, a document that has not been calculated is selected in S13 and read into the character string Y, and the processing up to S15 is repeated. If the calculation has been performed for all the documents, the registered tables are sorted in descending order of similarity in S17. In S18, a process of outputting a document having a high similarity is performed. Various modes are possible, such as outputting only one document, a plurality of documents, or all documents having a predetermined similarity or more.

The similarity is calculated using the character string score Scor shown in equation (15).
It is obtained by adding e (ξ). The value in the middle of this addition is hereinafter referred to as a score. Equations (11) and (1
As can be seen from the function MAX of 3), it is necessary to calculate the score obtained by adding the character string score and the score for the partial character string whose length is changed one character at a time, and select the maximum value. In order to efficiently advance the calculation and selection, a score table shown in FIG. 11 is used. In this score table, the horizontal direction corresponds to the character string X, and the vertical direction corresponds to the character string Y. Below,
The length of the character string X is x_len, and the length of the character string Y is y_len. By filling the score in this score table so that the value increases from the upper left to the lower right, Expressions (10) and (1
The function MAX of (1) and (13) is performed.

More specifically, scores corresponding to the i-th character of the character string X and the j-th character of the character string Y are score [i] [j]. In the character string X and the character string Y, the matching character strings γ are the i-th to i + m-th characters of the character string X and the j-th to j + n-th characters of the character string Y, and score [i]
Value obtained by adding the character string score of substring γ to [j]
And Then, score [i + m + 1] [j + n to satisfy the relationship of score [i + m + 1] [j + n + 1] ≧ value (32)
If +1] is smaller than value, the value is score [i + m + 1] [j
+ N + 1]. If this processing is performed for all the partial character strings γ, the function MAX of Expression (11) and the SIM _3s (α,
The function MAX for β) is implemented.

Score [i + 1] [j] ≧ score [i] [j] (33) score [i] [j + 1] ≧ score [i] [j] (34) score [i + 1] [j + 1] ≧ score [i] [j] (35) In order to hold the relationship, score [i + 1] [j], scor
If e [i] [j + 1] and score [i + 1] [j + 1] are smaller than score [i] [j], write score [i] [j] respectively. If this process is performed while sequentially increasing i and j, the function MAX of Expression (13) is obtained.
Then, the function MAX relating to SIM _3g (α, β) in Expression (10) is performed.

The comparison and writing are sequentially processed according to the flow shown in FIG. 7 to complete the score table.
Then, score [x_len] [y_len] at the lower right of FIG. 11 becomes the largest value in the score table, and this is the similarity to be obtained.
SIM ₃ (X, Y). The procedure for obtaining the similarity from the score table is an example of the DP method.

FIG. 7 shows a character string X from which a search sentence is read.
5 shows a flow of processing for calculating the similarity of a character string Y obtained by reading a document in a document database. In the following, the character string X
X [i, m] is a substring of m characters from the i-th character of
A partial character string of n characters from the jth character of is denoted by Y [j, n]. For example, the substring consisting of the first two characters of the character string X is X
[0,2].

First, at S21 shown in FIG.
Initialize all values of score table score [i] [j] of X and character string Y to 0. Next, in S22, i indicating the head of extracting a partial character string from the character string X is set to 0. In S23, j indicating the beginning of extracting the partial character string from the character string Y is set to 0. In S24, the score score [i] [j] currently focused on in the score table is stored in a temporary variable current for convenience of processing.

In S25, k indicating the length of the portion where the character strings X and Y match is set to 0. In S26, i + k indicating the part for which a match is determined in the character string X is the character string X
Is smaller than the length x_len, and whether the characters X [i + k, 1] and Y [i + k, 1] of the character strings X and Y match each other. If this condition is satisfied, proceed to S27,
If this condition is not satisfied, the process proceeds to S32.

Steps S27 to S31 are processes for sequentially increasing the length of the matching portion while adding the character string score to the score table for the matching portion. When the process proceeds from S26 to S27, the partial character string X [i, k + 1] matches the partial character string Y [j, k + 1]. First, in S26, a character string score of the partial character string X [i, k + 1] is obtained. This is given by equation (1
This is the process of calculating Score (ξ) in 5) , which will be described later with reference to FIGS. Let this character string score be tmp_score. Next, in S28, tmp_score is added to the score (current) currently focused on, and this value is added to a temporary variable value
To memorize. S29 and S30 are comparison and writing for satisfying the expression (32). In S31, in order to lengthen the matching part and repeat S26 to S30,
Increase k by 1.

S32 and S33, S34 and S35, S36
Steps S37 and S37 are comparison and writing, respectively, for satisfying the equations (33), (34), and (35).

In S38, it is determined whether j indicating the head of extracting the partial character string from the character string Y is less than the length y_len of the character string Y. If this condition holds, S3
In step 9, j is incremented by 1 and the processing from S24 to S37 is repeated. If the processing is completed up to the end of the character string Y, S38 to S
Proceed to 40

In S40, it is determined whether or not i indicating the head of extracting the partial character string from the character string X is less than the length x_len of the character string X. If this condition holds, S
In step 41, i is incremented by 1 and the processing from S23 to S38 is repeated. When the process is completed up to the end of the character string X, the process proceeds from S40 to S42.

If the processing is completed up to the end of the character string X, the score table is completed. Therefore, in S42, scor
A process of returning e [x_len] [y_len] as the similarity between the character strings X and Y is performed.

FIG. 8 shows a flow of processing for calculating the character string score sc of the character string a using the suffix file. In the character string a, the partial character string X [i, k +
1] is given. First, in S51 of FIG.
Find the number df of documents in which appears. The specific processing of S51 will be described later with reference to FIG. Next, in S52, the number of occurrences tf of the character string a in all documents is obtained. The specific processing of S52 will be described later with reference to FIG.

[0133] In S53, the number of occurrences tf of string it is determined whether or not less than 2 times, giving a 0.0 to a temporary variable sc in if less than 2 times S54. This is because a character string that appears rarely in a document is not a meaningful word, that is, it is not considered useful for information retrieval. If the number of appearances tf of the character string is two or more, the determination in S55 is performed.

At S55, the character string a for the number N of all documents
Is a process for determining whether or not the ratio of the number df of documents in which. When than 5% larger, giving a 0.0 <br/> score sc in S54. This is because a character string that frequently appears in a document is a word (stop word) serving as a delimiter and is not a useful character string, that is, it is not considered useful for information retrieval.

If the ratio between N and df is 5% or less, the flow goes to S56.
Then, calculation for obtaining the score sc is performed by the equation (15). Finally, sc is returned as a character string score in S57.

FIG. 9 shows a flow of processing for obtaining the number df of documents in which the character string a appears. In this processing, in order to reduce the processing time for the same character string, the character string a and the calculated df are registered in a hash table for storing the number of documents (hereinafter, a document number hash table).
It eliminates the need for recalculation. First, in S61, a character string
It is determined whether or not a is registered in the document number hash table. If registered, df registered in S62 is obtained. On the other hand, if it is not registered in the document number hash table, df is calculated in S63 to S68.

In step S63, a character string a is searched in order from the beginning of the suffix file, and the suffix is set to min. When min cannot be obtained, that is, when the character string a is not included in the suffix file, the character string a does not appear in the document. This is determined in S64, and df is set to 0 in S65.

Step S64 is a process for sequentially searching for a character string a from the end of the suffix file and setting the suffix to max. In the suffix file, the range of the suffix from min to max is a character string that matches the character string a. S67 is a process of calculating the numbers of different document numbers assigned to these character strings. This number is
The number of documents in which the character string a appears is df.

In step S68, the character string a and the number of documents df are registered in the document number hash table. S69 is
This is the process of returning the number of documents df.

FIG. 10 shows a flow of processing for obtaining the number of times tf that the character string a appears in all the documents. S71 is a process of searching for a character string a in order from the beginning of the suffix file and setting the suffix to min. When min cannot be obtained, that is, when the character string a is not included in the suffix file, the character string a does not appear in the document. This is determined in S72, and tf is set to 0 in S73.
And

S74 is a process of searching for a character string a in order from the end of the suffix file and setting the suffix to max. In S75, calculation of Expression (31) is performed. S76
Is a process for returning the number of appearances tf.

It should be noted that instead of equation (15), equations (36) to (39)
In the case of using the method shown in FIG. 8 and FIG.
Instead, the flow shown in FIGS. 12 and 13, respectively, is used.
Just do it. The same applies when using Equation (36) and Equations (40) to (41).
It can be realized using a flow. S82 and S83 in FIG.
The processes shown in S84, S85, and S89 are respectively performed in FIG.
Shown in S52, S53, S54, S55 and S57
Since the processing is the same as the above, the description is omitted. S81 in FIG.
Is the number of documents in which character string a appears df ₁ and character string a occurs twice
Determine the number of document df ₂ occurs more than. Specific of S81
The processing will be described later with reference to FIG. In S86 of FIG.
Obtains the appearance concentration degree shown in Expression (36). S87 is
Compare the occurrence concentration with a positive real constant K (for example, 2.0)
The processing is performed in S88 or S84 according to the magnitude relation.
move on. S88 is a process for obtaining a character string score according to equation (38).
In S84, the character string score is set to 0.0 according to the equation (39).
Is the processing to be performed.

[0143]FIG. 13 shows the number of documents df _{1 in} which the character string a appears.
And the number of documents in which the character string a appears twice or more df ₂ Where to seek
The flow of the processing is shown. This processing is the same as the processing shown in FIG.
To reduce the processing time for the same character string
Use document count hash table, but with strings a and df ₁ With
To df ₂ Are also different in that they are also registered. First, in S91,
Whether the character string a is registered in the document count hash table
Is determined, and if registered, registered in S92.
df ₁ And df ₂ Ask for. On the other hand, the
If it has not been recorded, in steps S93 to S99, df ₁ And d
f ₂ Is calculated. S93, S94, S96 shown in FIG.
S97 corresponds to S63, S64, S6 shown in FIG.
6, since the processing is the same as that shown in S67, the description is omitted.
S95 shown in FIG. 13 is used when character string a does not appear in the document.
If df ₁ And df ₂ Is a process of setting the value to 0. S93 and S96
The range from min to max determined by is the suffix
This is a character string that matches the character string a in the file.
In S98, reference is made to the document numbers assigned to these character strings.
Then, find the number of documents with two or more identical document numbers.
This is the process to be performed. This number indicates that the character string a appears twice or more
Number of documents df ₂ become. In S99, the character string a and the number of documents df
₁ , Df ₂ Is registered in the document count hash table.
You. S100 is df ₁ , Df ₂ Each occurrence of the string a
The number of documents to be written, the number of documents where the character string a appears twice or more,
And return it.

(Sixth Embodiment)

Next, an example in which a document search by the similarity SIM _{3a is performed} by software will be described. The computer system shown in FIG. 5 is used for executing the software. The point that a document search program and the like can be recorded on a recording medium is the same as in the fifth embodiment. The flow of the document search program is shown in FIG . 6, FIG. 14 , FIG. 8 to FIG. Instead of the flow shown in FIGS. 8 and 9,
The flow shown in FIGS. 12 and 13 may be used in the fifth embodiment.
This is the same as in the case of the embodiment. This embodiment differs from the DP similarity SIM ₃ having a character string weight in that the flow shown in FIG. 14 is used instead of the flow shown in FIG . Therefore, the common description is omitted, and only the flow of FIG. 14 will be described below.

FIG. 14 shows the flow of the processing for calculating the similarity between the character strings X and Y performed in S14 shown in FIG. 6 in this embodiment. Note that S121 to S142 shown in FIG. 14 are respectively S21 to S42 shown in FIG.
Related to The similarity SIM _3a is different from the similarity SIM ₃ in that a character string score is calculated after obtaining the maximum length of a matching character string. The flow shown in FIG. 14 is different in this point from the flow shown in FIG. The flow shown in FIG. 14 will be described below in order.

First, in S121, all values of the score tables score [i] [j] of the character strings X and Y are initialized to zero. Next, in S122, i indicating the head of extracting a partial character string from the character string X is set to 0. In S123, j indicating the beginning of extracting the partial character string from the character string Y is set to 0. S
At 124, the score currently focused on in the score table
score [i] [j] is a temporary variable cur for convenience of processing
Remember in rent.

In S125, k indicating the length of the part where the character strings X and Y match is set to 0. In S126, i + k indicating the part of the character string X for which a match is determined is less than the length x_len of the character string X, and one character of each of the character strings X and Y X [i + k, 1], Y [i + k, 1 ] Are matched. If this condition holds, S131
Then, S126 is performed again. By doing so, a search is made to find out how far they match. If they do not match, S12
Go to 0.

In S120, it is determined whether there is a matching part. If k> 0, that is, if there is a matching part, the process proceeds to S127. On the other hand, if there is no match,
Proceed to S132.

Steps S127 to S130 are processing for adding the character string score to the score table for the matching part. When proceeding from S120 to S127, a partial character string
X [i, k + 1] and partial character string Y [j, k + 1] match. First, in S127 , a character string score of the partial character string X [i, k + 1] is obtained. This is a process for obtaining Score (ξ) in Expression (15), which has been described above with reference to FIGS. Alternatively, equation (3
6) is a process of calculating Score (ξ) using (39),
This is carried out according to the flowcharts shown in FIGS. Ah
Alternatively, using the Score function shown in Equation (36) and Equations (40) to (41)
It may be. The calculated string score is tmp_
Score. Next, in S128, tmp_score is added to the current score (current), and this value is stored in a temporary variable value. S129 and S130 are calculated by the formula (3)
Comparison and writing to make 2) hold. Thereafter, the process proceeds to S132.

S132 and S133, S134 and S13
5, S136 and S137 are expressed by equations (33) and (3
4) and comparison and writing to make equation (35) hold.

In S138, it is determined whether or not j indicating the head of extracting the partial character string from the character string Y is less than the length y_len of the character string Y. If this condition holds, S
At 139, j is incremented by 1 and the processing from S124 to S137 is repeated. When the processing is completed up to the end of the character string Y, S1
Proceed from S38 to S140

In S140, it is determined whether or not i indicating the head of extracting the partial character string from the character string X is less than the length x_len of the character string X. If this condition holds,
In S141, i is increased by 1, and the processing from S123 to S138 is repeated. When processing is completed up to the end of the character string X,
The process proceeds from S140 to S142.

If the processing is completed up to the end of the character string X, the score table is completed. Therefore, in S142, sc
ore [x_len] [y_len] is returned as the similarity between the character strings X and Y.

(Seventh Embodiment)

Next, an example will be described in which a document search is performed by software using the DP similarity SIM _{4 having} a character string weight with synonym dictionary information. The computer system shown in FIG. 5 is used for executing the software.
The point that a document search program and the like can be recorded on a recording medium is the same as in the fifth embodiment. The flow of the document search program is shown in FIGS. 15 to 18 and FIGS. 8 to 10. That is, the processing for obtaining the scores shown in FIGS. 8 to 10 is performed by the DP similarity SIM ₃ and the similarity SIM
Common to _3a .

FIG. 15 shows a flow of processing for searching a document database based on a search sentence, selecting and outputting a document having a high degree of similarity. Note that S211 to S218 shown in FIG. 15 correspond to S11 to S18 shown in FIG.
This is the same processing as.

In S210 shown in FIG . 15 , information on synonyms is read from a recording medium such as a CD-ROM and a synonym dictionary is created. FIG. 16 shows the flow of the process of creating this synonym dictionary. In this synonym dictionary, a set of a synonym key serving as a dictionary head and a plurality of synonyms is registered in a hash table (hereinafter, synonym hash table). Searching for a synonym from the dictionary is performed by extracting all synonyms corresponding to the synonym key. On the other hand, recording media such as floppy disks and CD-ROMs do not always have all the synonyms corresponding to synonym keys recorded in one place in order to easily edit and expand the synonym dictionary. Absent. Here, a description will be given of an embodiment in which a synonym key and a synonym are recorded in pairs on a recording medium.

First, in step S221 , a set of a synonym key and a synonym is read from the recording medium. Next, in S222 , it is determined whether the read synonym key is already registered. If not registered, the synonym key and the synonym are registered in the synonym hash table in S223 . If the synonym key has already been registered, in S224 , after the synonym corresponding to the synonym key, a synonym to be registered as a delimiter such as "," (comma) or "\ t" (tab) Register additional words in the synonym hash table so that words are added. In the following, one registered synonym or a plurality of synonyms separated by delimiters will be referred to as a synonym string.

After S223 and S224 , it is determined in S225 whether a synonym to be registered still exists. If it still exists, the process from S221 to S224 is repeated and registered. Needless to say, when there are a plurality of recording media (for example, a CD-ROM in which basic synonyms are registered)
Is used together with a floppy disk that registers synonyms for a specific field of use).
The above processing is performed. When all the synonyms have been registered in this way, the creation of the synonym dictionary ends.

Next, in step S211 shown in FIG. 15 , all the documents included in the document database are integrated to create the above-mentioned suffix file in order to efficiently calculate the number of appearances of a certain character string. Then, in S212 , the search text is read into the character string X. In S213 , a certain document is selected from the documents in the document database and the character string Y is selected.
Read in. Next, in S214 , the similarity SIM ₄ (X, Y) between the character strings X and Y is calculated. The processing performed in S214 is the same as that in FIG.
7 and FIG . In S215 , the obtained similarity and the document number are registered as a set in the document management table. In S216 , it is determined whether the similarity has been calculated for all the documents included in the document database. If the calculation has not been performed for all the documents, a document that has not been calculated is selected in S213 and read into the character string Y, and the processing up to S215 is repeated. If the calculation has been performed for all the documents, in S217 , the registered tables are rearranged in descending order of similarity. In S218 , a process of outputting one or several documents from a document having a high degree of similarity is performed.

The similarity is calculated using the synonym score Syno shown in equation (30).
It is determined by adding nymScore (T). In the following, the value in the middle of this addition is also called a score. Equation (2
As can be seen from the functions MAX in (6) and (29), it is necessary to calculate the score for the sum of the synonym scores and for the substring in which the length is changed one character at a time, and select the maximum value. is there. Therefore, the score table described with reference to FIG. 11 is used. In this embodiment, when the score table is completed, the lower right score [x_len] [y_len] becomes the similarity SIM ₄ (X, Y).

FIGS. 17 and 18 show a flow of processing for obtaining the similarity between the character string X from which the search text has been read and the character string Y from which the document in the document database has been read. First, in S231 shown in FIG. 17 , all values of the score tables score [i] [j] of the character strings X and Y are initialized to zero. next,
In S232, i indicating the beginning of extracting the partial character string from the character string X is set to 0. In S233, m indicating the length of the partial character string extracted from the character string X is set to 1.

In S234, the substring X [i,
m] exists in the synonym key of the synonym hash table. If it does not exist in the synonym key, S23
Proceed to 6. If it exists in the synonym key, S235
Then, the synonym string A corresponding to X [i, m] is extracted from the synonym dictionary hash table.

Next, in S236, j indicating the beginning of extracting the partial character string from the character string Y is set to 0. In S237, the score score [i] currently focused on in the score table
[j] is stored in a temporary variable current for convenience of processing.

In step S238, the character string is set in the same manner as in step S234.
It is determined whether the substring X [i, m] of X exists in the synonym key of the synonym hash table. If it does not exist in the synonym key, there is no need to consider the synonym score, so the process proceeds to S251 shown in FIG . If it exists in the synonym key, the processing is performed by searching for the maximum synonym score from S239 to S250 shown in FIG . This is processing corresponding to equation (27).

In S239, one synonym a is extracted from the synonym string A registered in the synonym hash table. It is confirmed from S240 to S243 whether this matches the partial character string Y [j, n] of the character string Y. In S240, n indicating the length of the partial character string extracted from the character string Y is set to 1. Then, in S241, it is determined whether or not the partial character string Y [j, n] is a character string equal to the synonym a. If not equal, S2
At 42, it is determined whether j + n <y_len, that is, whether the end of the partial character string has not reached the end of the character string Y. If this condition is met, the length of the partial character string is increased by increasing n by 1 in S243, and the determination in S241 is performed again. If this condition is not met, the synonym a is
Since it does not match [j, n], the process proceeds to S250.

If the partial character string Y [j, n] is equal to the synonym a in S241, the synonym score of the synonym a is determined in S244. This is a process for calculating the SynonymScore in Expression (30). As can be seen from the fact that the expressions (15) and (30) have the same form, when the character string score for the synonym a is obtained,
It is a synonym score. Therefore, a synonym score is obtained by the processing described above with reference to FIGS.

In S244 of FIG . 17 , the synonym score of the synonym a is set to tmp_score. Tmp_score is 0 in S245
Is determined. If 0, this synonym a is useless for searching. Therefore, in order to save the trouble of obtaining the next time tmp_score, it is deleted from the synonym hash table in S246. On the other hand, if tmp_score (a) is not 0, S
At 247, the value is added to the current score (current), and this value is stored in a temporary variable value. S248
And S249 are comparison and writing for satisfying the expression (32).

In S250, it is determined whether all synonyms included in the synonym string A have been extracted. If not all have been extracted, the processing from S239 to S249 is repeated for the remaining synonyms. If all synonyms have been extracted, the process proceeds to S251 shown in FIG .

S251 and S252, S253 and S25
4, S255 and S256 are expressed by equations (33) and (3
4) and comparison and writing to make equation (35) hold.

In S257, it is determined whether j indicating the head of extracting the partial character string from the character string Y is less than the length y_len of the character string Y. If this condition holds, S
In step 258, j is incremented by 1, and the processing from S237 to S256 is repeated.

In S259, the number m of characters of the partial character string X extracted from the character string X is increased by one. And S260
It is determined whether m is smaller than MAX_JA and i + m is smaller than the length x_len of the character string X. If this condition is satisfied, the processing from S234 to S259 is repeated. Here, MAX_JA indicates the maximum number of characters of a meaningful character string. Synonyms consisting of too long strings are not listed in the synonym dictionary. Therefore, a limit by MAX_JA is provided in order to eliminate the waste of referring to the synonym dictionary with such a long character string. In this embodiment that implements Japanese, MAX_JA is
20. It is more preferable to set this value to the longest synonym key or the length of the synonym registered in the synonym hash table without waste.

If the condition shown in S260 is not satisfied, the flow advances to S261. In S261, i indicating the beginning of extracting a partial character string from the character string X is the length x_le of the character string X
It is determined whether it is less than n. If this condition is satisfied, i is incremented by 1 in S262, and S233 to S
The processing up to 260 is repeated. If this condition is not satisfied, it means that the score table has been completed.
Then, a process of returning score [x_len] [y_len] as the similarity between the character strings X and Y is performed.

(Eighth Embodiment)

As described in the fourth embodiment,
If a bilingual dictionary is used as the synonym dictionary in the seventh embodiment, the similarity between documents in different languages can be obtained. That is, an information search device that performs cross-language information search by software can be implemented.

As can be seen from the above description of the embodiment, the character string similarity calculation method according to the first invention is the same as that of the first embodiment,
This is performed in the second embodiment, the fifth embodiment, and the sixth embodiment.

The character string similarity calculating method according to the second aspect of the present invention includes a first embodiment, a second embodiment, a fifth embodiment, and a sixth embodiment.
The embodiment is implemented using a Score function having the case shown in equation (16).

The character string similarity calculating method according to the third aspect of the present invention includes the first embodiment, the second embodiment, the fifth embodiment, and the sixth embodiment.
The embodiment is implemented by using the Score function shown in Expression (15).

A character string similarity calculation method according to the fourth invention
Are the first embodiment, the second embodiment, the fifth embodiment, and the sixth embodiment.
In the example, the expression (38) is used in accordance with the appearance concentration shown in the expression (36).
Use the Score function shown, or use the Score shown in equation (40)
It is implemented using functions.

In the character string similarity calculating method according to the fifth invention, the second and sixth embodiments are implemented by using equation (21).

The character string similarity calculating method according to the sixth aspect of the present invention includes a third embodiment, a fourth embodiment, a seventh embodiment, and an eighth embodiment.
This is implemented in the embodiment.

The character string similarity calculating method according to the seventh aspect of the present invention includes a third embodiment, a fourth embodiment, a seventh embodiment, and an eighth embodiment.
The embodiment is implemented by using the SynonymScore function shown in Expression (30).

The character string similarity calculating method according to the eighth aspect of the present invention is implemented in the fourth and eighth embodiments by using a bilingual dictionary as a synonym dictionary.

The character string similarity calculation method according to the ninth aspect of the present invention is implemented in the first embodiment and the fifth embodiment by using equation (11). Further, the third, fourth, seventh, and eighth embodiments are also implemented by using equation (27).

The character string similarity calculating apparatus according to the tenth aspect of the present invention includes a first embodiment, a second embodiment, a fifth embodiment, and a sixth embodiment.
This is implemented in the embodiment. In the fifth embodiment, the character string score calculation unit is performed in S27 shown in FIG. The matching character string similarity calculation unit is performed in S28. The arbitrary character string similarity calculation unit is implemented by repeating the processes from S32 to S37 while sequentially increasing the values of i and j. The selection unit includes S29 and S30, S
This is implemented by performing both of steps S32 to S37. In the sixth embodiment, the character string score calculator is implemented in S127 shown in FIG . The matching character string similarity calculation unit is performed in S128. The arbitrary character string similarity calculator calculates the S while sequentially increasing the values of i and j.
This is implemented by repeating the processes from 132 to S137. The selection unit is implemented by performing both of S129 and S130 and S132 to S137.

The character string similarity calculating apparatus according to the eleventh aspect of the present invention comprises a first embodiment, a second embodiment, a fifth embodiment, and a sixth embodiment.
The embodiment is implemented by using the Score function shown in Expression (15).

A character string similarity calculating apparatus according to the twelfth invention
Are the first embodiment, the second embodiment, the fifth embodiment, and the sixth embodiment.
In the example, the expression (38) is used in accordance with the appearance concentration shown in the expression (36).
Use the Score function shown, or use the Score shown in equation (40)
It is implemented using functions.

A character string similarity calculating apparatus according to a thirteenth aspect of the present invention includes a third embodiment, a fourth embodiment, a seventh embodiment, and an eighth embodiment.
This is implemented in the embodiment. In the seventh embodiment, the synonym score calculation unit is performed in S244 shown in FIG . The matching character string similarity calculation unit is performed in S247. The arbitrary character string similarity calculating unit sequentially increases the values of i and j, and proceeds from S251 to S25 shown in FIG.
This is implemented by repeating the processing up to 6. Selecting unit, S248 and S249 shown in FIG. 17, S shown in FIG. 18
251 to S256 are performed.

The character string similarity calculating apparatus according to the fourteenth aspect of the present invention includes a third embodiment, a fourth embodiment, a seventh embodiment, and an eighth embodiment.
The embodiment is implemented by using the SynonymScore function shown in Expression (30).

A character string similarity calculating apparatus according to a fifteenth aspect is implemented in the first to eighth embodiments.

The character string similarity calculation program according to the sixteenth invention is implemented in the fifth embodiment and the sixth embodiment. In the fifth embodiment, the comparison setting process is performed by sequentially increasing i and j in S41 and S39 shown in FIG. The common substring identification process is as follows:
This is performed in S25, S26, and S31. The character string score setting process is performed in S27. The character string score adding process is performed in S28. The progress process is performed by both S29 and S30, and S32 to S37. In the sixth embodiment,
Contrast setting process is performed by increasing the i and j in sequence by S141 and S139 shown in FIG. 14. The common part character string specifying process is performed in S125, S126, and S131. The character string score setting process is performed in S127. The character string score adding process is performed in S128. The progress processing step is performed by repeating both of the above steps and performing both steps S129 and S130 and steps S132 to S137.

The character string similarity calculation program described in the seventeenth invention is implemented by the fifth and sixth embodiments by performing S56 shown in FIG. 8 in order to determine a character string score. ing.

A character string similarity calculation program according to the eighteenth aspect of the present invention.
The program is the same as the fifth and sixth embodiments except that the character string
In order to determine the score, it is necessary to respond to the determination in S87 shown in FIG.
First, S88 is performed.

The character string similarity calculation program according to the nineteenth aspect of the present invention is implemented by executing the processing shown in FIGS. 9 and 10 in the fifth and sixth embodiments .

The character string similarity calculation program according to the twentieth invention is implemented in the seventh and eighth embodiments. In each embodiment, the comparison setting process includes:
This is implemented by sequentially increasing i and j by S258 and S262 shown in FIG . The synonym identification process is shown in FIG.
S234 and S241 shown in FIG. The synonym score setting process is performed in S244. The synonym score adding process is performed in S247. In the progress processing process, S248 and S2
49, and is performed by performing both of S251 to S256 shown in FIG .

The character string similarity calculation program according to the twenty-first invention is implemented by the seventh and eighth embodiments by performing S56 shown in FIG. 8 in order to obtain a synonym score. ing.

The character string similarity calculation program described in the twenty-second invention is implemented by the seventh embodiment and the eighth embodiment by executing the processing shown in FIGS. 9 and 10.

The character string similarity calculation program according to the twenty- third aspect is provided so that the fifth to eighth embodiments allow the expressions (32) to (35) to be satisfied by using the score table shown in FIG. It is implemented by comparing and writing.

The text search program according to the twenty-fourth invention is implemented in the fifth to eighth embodiments.

The technical scope of the present invention is not limited to these embodiments, but includes the scope equivalent to the claims and can be variously modified without changing the gist of the invention. is there.

[0202]

According to the first aspect of the present invention, the degree of similarity can be obtained by focusing on the common partial character string, which is suitable for the order of the two character strings. That is, the similarity is calculated in consideration of the appearance order of the character strings.

According to the second invention, the longer the common partial character string, the higher the weight may be.
Therefore, the similarity tends to increase. Therefore, a longer one is selected as the common partial character string, and the obtained similarity becomes a more appropriate value.

According to the third aspect, it is possible to further obtain a similarity reflecting the amount of information. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

According to the fourth aspect, the degree of appearance concentration is further improved .
Can be obtained. Concentration of appearance
By using, more effective character strings can be
Can improve the search accuracy
You .

According to the fifth aspect of the present invention, it is not necessary to divide a common partial character string, so that the amount of calculation can be reduced.

According to the sixth aspect, the degree of similarity can be obtained by focusing on the partial character string which is suitable for the order of each of the two character strings and is an element of the synonym dictionary.
Therefore, even when a synonym is used, the similarity can be appropriately obtained.

According to the seventh aspect , a similarity reflecting the amount of information can be obtained. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

[0209] According to the eighth aspect , the similarity between character strings in different languages can be obtained.

According to the ninth aspect , further, a partial character string is obtained such that the similarity has the maximum value. Therefore,
A reasonable similarity is obtained in that the calculated similarity is the maximum value.

According to the tenth aspect , it is possible to realize a device that conforms to the order of two character strings and obtains a similarity by focusing on a common partial character string. That is, the similarity in which the appearance order of the character strings is considered is obtained by this device.

According to the eleventh aspect , it is possible to realize a device for obtaining a similarity reflecting the amount of information. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

According to the twelfth aspect, the appearance and concentration
A device that obtains a similarity reflecting the degree can be realized. Appearance collection
By using medium, more effective character strings can be searched
Can give more weight and improve search accuracy
Can be.

According to the thirteenth aspect , it is possible to realize an apparatus which conforms to the order of two character strings and obtains a similarity by focusing on a partial character string which is an element of a synonym dictionary. Therefore, even if synonyms are used,
With this device, the similarity can be determined appropriately.

[0215] According to the fourteenth aspect, an apparatus for obtaining a similarity reflecting the amount of information can be realized. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

According to the fifteenth aspect , it is possible to realize an apparatus for searching a document database for a document having a high similarity to a search sentence.

According to the sixteenth aspect , it is possible to realize a program that matches the order of two character strings and obtains a similarity by focusing on a common partial character string. That is, the similarity in consideration of the appearance order of the character strings is obtained by this program.

According to the seventeenth aspect , it is possible to realize a program for obtaining a similarity reflecting the amount of information. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

According to the eighteenth aspect, the appearance and concentration
A program that obtains a similarity reflecting the degree can be realized.
Character strings that are more effective in searching by using the appearance concentration
Can give more weight and improve search accuracy
Can be

According to the nineteenth aspect , a program that performs high-speed processing using a suffix file can be realized.

According to the twentieth aspect , it is possible to realize a program that conforms to the order of two character strings and obtains a similarity by focusing on a partial character string that is an element of a synonym dictionary. Therefore, even when synonyms are used, the similarity can be appropriately obtained by this program.

According to the twenty-first aspect , a program for obtaining a similarity reflecting an information amount can be realized. Therefore, it becomes easy to apply the knowledge of the information theory regarding the amount of information.

According to the twenty-second aspect , a program that performs high-speed processing using a suffix file can be realized.

According to the twenty-third aspect, a program which can be processed at a high speed by the dynamic programming technique can be realized.

According to the twenty-fourth aspect , an apparatus for searching a document database for a document having a high similarity to a search sentence can be realized.

Next, search performance according to the present invention will be described.
In order to compare the closest ones between the prior art and the embodiment of the present invention, a character weight DP similarity SIM ₂ and a character string weight DP
The search performance of the similarity SIM ₃ is shown in FIG . In order to unify the weights and compare them, for any similarity, a Score function obtained by the flow shown in FIG. Needless to say, when the character weight DP similarity SIM ₂ is obtained, what is given to the character string a in S51 shown in FIG. 8 is a character string (character) of length 1.

When the search text is variously changed, related documents (documents to be correctly searched) in the document database
Also vary. This is shown on the horizontal axis. The vertical axis shows the validity rate (precision) and the recall rate (recall). The legitimate rate is
This is the ratio of the number of relevant documents in the retrieved documents. If only relevant documents can be searched without confusing unrelated documents, the validity rate is 1. The recall is
It is the ratio of the number of relevant documents retrieved among all relevant documents in the document database. If relevant documents can be retrieved from the document database without omission,
The recall is 1.

The legitimacy rate 201 of the similarity SIM ₂ according to the prior art
As compared with the above, the validity rate 202 of the similarity SIM ₃ according to the present invention is greatly improved. Similarly, the similarity SIM ₂ according to the prior art
Of the SIM ₃ according to the present invention as compared with the recall 203 of
02 has also improved significantly. For example, when the number of relevant documents is 400, the search performance is about four times as high in both the correct rate and the recall rate. As described above, according to the present invention, search performance can be dramatically improved.

[Brief description of the drawings]

FIG. 1 is a diagram showing an embodiment of a document search device according to the present invention.

FIG. 2 is a diagram showing another embodiment of the document search device according to the present invention.

FIG. 3 is a diagram showing another embodiment of the document search device according to the present invention.

FIG. 4 is a diagram showing another embodiment of the document search device according to the present invention.

FIG. 5 is a diagram showing an embodiment of a computer system for performing a sentence search according to the present invention.

FIG. 6 is a flowchart of a process for performing a sentence search according to the embodiment of this invention.

FIG. 7 is a flowchart of a process for obtaining a similarity according to the embodiment of the present invention.

FIG. 8 is a flowchart of a process for obtaining a character string score or a synonym score.

FIG. 9 is a flowchart of a process for obtaining the number of documents.

FIG. 10 is a flowchart of a process for obtaining the number of appearances.

FIG. 11 is a diagram showing a score table.

FIG. 12 is a flowchart of a process for obtaining a character string score.
It is.

FIG. 13 is a flowchart of a process for obtaining the number of documents.
You.

FIG. 14 is a flowchart of a process for obtaining a similarity according to another embodiment of the present invention.

FIG. 15 is a flowchart of a process for performing a sentence search according to another embodiment of the present invention.

FIG. 16 is a flowchart of a process for creating a synonym dictionary.

FIG. 17 is a first half flowchart of a process for obtaining a similarity according to another embodiment of the present invention;

FIG. 18 is a second half flowchart following FIG. 17 ;

FIG. 19 is a diagram showing search performance according to the related art and search performance according to the present invention.

[Explanation of symbols]

10: Document database 10a, 10b, 10c: Document 11: Character string input unit 12: Search control unit 13: Search result output unit 14, 16, 18, 43, 46, 51, 52, 53, 5
5, 56, 57, 73, 81: Similarity calculating unit 15, 17, 19: Recursive execution control unit 20: Synonym dictionary 21, 24: Matching character string similarity calculating unit 22, 25, 29: Arbitrary character string similarity Degree calculation units 23, 26, 30, 33, 54, 58, 63, 84: maximum value selection unit 28: synonym similarity calculation unit 31: character string separation control unit 32: character string separation similarity calculation unit 34: match Character string determination units 42, 45: Character string score calculation unit 41: Character string separation units 44, 47, 74: Addition unit 61: Synonym separation control unit 62: Synonym separation similarity calculation unit 71: Synonym separation unit 72 : Synonym score calculation unit 91: Bilingual dictionary 101: Display 102: Printer 103: Keyboard 104: Floppy disk device 105: CD-ROM device 106: Read-only memory (ROM) 107: La Dam access memory (RAM) 108: magnetic disk device 109: a central processing unit (CPU) 110: Communication interface 111: bus 112: Floppy disk 113: CD-ROM 114: Communication Network 201: justification rate by SIM ₂ 202: SIM ₃ 203: Recall by SIM ₂ 204: Recall by SIM ₃

Claims (24)

[Claims] 1. A method for calculating a degree of similarity between two character strings, comprising: a plurality of partial character strings conforming to an order in each of the two character strings, the partial character strings being common to the two character strings. A character string similarity calculation method comprising: determining a weight for each of the plurality of obtained partial character strings; and calculating a similarity by summing the weights. 2. The weight of the partial character string may be heavier than the sum of the weights of the partial character strings obtained by dividing the partial character string into two or more. The character string similarity calculation method described in 1. 3. The method according to claim 1, wherein one of the two character strings is selected from a document database, and the weight corresponds to an information amount of a partial character string in the document database. The character string similarity calculation method described in 1. 4. The method according to claim 1, wherein the weight is stored in the document database.
Of substring information and the concentration of occurrence of substrings
4. The character string according to claim 3, wherein the character string corresponds to a degree.
Similarity calculation method. 5. As the degree of similarity is the highest, the string similarity calculation method according to any one of claims 1 to 4, characterized in that for obtaining a plurality without dividing the partial string. 6. A method for calculating the similarity between two character strings, comprising: a plurality of partial character strings that match the order of each of the two character strings and are included in a synonym dictionary element. A character string similarity calculating method, wherein a similarity is calculated by determining weights for elements of the synonym dictionary corresponding to the plurality of obtained partial character strings, and summing the weights. 7. A are those where one of the two strings is selected from the document database, claim the weight, characterized in that corresponding to the amount of information elements synonym dictionary in the document database 6 The character string similarity calculation method described in 1. Wherein said two strings are represented in different languages, the elements of the synonym dictionary, a character string according to claim 6 or 7, characterized in that it comprises a synonym of said different languages Similarity calculation method. 9. As the degree of similarity is the highest, the string similarity according to any one of claims 1 to 4, 6 8, characterized in that for obtaining a plurality to permit divided substrings Calculation method. 10. A character string similarity calculating apparatus for calculating a similarity between two character strings, comprising: a character string score calculating unit for determining a weight of a partial character string common to the two character strings; A matching character string similarity calculating unit that calculates the similarity by adding the similarities of the partial character strings; and the highest similarity among the similarities of the character strings reduced by one character at a time in one or both of the two character strings. A character string similarity calculating apparatus, comprising: an arbitrary character string similarity calculating unit that calculates the similarity; and a selecting unit that selects the highest similarity among the obtained similarities. 11. are those where one of the two strings is selected from the document database, according to claim 10, wherein the weights, characterized in that corresponding to the information amount of the partial strings in the document database Character string similarity calculation device. 12. The method according to claim 11, wherein the weight is stored in the document database.
Of information on substrings in substrings and appearances of substrings
The statement according to claim 11, characterized in that it corresponds to medium.
Character string similarity calculation device. 13. A character string similarity calculating apparatus for calculating a similarity between two character strings, wherein the character string similarity calculating apparatus calculates a partial character string corresponding to a partial character string included in a synonym dictionary element. A synonym score calculating unit that determines the weight of the elements of the synonym dictionary; a matching character string similarity calculating unit that calculates the similarity by adding the similarity of the remaining partial character strings to the weight; An arbitrary character string similarity calculating unit that obtains the highest similarity among the similarities of the character strings reduced by one character in one or both of them; a selecting unit that selects the highest similarity among the obtained similarities; A character string similarity calculation device characterized by having: 14. are those where one of the two strings is selected from the document database, claim the weight, characterized in that corresponding to the amount of information elements synonym dictionary in the document database 13
The character string similarity calculating device according to item 1. 15. A sentence search apparatus for selecting a document similar to a search sentence from a document database, wherein the search sentence and a document in the document database are selected.
A character string similarity calculating apparatus according to claim 10 , wherein similarities are obtained as two character strings, and a document having a high calculated similarity is selected from a document database. Search device. 16. A character string similarity calculation program for calculating a similarity between two character strings, comprising: a comparison setting step of sequentially setting a part to be compared with the two character strings; and a part starting from the part to be compared. A character string, a common partial character string specifying step of specifying a partial character string common to the two character strings; a character string score setting step of determining a weight of the specified partial character string; A computer-readable recording medium that records a character string similarity calculation program for causing a computer to execute a character string score adding step of adding a character string score to a computer, and a progress processing step of performing these steps to increase similarity. 17. are those where one of the two strings is selected from the document database, according to claim 16, wherein the weights, characterized in that corresponding to the information amount of the partial strings in the document database A computer-readable recording medium on which a character string similarity calculation program is recorded. 18. The method according to claim 18, wherein the weight is stored in the document database.
Of information on substrings in substrings and appearances of substrings
The sentence according to claim 17, characterized in that it corresponds to medium.
Computer reading recorded character string similarity calculation program
A removable recording medium. 19. The character string similarity calculation program according to claim 17 , wherein the character string score setting step obtains the information amount by using a suffix file. A computer-readable recording medium on which a program is recorded. 20. A character string similarity calculation program for calculating a similarity between two character strings, comprising: a comparison setting step of sequentially setting a part to be compared with the two character strings; and a part starting from the part to be compared. A synonym identification step of identifying a character string and a substring included in a synonym dictionary element; and a synonym score setting step of determining a weight of an element of the synonym dictionary corresponding to the identified substring. A character string similarity calculation program for causing a computer to execute a synonym score adding step of adding the weight to the similarity, and a progress processing step of advancing these steps to increase the similarity. Computer readable recording medium. 21. are those where one of the two strings is selected from the document database, claim the weight, characterized in that corresponding to the amount of information elements synonym dictionary in the document database 20
A computer-readable recording medium recording the character string similarity calculation program described in 1. above. 22. The character string similarity calculation program according to claim 21 , wherein the synonym score setting step obtains the information amount using a suffix file. A computer-readable recording medium that has been recorded. 23. A computer-readable program storing a character string similarity calculation program according to claim 16 , wherein the progress processing step is performed by a dynamic programming method. Recording medium. 24. A sentence search program for selecting a document similar to a search sentence from a document database, wherein the search sentence and a document in the sentence database are
A similarity is calculated by the character string similarity calculation program according to any one of claims 16 to 23 as two character strings, and the computer is caused to select a document having a high calculated similarity from a document database. Computer-readable recording medium on which a document search program for recording is stored.