JP2002297660A - Method, device, program, and recording medium for character similarity calculation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To speed up document retrieval by selecting a partial character string used for similarity calculation. SOLUTION: An input character string X and a document Y in a document database are regarded as two character strings and their similarity is calculated. Partial character strings cut out of the input character string are sorted according to their appearance frequencies and recorded in a partial character string management table. Then matching information is gathered as to the respective partial character strings in the partial character string management table and recorded in a matching information management table. A list regarding the document Y is taken out of the table and the similarity to the input character string X is calculated. The document number and the similarity are recorded in a pair in a document management table. Those processes are repeated for all documents. Lastly, the document management table is rearranged in the decreasing order of the similarity and a document having high similarity is selected as a retrieval result from the database.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to similarity determination between two character strings, and more particularly, to information retrieval.
It is suitable for use in determining the similarity between an input character string and a document registered in a database.

[0002]

2. Description of the Related Art Information retrieval for extracting a desired document from a document database is often performed. In such information retrieval, a document is treated as a set of character strings obtained by combining words including a plurality of characters. Then, the information search is performed by comparing the search character string with the character strings in the search target document, and selecting one or a plurality of characters having a high degree of similarity. The similarity between these character strings can be roughly divided into
There are two methods, a method using morphological analysis and a method for finding a match between partial character strings of length n (hereinafter, referred to as n-grams).

A method using morphological analysis is described, for example, in Gera
rd Salton and Christopher Buckley, Term-Weighting
Approaches in Automatic Text Retrieval, Informatio
n Proceeding and Management, 24, pp.513-523, 1988.
Is disclosed. The basic procedure for obtaining the similarity between two character strings by this method is as follows. First, both character strings are decomposed into word strings by morphological analysis using a dictionary and grammatical knowledge. Next, compare both word strings,
Find matching words. Then, weights are set for the matching words. The weight is then added for all matching words. The sum obtained as a result of this addition is the similarity based on the morphological analysis.

[0004] The method using morphological analysis has an essential problem that if the accuracy of the morphological analysis itself is low, the information retrieval ends abnormally. To improve the accuracy of morphological analysis,
Word dictionaries and grammar rules must be large, making it difficult to easily use information retrieval. Furthermore, for documents in which buzzwords, coined words, and technical terms used only in limited fields appear, the time and effort required to maintain a word dictionary is a large burden.

[0005] Next, the method using the n-gram is, for example, Ya
sushi Ogawa and Toru Matsuda, Overlapping statisti
cal word indexing: A new indexing method for Japan
esetext, In proceeding of SIGIR'97, Philadelphia P
A, USA, pp. 226-234, 1997. The basic procedure for obtaining the similarity between character strings by this method is as follows. First, a partial character string of n characters included in both character strings is obtained. Next, a weight is set for the common partial character string. Then, this weight is added for all matching parts. The sum obtained as a result of this addition is the similarity based on n-grams.

Regarding the setting of the weight of a common partial character string, a large value is given to a character string that appears intensively in a specific document and does not appear in other documents. Conversely, strings that appear in many documents are given only small values. This reflects that character strings appearing in many documents are not elements that characterize documents, and cannot be used effectively in searching.

The method based on n-grams does not require morphological analysis, and therefore can deal with unknown words such as new technical terms and can be used easily.

[0008] Among the methods using n-grams, particularly, in calculating the similarity using a character string of length 2 (hereinafter, referred to as bigram), not all the extracted bigrams are used, but no hiragana is included. limited to bigram,
There is a method of calculating coincidence between character strings and calculating similarity. This is because bigrams including Hiragana are highly likely to appear in many documents in the document database, and the probability of becoming a character string characterizing each document is extremely small. If you include bigrams including hiragana in the comparison of character strings, the amount of calculation will only increase,
It is recognized that search accuracy cannot be expected to significantly improve.

[0009]

As described above, the method based on n-grams has many advantages in comparison with the method based on morphological analysis, and is often used in the field of information retrieval. n
The problem with the Gram method is that the amount of calculation increases as the size of the document database increases, and it takes time to obtain search results.

By the way, the partial character strings cut out from the character string are not necessarily effective in calculating the similarity, that is, they are included in many documents in the document database, so that the weight is small and given to the similarity. It is thought that many substrings with small effects are included. Therefore, the method of checking whether all n-grams extracted from a character string match each other is not efficient in terms of calculation time.

The method of not treating a character string including hiragana as matching information in the extracted bigram is efficient in terms of calculation time, but even a character string that is originally effective for retrieval is used. It will be truncated uniformly. As a result, there is a problem that search accuracy is reduced.

The present invention solves the above-mentioned problem in the n-gram method, and provides a method that achieves both reduction in calculation time and improvement in search accuracy.

[0013]

According to a first aspect of the present invention, there is provided a method for calculating a similarity between two character strings, wherein the effect on the similarity calculation among partial character strings cut out from the first character string is provided. Collecting the matching information with the second character string for the partial character string selected on the basis of, calculating the weight of the matching partial character string from the matching information, and calculating the similarity based on the weight. Features. As described above, when obtaining matching information between character strings, n-grams that are effective in calculating the similarity are selected by estimating the effect on the calculation of the similarity instead of treating all the n-grams cut out. ing.

With such a configuration, it is possible to reduce the calculation time while maintaining the search accuracy almost equal to the search accuracy by the method of not selecting.

According to a second aspect of the present invention, in the method for calculating the similarity between two character strings, a part selected based on an effect on similarity calculation among partial character strings cut out from the first character string. For character strings, matching information with the second character string is collected, and among partial character strings included in the matching information, weights of the partial character strings in which the order of appearance in the first and second character strings matches The similarity is calculated based on As described above, in calculating the similarity between character strings, a partial character string whose order of appearance in each character string matches is further selected from the partial character strings recorded in the matching information, and the weight is determined by the similarity degree. Is used to calculate

With such a configuration, the partial character string for calculating the weight is limited, and the calculation can be further speeded up while maintaining the search accuracy.

According to a third aspect of the present invention, in the first or second aspect of the present invention, the selection of the partial character string is performed such that the partial character string cut out from the first character string appears in the second character string. It is characterized in that it is performed taking the number of times into account.

With such a configuration, partial character strings that are effective in calculating the similarity can be efficiently selected, and the calculation time can be further reduced while maintaining the search accuracy.

The invention according to claim 4 is the invention according to claims 1 to 3.
In the invention according to any one of the above, as the matching information between the character strings, the length of the partial character string, the location of the partial character string in the first character string, the location of the partial character string in the second character string, And a sequence number indicating the number of matches in the second character string.

With such a configuration, not only a method of calculating the similarity by adding the weights of the matched n-grams but also many similarity calculating methods can be used together.

According to a fifth aspect of the present invention, in the character string similarity calculating device for calculating the similarity between two character strings,
A partial character string selecting unit that selects based on an effect on similarity calculation among partial character strings cut out from the character string of
A matching information collecting unit that collects matching information with the second character string; and a similarity calculating unit that calculates a weight of the partial character string that matches based on the matching information, and calculates a similarity by summing the weights. And a character string similarity calculating device.

With such a configuration, it is possible to realize a device that shortens the calculation time while maintaining the search accuracy substantially equal to the search accuracy by the method of not selecting.

According to a sixth aspect of the present invention, in the character string similarity calculating device for calculating the similarity between two character strings,
A partial character string selection unit that selects a partial character string extracted from the character string based on the effect on similarity calculation, and a match information collection unit that collects match information of the selected partial character string with a second character string Part, a matching partial character string selecting means for selecting a matching partial character string from among the partial character strings included in the matching information, the order of appearance in each character string, and a matching partial character string attached to the matching partial character string. A character string similarity calculation device, comprising: a similarity calculation unit that sums weights.

With such a configuration, a partial character string for calculating the weight is limited, and an apparatus can be realized in which the calculation speed can be further increased while maintaining the search accuracy.

According to a seventh aspect of the present invention, there is provided a character string similarity calculating program for calculating a similarity between two character strings, wherein the computer is configured to execute a partial character string cut out from the first character string. Selecting means for selecting based on the number of appearances in the character string; matching information collecting means for collecting matching information of the selected partial character string with a second character string; and a partial character string matching based on the matching information Calculate the weight of
A character string similarity calculation program that functions as similarity calculation means for calculating a similarity by summing the weights.

With such a configuration, the computer can be made to function as means for shortening the calculation time while maintaining the search accuracy almost equal to the search accuracy by the method of not selecting.

An eighth aspect of the present invention is a character string similarity calculation program for calculating the similarity between two character strings, wherein the computer is configured to execute the similarity degree calculation on the partial character string cut out from the first character string. Selecting means for selecting based on the effect on the calculation;
The length of the partial character string, the first
Matching information collecting means for collecting the occurrence position of the partial character string in the character string, the occurrence position of the partial character string in the second character string, and the sequence number indicating the number of the match in the second character string; Functioning as similarity calculating means for calculating a similarity based on a weight given to a partial character string whose order of appearance in each character string matches from among partial character strings included in the matching information. This is a character string similarity calculation program as a feature.

With such a configuration, the computer can be made to function as a means for further increasing the calculation speed while maintaining the search accuracy by limiting the partial character string for calculating the weight.

According to a ninth aspect of the present invention, there is provided a computer-readable recording medium storing the program according to the seventh or eighth aspect.

With such a configuration, the recorded function can be realized at a necessary place.

[0031]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention relates to a method for calculating similarity between character strings using n-grams, and an apparatus, a program, and a recording medium for realizing the method. Although it is assumed that the similarity between the input character string and a plurality of documents registered in the database is calculated, other applications are also possible.
When finding a matching part between character strings, instead of obtaining an n-gram common to the input character string and the document for each document in the database, an n-gram is cut out from the input character string and A method is used in which a document including an n-gram is efficiently searched from a database by using a suffix file.

It is considered that the n-grams cut out from the input character string include many n-grams having little influence on the calculation of the similarity. The number of partial character strings cut out from the input character string is, for example, m
And if the partial character string is limited to bigram which is two characters, m
It becomes -1. Therefore, if matching information is obtained for all of the extracted bigrams without selecting the partial character strings, the calculation time increases as the input character string length m increases. Therefore, in the present invention, n-grams are selected by estimating the effect on the calculation of similarity from the extracted n-grams, and only a constant number of n-grams are applied to the collection of coincidence information.

In selecting a partial character string, it is necessary to select a partial character string having a large weight added to the similarity in order to improve search accuracy. Since the weight is determined by the number of documents in the database including the partial character string (hereinafter referred to as df), it is natural to select based on the value of df. However, in one aspect of the present invention, it is recommended to select a partial character string based on the number of occurrences of an n-gram in a document in a database (hereinafter, referred to as occurrence frequency or tf) instead of df. Is done.

Although tf can be calculated directly and the calculation time hardly changes even if its value increases, to calculate df, the duplication in the document must be recalculated after calculating tf. On the other hand, when the value of tf is large, the calculation time becomes long. On the other hand, there is a large correlation between tf and df, and df
Even if tf is used instead of f, it can be expected that search accuracy is not affected.

The collection of the matching information is performed on each of the selected n-grams in the following manner. First, the weight of the n-gram is calculated. Next, a document containing the n-gram is obtained from the entire document database, and the appearance location of the n-gram in the document, the appearance location of the n-gram in the input character string, the length of the n-gram, and the number of matches in the document And the weight of the n-gram is recorded as matching information.

Normally, the obtained coincidence information is directly added to the weight without calculating and managing the similarity, and the similarity is calculated. In the present invention, the matching n-gram is obtained by recording and managing the similarity. In addition to the method of calculating the similarity by adding the weights of the above, the present invention can be applied to many similarity calculation methods while maintaining high speed.

The similarity between the input character string and the document in the database is calculated by adding the weights given to the matched n-grams. However, even if the matching n-gram appears more than once in the same document, its weight is 1
Only degrees are added. That is, just because the matching n-gram appears twice does not mean that the added weight is doubled. Rather, the weight of the n-gram stored as the matching information differs for each document according to the number of times that the n-gram appears in the document (hereinafter, dtf).
The greater the value of f, the greater the weight given. This weight is given by dtf =
It is also defined as 0, and a weight of 0 or less is added to all documents that do not include the n-gram. This is based on the idea that not including an n-gram indicates that there is no similarity, and it can be said that the weight to be added is a numerical value of the degree of dissimilarity. Accordingly, the value indicating the similarity takes a real value including a negative value, and the greater the value on the number line, the higher the similarity with the input character string.

The method of calculating the similarity is not limited to the above-described method of adding the weights assigned to the matched n-grams, and other similarity calculation methods can be applied. . One example is n that matches the order in which each character string appears in the matched n-grams.
In this method, only the gram is selected, and the value obtained by summing the weights assigned to the n-grams is used as the similarity.

A method of calculating the similarity in consideration of the appearance order of character strings has been proposed. However, since they calculate the weight for the substrings of all matching lengths, the amount of calculation increases as compared with the method of adding the weights of the matched n-grams. growing.

In the present invention, weights are assigned to the selected n-grams, and the weight is added to the similarity only when the n-grams match. Therefore, it is only necessary to consider the match and the appearance order only for the selected partial character strings, so that the calculation can be performed at a higher speed as compared with the conventional method. The effect is small. In this method, to efficiently find the combination with the highest similarity among the combinations of matching n-grams,
The similarity is calculated using dynamic programming (hereinafter, referred to as DP). In the following, a method for obtaining n-gram DP similarity using DP will be described.

Α, β, γ, δ are character strings of length 0 or more,
ξ, ζ, η are character strings of length 1 or more, and “” is an empty character string. A character string (for example, α) connecting a plurality of character strings (for example, ξ and γ) is represented by successively writing symbols of the character strings that are elements. (Eg, α = ξγ).

The n-gram DP similarity SimDP is obtained by recursively applying the following equation according to the matching pattern for the character string portion of the argument. First, when both are empty character strings, SimDP ("", "") = 0 (1). Otherwise, SimDP (α, β) = MAX (SimDPs (α, β), SimDPg (α, β)) (2) Here, SimDPs (α, β) is expressed as follows: ξ is a character string recorded in a list regarding α and β in the matching information management table, and α = ξγ, β = ξδ, SimDPs (α, β) = MAX (Score (ξ) + SimDP (γ, δ)) (3) is obtained by calculating for all ξ. If there is no such character string ξ, SimDPs (α, β) = 0.0 (4). Score (ξ) is a function that returns the weight of ξ recorded in the matching information management table.

If す る と and η are the largest character strings that satisfy α = ζγ and β = ηδ and do not have a common part with the character strings recorded in the lists related to α and β in the coincidence information management table, SimDPg (α, β) is obtained by SimDPg (α, β) = MAX (SimDP (α, δ), SimDP (γ, β), SimDP (γ, δ)) (5). This expression is equivalent to the character strings (α and δ, γ and β, γ and β, and γ) that are obtained by removing one of ζ and η or both ζ and η from the two character strings (ζγ and ηδ).
This means that the highest similarity among the similarities between γ and δ) is adopted.

By applying the above expression recursively, a partial character string conforming to the order of each of the two character strings is obtained from the partial character strings recorded in the matching information management table, and the similarity is obtained. The degree is maximum.

The n-gram DP similarity shown above is not a character (a character string of length 1) but n, as shown in equation (3).
A similarity calculation method that takes into account character continuity by giving weights in consideration of matching with a gram (character string of length n). In addition, instead of considering the match between the character strings for all the partial character strings, the character strings are limited to n-grams recorded in the match information management table, and are recorded in the match information management table as shown in Expression (5). The part irrelevant to the set partial character string can be removed without the need to determine the presence or absence of a match. Thus, no matter how long the length of the character string for which the similarity is to be calculated, only the part related to the partial character string recorded in the match information management table needs to be considered. The similarity can be calculated at a higher speed than in a conventional method that considers the matching between character strings.

(First Embodiment) First, an embodiment of a method of calculating the similarity between character strings using n-grams will be described. FIG.
Is an example of a document search device that searches for a document having the highest similarity to the input character string based on the selected partial character string. This document search apparatus includes a document database 10, a character string input unit 11, a partial character string selection unit 13, a coincidence information collection unit 14, a similarity calculation unit 15, a similarity calculation control unit 16, and a search result output unit 12. It is configured.

A plurality of documents 10a, 10b,..., 10c to be searched are registered in the document database 10. For the search, a keyword, a word, a phrase, a sentence, a sentence, and the like are input (hereinafter, referred to as a search sentence).
The character string input unit 11 provides the search sentence to the partial character string selection unit 13 as a character string X.

The partial character string selection section 13 is a character string input section 1
Cut out a partial character string from the character string X given from 1,
After calculating the appearance frequency tf, a constant number is extracted from the one with the smaller calculated tf and registered in the partial character string management table T1. The partial character string selection unit 13 includes a partial character string cutout control unit 31, a partial character string cutout unit 32, a character string appearance frequency calculation unit 33, and a partial character string registration unit 34. The partial character string cutout control unit 31
2 controls which partial character string is cut out. The partial character string extracting unit 32 extracts a partial character string x from the character string X. The character string appearance frequency calculation unit 33 calculates an appearance frequency tf of the partial character string x in the document database 10. The partial character string registration unit 34 selects a constant number of the extracted partial character strings in ascending order of the value of tf, and sets the partial character string management table T
Register to 1.

The matching information collecting unit 14 calculates the weight given to the character string for each partial character string registered in the partial character string management table T1, detects the appearance position of each partial character string in the document database, As the matching information, the occurrence location in the document, the occurrence location in the input character string, the length of the character string of the partial character string, the sequence number indicating the number of the match in the document, and the weight of the character string are described in the matching information management table T2. To record. The match information collection unit 14 includes a match information collection control unit 41, a character string appearance location search unit 42, a character string weight calculation unit 43, a match information registration control unit 44, and a match information registration unit 45. The matching information collection control unit 41 extracts the partial character strings a registered in the partial character string management table T1 one by one, and gives them to the character string appearance location search unit 42 and the character string weight calculation unit 43. The character string appearance location search unit 42 determines, for all occurrence locations of the given partial character string a in the document database, the appearance location, the length of the partial character string a, and a sequence number indicating the number of matches in the document. Ask. The character string weight calculator 43 calculates the weight given to the given character string a. The matching information registration control unit 44 selects the occurrence positions of the partial character string a one by one, and determines the occurrence position in the document, the occurrence position in the input character string, the length of the partial character string a, the sequence number in the document, the partial character string The combination with the weight of a is given to the coincidence information registration unit 45. The matching information registration unit 45 registers each received pair as matching information in a list of corresponding document numbers in the matching information management table T2.

The similarity calculation control unit 16 extracts a list relating to a certain document Y from the coincidence information management table T2, and supplies the list to the similarity calculation unit 15.

The similarity calculator 15 calculates the similarity between X and Y from the given list of matching information. The similarity calculation unit 15 includes a character string weight addition control unit 51 and a character string weight addition unit 52. The character string weight addition control unit 51 selects one piece of matching information from the list of matching information, and if the sequence number of the matching information is 1, the weight scor of the character string
e is given to the character string weight adding unit 52. The character string weight adding unit 52 adds the given weight to the similarity Sim (X, Y) between X and Y.

The search result output unit 12 selects and outputs a document having the highest similarity. At this time, documents having a similarity equal to or higher than a certain value or documents from a higher rank to a certain order may be output together.

(Second Embodiment) An embodiment in which a text search according to the present invention is implemented by software will be described below.

FIG. 2 shows an example of a computer system used for executing a sentence search. The computer system includes a display 101, a printer 102, a keyboard 103, a floppy (R) disk device 104, a CD-ROM (Comp
act Disk-Read Only Memory (Device) 105, Read Only Memory (hereinafter referred to as ROM) 10
6. Read / write random access memory (Random A)
ccess Memory. Hereinafter, a RAM 107, a magnetic disk device 108, and a central processing unit (Central Processing Unit).
Hereinafter, CPU) 109, communication interface 110,
And a bus 111 for connecting them. The floppy (R) disk device 104 reads and writes a floppy (R) disk 112, and stores a CD-ROM.
The device 104 reads the CD-ROM 113. The computer system is connected to a communication network 114 via a communication interface 110.

A sentence search program for implementing the present invention is:
It is stored in the ROM 106. Alternatively, the text search program may be stored in the floppy (R) disk 112, the CD-ROM 113, or the magnetic disk device 108, transferred to the RAM 107, and then executed by the CPU 109. The CPU 109 executes the text search program using the RAM 107 as a work area. If necessary, the magnetic disk device 108 may be used as a work area. The instruction to execute the sentence search program is sent to the keyboard 10
3 and the execution result is the display 101 or
Output to the printer 102. It goes without saying that the execution of the text search program may be instructed from the floppy (R) disk 112 and the execution result may be written to the floppy (R) disk 112.

The document database is stored on the floppy (R) disk 112, the CD-ROM 113, or the magnetic disk 108. RAM for fast access
It may be transferred to 107. When transferring the data to the RAM 107, the data may be converted into a format that can be easily processed.
Also, a text search program, a document database, or
Needless to say, an execution instruction may be input to the computer system via the network 114, or an execution result may be output from the computer system via the network 114.

Further, the present invention is not limited to those shown in the drawings, and various recording media, input means, and output means can be used to input and output to the computer system, and can be modified into various embodiments. Needless to say. These recording media,
The input means and the output means may, of course, be those directly accessed by the computer system or those accessed via a communication network.

FIGS. 3 to 7 show the processing flow of a document search program based on character string similarity calculated by selecting n-grams to be calculated.

FIG. 3 shows a processing flow for searching a document database based on a search sentence, selecting and outputting a document having a high degree of similarity.

First, in step S11 (hereinafter abbreviated as S11), in order to efficiently calculate the number of occurrences of a certain character string, all documents included in the document database are integrated and a suffix file (Suffix File) is created. create.
For information on how to create and use suffix files, see M. Yamamot
o and KW Church, Using Suffix Arrays to Compute
Term Frequency and Document Frequency for All Sub
strings in a Corpus, In proceeding of 6 ^th Workshop
on Very Large Corpora, Ed.Eugene Charniak, Motrea
1, pp 28-37, 1998.

The use of a suffix file makes it possible to quickly determine the number of times a character string appears in a document database. The suffix file is implemented by rearranging the character strings of parts that can occur in all documents in the order of character codes and adding serial numbers (suffixes). The number of times the string appears in the document database is
It is obtained by calculating the number of character strings that match the character string in the suffix file.

Specifically, first, the minimum value min and the maximum value max of the suffix of the matching character string are respectively obtained by the binary search method. If there is no matching character string, the number of occurrences in the document database is zero. Once min and max are determined, the number of times tf that a character string appears can be determined by tf = max-min + 1.

Documents in the document database are distinguished from each other by a document number, and this document number is assigned to a partial character string registered in a suffix file.
Thus, a document including a certain partial character string can be efficiently searched. Df can be calculated by counting the number of duplicate document numbers and subtracting the number from tf.

Next, in step S12, the search text is read into the character string X.

In S13, the partial character string cut out from the character string X is converted into an appearance frequency tf in the document database.
, And recorded in the partial character string management table as a pair with tf. The processing performed in S13 will be described later with reference to FIG.

In S14, the matching information is collected for each of the partial character strings recorded in the partial character string management table, and is recorded in the matching information management table. The matching information management table records a list of matching information for each document number. The processing performed in S14 will be described later with reference to FIG.

In S15, from the matching information management table,
A list of one document Y is extracted.

Next, in S16, the similarity between X and Y is calculated from the extracted list. Regarding the processing performed in S16,
This will be described later with reference to FIG.

In S17, the obtained similarity and the document number are paired and registered in the document management table.

In S18, it is determined whether or not the similarity has been calculated for all the lists recorded in the matching information management table. If the similarity has not been calculated for all the lists, a list for which the similarity has not been calculated is selected and extracted in S15, and the processing up to S17 is repeated. If the calculation has been performed for all the lists, the registered tables are rearranged in descending order of similarity in S19.

In S20, a process of outputting a document having a high degree of similarity is performed. Various modes are possible, such as outputting only one document, outputting a plurality of documents, or outputting all documents having a predetermined similarity or more.

FIG. 4 shows a process of extracting a partial character string from a character string X from which a search sentence has been read, selecting a partial character string to be used for collecting matching information based on the frequency of appearance, and recording the partial character string in a partial character string management table. Shows the flow.

First, in S31, a variable num_substring representing the number of partial character strings recorded in the partial character string management table
And a variable j representing the length of the partial character string to be cut out. MinNgramLength is a parameter that determines the minimum value of the length of the partial character string to be cut out.

Next, in S32, one character string of length j is cut out from the character string X, and the appearance frequency tf in the document database is calculated.

At S33, the t of the cut-out partial character string
It is determined whether the value of f is 0. If tf = 0, the partial character string does not exist in the document database, so that it is inappropriate to use it for collecting matching information. Therefore, the process of S34 is skipped, and the process proceeds to S35. t
If f ≠ 0, the process proceeds to S34.

At S34, the cut-out partial character string is
Record in the substring management table with f, num_substr
Add 1 to the value of ing.

In S35, it is determined whether tf has been calculated for all partial character strings of length j cut out from X. If the calculation has not yet been performed for all the partial character strings of the length j, a partial character string of the length j that has not been calculated is selected in S32, tf is calculated, and the processing up to S34 is repeated. If the calculation has been performed for all the partial character strings having the length j, 1 is added to j in S36.

In S37, it is determined whether or not the value of the length j of the partial character string to be extracted from X is greater than the parameter MaxNgramLength which determines the maximum value of the length of the partial character string to be extracted. If the value of j is less than MaxNgramLength, S3
2, the process up to S36 is repeated for the partial character string of length j. If the value of j is greater than MaxNgramLength, the calculation of tf has been completed for all partial character strings whose length is equal to or greater than MinNgramLength and equal to or less than MaxNgramLength. The character strings are rearranged in ascending order of tf.

In S39, it is determined whether or not the number num_substring of partial character strings registered in the partial character string management table is larger than a parameter SubStringLimit which determines the upper limit of the number of partial character strings used for collecting matching information. If num_substring is greater than SubStringLimit,
In S40, SubStringLimit partial character strings are extracted in ascending order of tf, and these partial character strings are newly recorded in the partial character string management table. If num_substring
If is less than or equal to SubStringLimit, skip S40 and proceed to S
Go to 41.

Step S41 is a process of returning the recorded partial character string management table.

FIG. 5 is a flowchart showing a process of collecting matching information between each partial character string recorded in the partial character string management table and each document in the document database and recording the information in the matching information management table. Represent.

First, in S51, a variable p0fit_sum is initialized to 0. The variable p0fit_sum is a variable used to speed up the calculation when calculating the similarity by adding the weights of the matched n-grams, and is an offset of the similarity for the entire document in the document database.

At S52, one partial character string is selected from the partial character string management table and read into a.

In S53, p0fit, p1fit, p2fit, p3fi
Calculate t and p4fit, and add p0fit to p0fit_sum. p0fi
t, p1fit, p2fit, p3fit, and p4fit are the documents when a did not appear, appeared once, appeared twice, appeared three times, and appeared four or more times in a document. Is the weight of a. p0fit, p1fit, p2fit, p3f
The method of calculating it and p4fit will be described later with reference to FIG.

In S54, all places where a appears in the document database are obtained, and the places are rearranged in the order of the places where they appear.

In S55, the document number of the document containing a is obtained for each occurrence of a. At this time, since a is arranged in the order of appearance, the obtained document numbers are also arranged in ascending order.

In S56, one appearance location a is selected in the order of appearance locations.

In S57, it is determined whether or not the selected appearance location is the forefront occurrence location in the document including the selected a. That is, if the document at the selected appearance location is different from the document at the immediately preceding appearance location, it is the first appearance location, and if the same, it is the second and subsequent appearance locations. If it is the first appearance location, the process proceeds to S58, where the number of appearances dtf of a in the document is calculated, and the weight of a in the document is determined. Also, sequence_num = 1 is set.
sequence_num is a sequence number indicating the order of occurrence of the selected appearance location in the document.

At S59, the sequence number sequ
ence_num, the appearance position of a in the input character string X (hereinafter, referred to as
startX), where a appears in the document (startd
oc), the length of a (hereinafter termlength), and the weight of a (hereinafter score) are recorded as a set in the coincidence information management table, and 1 is added to sequence_num. However, the value recorded in the score is not the weight of a as it is, but the weight of a.
Record the value after subtracting p0fit. When calculating the similarity by adding the weights of the matched n-grams, instead of adding the p0fit of each substring to the similarity of the document that does not include each selected substring, A value obtained by subtracting p0fit from the weight of the matched n-gram is added, and finally, the weight offset p0fi for all similarities
By adding t_sum, it is a contrivance to reduce the computational effort.

In S60, the value of sequence_num is compared with the value of tf to determine whether or not the coincidence information has been recorded for all the occurrence locations of a. If there is matching information that has not been recorded, the next appearance location of a is selected in S56, and S5 is selected.
The processing up to 9 is repeated. If the matching information has been recorded for all the appearance locations of a, the process proceeds to S61.

In S61, it is determined whether or not matching information has been collected for all partial character strings in the partial character string management table. If there is a partial character string for which matching information has not been collected, a partial character string not yet selected is read into a in S52, and the processing up to S60 is repeated.
If the matching information has been collected for all the partial character strings, the obtained matching information management table is returned in S62.

FIG. 6 shows the similarity between the input text X and the document Y.
Using the list extracted from the match information management table,
It is a processing flow obtained by adding the weights of the matched character strings.

First, at step S71, the similarity between X and Y (hereinafter, referred to as X)
sim) is initialized to zero.

At S72, one of the lists related to Y recorded in the matching information management table is selected and read into I.

In S73, sequence_num of the read I
Is determined to be 1 or not. This is the same substring
This is a process to avoid adding score to sim repeatedly. If sequence_num is not 1, skip S74 and proceed to S75. If sequence_num = 1, S74
Then, the score of I is added to sim.

In S75, it is determined whether all pieces of matching information recorded in the list of matching information regarding Y have been checked. If all pieces of matching information have been checked, in step S76, the weight offset p0fit of the entire document is added to sim.
Add _sum. If there is any matching information that has not been checked yet, in S72, matching information that has not been checked is selected and read into I, and the processing up to S74 is repeated.

S77 is a process of returning the obtained sim as the similarity between X and Y.

FIG. 7 shows p0fit, p in S53 of FIG.
The calculation processing flow of 1fit, p2fit, p3fit, and p4fit is shown.

First, in S81, p0fit, p1fit, p2fit, p
3fit and p4fit are all initialized to 0.

In S82, df of the partial character string a is calculated, and in S83, idf is calculated from df and the total number N of documents in the document database. This idf is a value based on the amount of information in the field of information theory, and a method of using this value as the weight of a partial character string is well known.

In S84, a threshold value tf_thr for determining whether or not the partial character string a is a partial character string effective for retrieval.
Calculate eshold.

In S85, it is determined from the values of tf and df whether or not the partial character string a is valid for retrieval. tf / df> t
If f_threshold, it is determined that the partial character string is valid for the search, and the process proceeds to S86. Otherwise, it is determined that the search is not valid, S86 and S87 are skipped, and S
At 88, p0fit, p1fit, p2fit, p3fit, p4fit are returned. That is, 0 is returned for the values of p0fit, p1fit, p2fit, p3fit, and p4fit.

In S86, p0fit, p1fit, p2fit, p3fi
Calculate t, p4fit.

The functions MAX and MIN in S87 are functions that return the maximum value or the minimum value of the numerical value given as an argument, respectively, and are used to calculate p0fit, p1fit, p2fit, p3fit, p4
The value of fit is limited to the range from LB to UB. LB and UB
Are parameters that limit the distribution of p0fit, p1fit, p2fit, p3fit, and p4fit. S88 is p0fit, p1fit, p2
This is the process that returns fit, p3fit, and p4fit.

As can be seen from the above description, the weight p0fi
t, p1fit, p2fit, p3fit, p4fit are tf, df, id
It is obtained as a function of f. The coefficients used for S84 and S86 are based on Christopher D. Manning and Hinrich Schut
ze, Foundations of Statistical Natural Language Pr
ocessing, The MIT Press, Cambridge, Massachusetts,
pp.529-574, 1999. Based on the theory disclosed in this document, it was determined by obtaining observations of document data. Note that these coefficients are not limited to the numerical values shown, and may be set to appropriate values according to the purpose.

P0fit, p1fit, p2 in S53 of FIG.
As a calculation processing flow of fit, p3fit, and p4fit, a flow shown in FIG. 15 can be applied instead of the flow shown in FIG. The calculation processing flow in FIG. 15 will be described below.

First, in S181, p0fit, p1fit, p2fi
t, p3fit, p4fit are all initialized to 0.

In S182, the df of the partial character string a and the number of documents in the document database where the partial character string a appears twice or more (hereinafter, referred to as df2) are calculated. S183
And id from df and the total number N of documents in the document database
Calculate f.

In S184, a threshold value df2_ for determining whether or not the partial character string a is a partial character string effective for retrieval.
Set threshold to 0.22.

In S185, it is determined from the values of df and df2 whether the partial character string a is valid for the search. df2 / d
If f> df2_threshold, it is determined that the partial character string is valid for search, and the process proceeds to S186. Otherwise, it is determined that the search is not valid, S186 and S187 are skipped, and p0fit, p1fit, p2fit, p3fit, p4
Returns fit. That is, p0fit, p1fit, p2fit, p3fit, p4f
All the values of it return 0.

In S186, p0fit, p1fit, p2fit, p3f
Calculate it and p4fit.

S187 is the same as p87 in FIG.
This limits the values of t, p1fit, p2fit, p3fit, and p4fit to a range from LB to UB. LB and UB are both p0fit and p1fi
These parameters limit the distribution of t, p2fit, p3fit, and p4fit. S188 is p0fit, p1fit, p2fit, p3fit, p4fi
This is the process that returns t.

As can be seen from the above description, in the calculation processing flow of FIG. 15, the reference for judging whether or not the partial character string a is valid for the search is tf instead of tf in the calculation processing flow of FIG. df2 is used. df2
/ Df indicates the degree of occurrence and concentration of partial character strings, that is, the degree to which a certain partial character string is concentrated and appears only in a specific document. By using this information to select partial character strings, The search accuracy has been improved.

The threshold value df2_threshold in S183 and the coefficient used in S186 are not limited to the numerical values shown, but may be set to appropriate values according to the purpose.

FIG. 8 shows a configuration diagram of the coincidence information management table. The matching information management table includes a list of matching information for each document number. In FIG. 8, document number 00
02, the match information 1 and the match information 5 are stored in the document number 0100, and the match information 2 and the match information 3 and the match information 6 are stored in the document number 01.
11, the match information 4 and the match information 7 are recorded as a list. The respective pieces of matching information include the sequence number sequence_num of the partial character string in the document, the location of the partial character string in the input character string X (startX), the location of the partial character string in the document (startdoc), and the Length (termlength), weight given to substring (scor)
e) is stored.

When the new match information 8 relating to the document number 0002 is newly obtained, as shown in FIG. 8, the pointer that points to the head of the list that has previously pointed to the match information 5 points to the match information 8 and the match information 8 , A pointer to the matching information 5 is set, and the matching information 8 is recorded at the head of the list of the document number 0002.

(Third Embodiment) Next, FIG. 9 shows an embodiment of a document search apparatus for searching for a document having the highest similarity to an input character string based on the n-gram DP similarity. This document search device includes a document database 10, a character string input unit 1,
1, partial character string selection unit 13 (internal illustration is omitted), coincidence information collection unit 14 (internal illustration is omitted), similarity calculation unit 1
7, a similarity calculation control unit 18, a recursive execution control unit 19, and a search result output unit 12.

Document database 10, character string input unit 1
1, the partial character string selection unit 13, the match information collection unit 14, and the search result output unit 12 have the same functions and configurations as the parts denoted by the same reference numerals in the first embodiment, and a description thereof will be omitted.

The similarity calculation control unit 18 extracts a list relating to a certain document Y from the coincidence information management table T2, and supplies the list to the similarity calculation unit 17 together with the character strings X and Y.

The similarity calculating section 17 calculates the similarity between X and Y based on the expression (1) or (2) from the given list of matching information. In the process of calculating this similarity,
It is necessary to similarly determine the similarity for a part of the character string. This is performed by the recursive execution control unit 19 using the similarity calculation unit 17 repeatedly. Similarity calculator 1
Reference numeral 7 includes a matching character string similarity calculating unit 61, an arbitrary character string similarity calculating unit 62, and a maximum value selecting unit 63. The matching character string similarity calculation unit 61 calculates SimDPs (α, β) in Expression (3).
The arbitrary character string similarity calculation unit 62 calculates SimDPg (α,
Calculate β). The maximum value selecting unit 63 calculates SimDP (α, β) in Expression (2) by performing the function MAX on these. When both the character strings α and β received by the similarity calculation unit 17 are empty characters, the recursive execution control unit 19 sets SimDP (α, β) = 0.0. At this time, the matching character string similarity calculating unit 61, the arbitrary character string similarity calculating unit 62, and the maximum value selecting unit 63 are not operated. Needless to say, this SimD
The value of P (α, β) = 0.0 implements equation (1).

The matching character string similarity calculation unit 61 includes a character string separation control unit 71, a character string separation unit 72, a similarity calculation unit 73,
This is performed by the addition unit 74 and the maximum value selection unit 75,
SimDPs (α, β) in equation (3) is calculated. SimDPs (α, β) is calculated only when the leading character string where α and β match is the character string recorded in the match information management table T2.

First, the character string separation control unit 71 sets the character strings α (= ξγ), β
In (= ξδ), if there is no matching character string 、, that is, if the matching character string の is an empty character string, the expression (1)
And SimDPs (α, β) = 0.0.

Next, the character string separation control unit 71 converts the character strings α (= ξγ), β
In (= ξδ), if there is a matching character string ξ, the character string separating unit 72 and the similarity calculating unit 7
3. By operating the adder 74, the Scor included in the equation (3) is calculated.
Let e (ξ) + SimDP (γ, δ) be calculated. Then, the largest value is selected by the maximum value selector 75. As a result, SimDP _s (α, β) shown in Expression (3) is obtained.

The character string separating section 72 converts the character string α into ξ and γ
Then, after separating the character string β into ξ and δ, the weight Score (ξ) of ξ is given to the adding unit 74 and γ and δ are given to the similarity calculating unit 73 with reference to the coincidence information management table T2. The similarity calculation unit 73 calculates SimDP (γ, δ) in Expression (3). The similarity calculation unit 73 is actually implemented by the recursion execution control unit 17 applying the similarity calculation unit 16 to γ and δ. The adding unit 74 performs addition of Expression (3).

The arbitrary character string similarity calculating section 62 is implemented by the similarity calculating sections 81 to 83 and the maximum value selecting section 84, and calculates SimDPg (α, β) in equation (2). If the leading character strings are different, or if the leading character strings match but are not registered in the matching information management table, the arbitrary character string similarity calculating unit 62 is executed. The similarity calculators 81, 82, 83 correspond to the respective cases regarding the presence or absence of the first character ζ, η of the received character strings α (= ζγ), β (= ηδ).
Are SimDP (α, δ), SimDP (γ, β),
Find SimDP (γ, δ). Actually, the similarity calculating sections 81 to 83 are controlled by the recursive execution control section 19 to execute the similarity calculating section 17.
Is applied to each of α and δ, γ and β, and γ and δ. The maximum value selection unit 84 implements the function MAX of Expression (5).

(Fourth Embodiment) FIGS. 10 and 11 show the processing flow for obtaining the n-gram DP similarity between the character strings X and Y by software. This processing can be used as the internal processing of S16 in FIG. 3 instead of the processing described in FIG. In the execution, the computer system shown in the second embodiment is used.

First, in S91, for each of X and Y, among the first character and the last character of the partial character string recorded in the list, the position of the earliest first character minX, minY,
Find the location of the last character at the end, maxX, maxY, and length
An array X_index of maxX-minX + 1 and an array Y of length maxY-minY + 1
Prepare _index and initialize all elements to -1. These arrays correspond to characters from X minX to maxX and characters from Y minY to maxY, respectively.

In the processing from S92 to S94,
An array X_in corresponding to each character from minX to maxX, each character from minY to maxY in Y, the character corresponding to the first character or the last character of each partial character string recorded in the list
Substitute 0 for the elements of dex and Y_index.

In the processing from S95 to S99, X_index
[i] = 0, i, 0, 1, 2,..., X_i
Assign a serial number to ndex_num-1 and assign that number to X_index [i]
Substitute for Therefore, X_index_num is X_index [i] ≠ −
It is the number of i that is 1.

In S100 to S104, similar processing is performed on Y_index. With respect to j for which Y_index [j] = 0, serial numbers 0, 1, 2,..., Y_index_num-1 are assigned in order from the front, and the number is substituted for Y_index [j]. Therefore, Y_index_num is the number of js for which Y_index [j] ≠ −1.

In S105, X and Y recorded in the list
Are first sorted in the order in which the partial character strings in X appear. Next, the same order of appearance of the partial character strings in X is rearranged in the order of appearance of the partial character strings in Y, and the number of pieces of matching information is read into m.

Next, as a preparation for efficiently obtaining the similarity by DP, the (X_index_num + 2) row (Y_index_num +
2) Create a score table scoretable for the columns, and set all elements of the table to 0
Initialize to This table shows that the vertical direction is
The character corresponding to the first character or the last character of the partial character string recorded in the list, and the horizontal direction corresponds to the character corresponding to the first character or the last character of the partial character string recorded in the list in the character string Y.

In S106, k and i are initialized to k = 1 and i = 0.

In S107, j is initialized to j = 0. The variable k indicates that attention is currently focused on the k-th matching information from the beginning, and i and j are the first character or the last character of the partial character string recorded in the list in X and Y, respectively. , A variable indicating which character is being focused on. In S108, the current score is currents
Substitute scoretable [i] [j] for core.

Here, for convenience of explanation, the k-th matching information from the head of the list is referred to as startX (k) and startdoc, respectively.
(k), termlength (k), and score (k).

In S109, it is determined whether or not the location indicated by i and j matches the location where the partial character string of the kth matching information appears from the beginning. If they match, S11
The process proceeds to 0, and if they do not match, the process proceeds to S114.

In S110, a line number and a column number corresponding to the last character of the matched partial character string are obtained from the score table, and are substituted into target_i and target_j, respectively.

At S111, scoretable [target_i] [tar
get_j], the current score and curr
Compare the scores obtained by adding the score (k) of the substring that matches entscore, and if scoretable [target_
If currentscore + score (k) is larger than i] [target_j], in S112, scoretable [target_i] [target_j]
Substitute currentscore + score (k). Otherwise,
Skip S112 and proceed to S113.

In S113, 1 is added to the value of k, and attention is paid to the next matching information in the list, and the flow returns to S109. S109
Then, if i and j also appear in the next matching information in the appearance location, S1
13 is repeated, otherwise, S114
Proceed to.

In the processing from S114 to S119, the current score currentscore is compared with the scores at the right, lower, and lower right of the score table. If the currentscore is larger, the currentscore is
Substitute core.

In S120, it is determined whether j has reached the right end of the score table. If the right end has not been reached, 1 is added to j in S121, the process returns to S108, and S11
The processing up to 9 is repeated. If it has reached the right end, S12
Proceed to 2.

In S122, it is determined whether i has reached the lower end of the score table. If it does not reach the lower end, 1 is added to i in S123, the process returns to S107, and S12
The process up to 0 is repeated. If it has reached the lower end, S124
And returns scoretable [X_index_num + 1] [Y_index_num + 1] as the similarity between X and Y.

[0143]

According to the first aspect of the present invention, a character string that is effective in calculating similarity with a document in a document database is selected from partial character strings cut out from an input character string. , The degree of similarity can be obtained. That is, the similarity can be obtained by limiting the search to a character string having a high effect on the search.

According to the second aspect of the present invention, the degree of similarity can be obtained by focusing on a partial character string that conforms to the order of each of the two character strings and is common. That is, the similarity is determined in consideration of the appearance order of the character strings.

According to the third aspect of the present invention, similarities can be obtained by selecting character strings in descending order of the effect on similarity calculation. As a result, the determined similarity becomes a more appropriate value.

According to the invention described in claim 4, not only the method of adding the weights of the matched partial character strings but also other similarity calculation methods can be combined. Thereby, the search accuracy is further improved.

According to the fifth aspect of the present invention, a character string that is effective in calculating the similarity with a document in the document database is selected from the partial character strings cut out from the input character string and searched. It is possible to realize a device that obtains the similarity by limiting the character string to a highly effective one.

According to the sixth aspect of the present invention, there is provided an apparatus which obtains a similarity which conforms to the order of two character strings and takes into account the order of appearance of the character strings by paying attention to a common partial character string. realizable.

According to the seventh aspect of the present invention, a character string that is effective in calculating the similarity with a document in the document database is selected from the partial character strings cut out from the input character string, and this is effective in the search. It is possible to provide a program that obtains the similarity only for character strings having a high character string.

According to the eighth aspect of the present invention, there is provided a program which obtains a similarity which conforms to the order of each of two character strings and takes into account the appearance order of the character strings by paying attention to a common partial character string. Can be provided.

According to the ninth aspect, the effects described above can be realized on various computers.

Next, search performance in the first and second embodiments will be described. An evaluation experiment was performed to confirm the effect of selecting a partial character string cut out from an input character string and the effect of using tf instead of df as a selection criterion for a partial character string.

FIG. 12 shows the relationship between the number of selected partial character strings and the search accuracy when tf and df are used as the selection criteria, and FIG. 13 shows the relationship between the number of selected partial character strings and the search time. . For the search accuracy, a value called 11pt average accuracy was used. The 11 pt average precision is an evaluation index obtained by allocating 1 to 11 points in 0.1 increments to the recall (0 to 1) and averaging the respective values. For details, see G Salton and
M. J MacGill, Introduction to Modern Information
Retrieval, p174-181, MacGraw-Hill Book Co., New Yo
rk, 1983.

The length of the partial character string cut out from the input character string was set to 2, that is, only bigram. Also, S87 of FIG.
Parameters LB and UB used in LB = 0 and UB = i, respectively
df. The used document data is a representative similar information search test data called NTCIR1 test collection, which includes about 300,000 documents and 53 search sentences.

12 and 13, the horizontal axis indicates the number of selected partial character strings, that is, the value of SubStringLimit, the vertical axis in FIG. 12 indicates search precision, and the vertical axis in FIG. Indicates the time required to search for a sentence.

Referring to FIG. 12, both tf and df have SubStr.
When the value of ingLimit is small, it can be confirmed that the search accuracy is improved by increasing SubStringLimit. However, when the value of ingLimit is large, it can be confirmed that the search accuracy does not change much even if the value of SubStringLimit is increased. For this reason, if SubStringLimit is set to an appropriate value, a decrease in search accuracy can be suppressed even if a partial character string is selected. In addition, when comparing tf and df, it can be confirmed that the search accuracy by the selection based on df is higher, but when the value of SubStringLimit increases to some extent, even when the selection is performed based on tf, the time of df It can be confirmed that it has almost the same search accuracy as.

Referring to FIG. 13, both tf and df have SubStr.
As the value of ingLimit decreases, the search time becomes shorter, and the effect of speeding up by selecting the partial character strings can be confirmed. Further, when comparing tf and df, it can be confirmed that the search time using tf is shorter, and the difference becomes larger as the value of SubStringLimit is smaller. Suppose Su
Assuming that the value of bStringLimit is set to SubStringLimit = 22, a value indicating high search accuracy for both tf and df,
The search time at that time is as follows: tf is 264.6 seconds, df is 36
At 7.1 seconds, tf is about 1.4 times faster. As described above, the search time can be improved by the present invention.

Next, search performance when the method according to the third and fourth embodiments is applied to the calculation of the similarity in S16 of FIG. 3 will be described. The search accuracy is compared between the case of using the addition method of FIG. 6 and the case of using the DP method of FIGS. 10 and 11 for calculating the similarity, and the result is shown in FIG.
It is shown in FIG. 11 pt average precision was used for search precision.
The length of the partial character string cut out from the input character string is 2, that is, only the bigram is used, and the parameter LB used in S87 of FIG. 7 is used.
And UB are set to LB = 0 and UB = idf, respectively. The document data used is the NTCIR1 test collection.

In FIG. 14, the horizontal axis indicates the number of selected partial character strings, that is, the value of SubStringLimit, and the vertical axis indicates search accuracy. Referring to FIG. 14, when SubStringLimit is 15 or less, there is no significant difference in the search accuracy. However, when the value exceeds 15, the search accuracy using the DP is higher than the addition method. You can check. Also,
When the search time was measured, the DP method requires about twice the search time as compared to the addition method, and the existing DP
It was confirmed that the speed was several tens of times higher than that of the method described in the above.

As described above, according to the present invention, by applying the DP to the collected coincidence information, the search accuracy can be improved as compared with the method using addition, and the similarity can be calculated faster than the method using the existing DP. Can be calculated.

[Brief description of the drawings]

FIG. 1 is a diagram showing an embodiment of a document search device according to the present invention.

FIG. 2 is a diagram showing an embodiment of a computer system for performing a sentence search according to the present invention.

FIG. 3 is a flowchart of a process for performing a sentence search according to the embodiment of the present invention.

FIG. 4 is a flowchart of a process for selecting a partial character string cut out from a character string.

FIG. 5 is a flowchart of a process of collecting and recording matching information.

FIG. 6 is a flowchart of a process of calculating a similarity by addition from matching information.

FIG. 7 is a flowchart of a process for calculating a weight of a partial character string.

FIG. 8 is a configuration diagram of a matching information management table.

FIG. 9 is a diagram showing another embodiment of the document search device according to the present invention.

FIG. 10 is a flowchart of a first half of a process for obtaining an n-gram DP similarity from matching information.

FIG. 11 is a flowchart of a latter half of a process for obtaining an n-gram DP similarity from matching information.

FIG. 12 is a diagram illustrating a relationship between the number of selected partial character strings and search accuracy when tf and df are used as selection criteria.

FIG. 13 is a diagram illustrating a relationship between the number of selected partial character strings and a search time when tf and df are used as selection criteria.

FIG. 14 is a diagram illustrating a comparison of search accuracy between a similarity calculation method using DP and a similarity calculation method by addition according to the present invention.

FIG. 15 is a flowchart of another process for calculating the weight of a partial character string.

[Explanation of symbols]

 10: Document database 10a, 10b, 10c: Document 11: Character string input unit 12: Search result output unit 13: Partial character string selection unit 14: Matching information collection unit 15, 17, 73, 81, 82, 83: Similarity Calculation units 16, 18: Similarity calculation control unit 19: Recursive execution control unit 31: Partial character string cutout control unit 32: Partial character string cutout unit 33: Character string appearance frequency calculation unit 34: Partial character string registration unit 41: Match Information collection control unit 42: Character string appearance location search unit 43: Character string weight calculation unit 44: Match information registration control unit 45: Match information registration unit 51: Character string weight addition control unit 52: Character string weight addition unit 61: Match Character string similarity calculation unit 62: Arbitrary character string similarity calculation unit 63, 75, 84: Maximum value selection unit 71: Character string separation control unit 72: Character string separation unit 74: Addition unit 101: Display A 102: Printer 103: Keyboard 104: Floppy (R) disk device 105: CD-ROM device 106: Read only memory (ROM) 107: Random access memory (RAM) 108: Magnetic disk device 109: Central processing unit (CPU) 110: Communication interface 111: Bus 112: Floppy (R) disk 113: CD-ROM 114: Communication network

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B075 ND03 NK06 NR05 PP12 PP24 PR06 QM01 QM08 QS01 UU06

Claims (9)

[Claims] In a method for calculating the similarity between two character strings, a partial character string selected based on an effect on similarity calculation among partial character strings cut out from the first character string is subjected to a second character string. A character string similarity calculating method, wherein information on matching with a character string is collected, a weight of a partial character string matched from the matching information is calculated, and a similarity is calculated based on the weight. 2. A method for calculating a similarity between two character strings, wherein a partial character string selected based on an effect on similarity calculation among partial character strings cut out from the first character string is subjected to a second character string. Information on matching with a character string is collected, and a similarity is calculated from partial character strings included in the matching information based on the weight of the partial character string in which the order of appearance in the first and second character strings matches. A character string similarity calculation method. 3. The method according to claim 1, wherein the selection of the partial character string is performed in consideration of the number of times the partial character string cut out from the first character string appears in the second character string. Character string similarity calculation method described. 4. The character string matching information includes a partial character string length, a partial character string appearance location in a first character string, a partial character string appearance location in a second character string, and a second character string appearance location.
4. A character string similarity calculation method according to claim 1, further comprising: a sequence number indicating the number of a match in the character string. 5. A character string similarity calculating apparatus for calculating a similarity between two character strings, wherein a partial character string selected based on an effect on similarity calculation among partial character strings cut out from the first character string. A selecting unit, a matching information collecting unit that collects matching information with the second character string, and calculating a weight of the matched partial character string based on the matching information, and calculating the similarity by summing the weights A character string similarity calculation device comprising: a similarity calculation unit. 6. A character string similarity calculating apparatus for calculating a similarity between two character strings, wherein a partial character string cut out from the first character string is selected based on an effect on similarity calculation. Part, a matching information collection unit that collects matching information with the second character string for the selected partial character string, and an order in which each character string appears from among the partial character strings included in the matching information. A matching partial character string selecting unit that selects a matching partial character string, and a similarity calculating unit that sums weights assigned to the matching partial character strings,
A character string similarity calculation device characterized by having: 7. A character string similarity calculation program for calculating a similarity between two character strings, comprising:
Selection means for selecting based on the number of times a partial character string cut out from the character string appears in the second character string, and matching information collection for collecting matching information of the selected partial character string with the second character string A character string similarity calculation program that functions as means and similarity calculation means for calculating weights of the matched partial character strings based on the match information and calculating the similarity by summing the weights. . 8. A character string similarity calculation program for calculating a similarity between two character strings, comprising:
Selecting means for selecting a partial character string cut out from the character string based on the effect on the similarity calculation, and for the selected partial character string, as the matching information with the second character string, the length of the partial character string, Matching information collecting means for collecting the occurrence position of the partial character string in the first character string, the occurrence position of the partial character string in the second character string, and the sequence number indicating the number of the match in the second character string And functioning as similarity calculating means for calculating a similarity based on a weight given to a partial character string whose order of appearance in each character string matches from among the partial character strings included in the matching information. A character string similarity calculation program characterized by the following. 9. A computer-readable recording medium on which the program according to claim 7 is recorded.