JP4065695B2 - Character string similarity calculation device, character string similarity calculation program, computer-readable recording medium recording the same, and character string similarity calculation method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to determination of similarity between two character strings, and is particularly suitable for use in determining similarity between an input character string and a document registered in a database in information retrieval.
[0002]
[Prior art]
Information retrieval for retrieving a desired document from a document database is often performed. In such information retrieval, a document is handled as a set of character strings in which words composed of a plurality of characters are combined. Then, the search is performed by comparing the search character string and the character strings in the search target document and selecting one or a plurality of items having high similarity. The degree of similarity between character strings can be broadly divided into two methods: a method using morphological analysis and a method for obtaining a match between partial character strings of length n (hereinafter referred to as n-grams).
[0003]
A method using morphological analysis is disclosed in, for example, Gerard Salton and Christopher Buckley, Term-Weighting Approaches in Automatic Text Retrieval, Information Proceeding and Management, 24, pp. 513-523, 1988. The basic procedure for obtaining the similarity between two character strings by this method is as follows. First, both character strings are decomposed into word strings by morphological analysis using a dictionary and grammatical knowledge. Next, both word strings are compared to find a matching word. Then, a weight is set for the matching word. This weight is then added for all matching words. The sum obtained as a result of this addition becomes the similarity by morphological analysis.
[0004]
The method using morphological analysis has an essential problem that information retrieval ends abnormally if the accuracy of morphological analysis itself is low. In order to increase the accuracy of morphological analysis, the word dictionary and grammatical rules have to be large-scale, and it is difficult to easily use information retrieval. In addition, for documents that contain buzzwords, coined words, or technical terms that are used only in limited fields, the maintenance of the word dictionary is a heavy burden.
[0005]
Next, the method by n-gram is, for example, Yasushi Ogawa and Toru Matsuda, Overlapping statistical word indexing: A new indexing method for Japanese text, In proceeding of SIGIR '97, Philadelphia PA, USA, pp.226-234, 1997. Is disclosed. The basic procedure for obtaining the similarity between character strings by this method is as follows. First, an n-character partial character string included in both character strings is obtained. Next, a weight is set for the common partial character string. This weight is then added for all matching parts. The sum obtained as a result of this addition becomes the similarity by n-grams.
[0006]
Regarding the setting of the weight of the common partial character string, a large value is given to a character string that appears intensively in a specific document and does not appear in other documents. Conversely, only small values are given to character strings that appear in many documents. This reflects that a character string appearing in many documents is not an element that characterizes the document and cannot be effectively used for searching.
[0007]
Since the n-gram method does not require morphological analysis, it can handle unknown words such as new technical terms and can be used easily.
[0008]
Among the n-gram methods, in particular, in the calculation of similarity using a character string of length 2 (hereinafter referred to as bigram), not all biggrams to be cut out are limited to bigrams that do not include hiragana Then, there is a method for obtaining matching information about character strings and calculating a similarity. This is because the bigram including hiragana is highly likely to appear in many documents in the document database, and the probability of becoming a character string characterizing each document is extremely small. It is recognized that the inclusion of bigrams containing hiragana characters in the comparison of character strings only increases the amount of calculation and the search accuracy cannot be expected to improve significantly.
[0009]
[Problems to be solved by the invention]
As described above, the n-gram method has many advantages over the morphological analysis method, and is often used in the field of information retrieval. The problem with the n-gram method is that the amount of calculation increases as the document database grows, and it takes time to obtain search results.
[0010]
By the way, among the partial character strings cut out from the character string, the partial character strings are not necessarily effective in calculating the similarity, that is, they are included in many documents in the document database, so the weight is small and the influence on the similarity is small. It is considered that many substrings are included. For this reason, the method of checking whether all n-grams cut out from the character string match is not efficient in terms of calculation time.
[0011]
In addition, the method of not treating the string containing hiragana as matching information for the extracted bigram is efficient in terms of calculation time, but even the character string that is originally valid for search is uniformly truncated. It will be. As a result, there is a problem that the search accuracy is lowered.
[0012]
The present invention solves the problems in the above-described n-gram method, and provides a method that achieves both reduction in calculation time and improvement in search accuracy.
[0013]
[Means for Solving the Problems]
According to the first aspect of the present invention, in the method of calculating the similarity between two character strings, the partial character strings selected based on the effect on the similarity calculation among the partial character strings cut out from the first character string. The matching information with the second character string is collected, the weight of the matched partial character string is calculated from the matching information, and the similarity is calculated based on the weight. In this way, when obtaining matching information between character strings, not all the n-grams to be cut out are handled, but the n-grams effective for calculating the similarity are selected by estimating the effect on the calculation of the similarity. ing.
[0014]
Since it is configured as described above, it is possible to reduce the calculation time while maintaining a search accuracy substantially equal to a search accuracy by a method that does not select.
[0015]
According to a second aspect of the present invention, in the method of calculating the similarity between two character strings, the partial character strings selected based on the effect on the similarity calculation among the partial character strings cut out from the first character string. , Collecting the matching information with the second character string, based on the weight of the partial character string that matches the order of appearance in the first and second character strings from the partial character strings included in the matching information The similarity is calculated. In this way, in calculating the similarity between character strings, a partial character string that matches the order of appearance in each character string is further selected from the partial character strings recorded in the matching information, and the weight is set as the similarity. It is used for calculation.
[0016]
Since it is configured as described above, the partial character string for calculating the weight is limited, and the calculation speed can be further increased while the search accuracy is maintained.
[0017]
The invention according to claim 3 is the invention according to claim 1 or 2, wherein the partial character string is selected in consideration of the number of times the partial character string cut out from the first character string appears in the second character string. It is characterized by being performed.
[0018]
Since it is configured as described above, it is possible to efficiently select partial character strings that are effective in calculating similarity, and it is possible to further reduce calculation time while maintaining search accuracy.
[0019]
The invention according to claim 4 is the invention according to any one of claims 1 to 3, wherein as the matching information between the character strings, the length of the partial character string, the appearance location of the partial character string in the first character string , An appearance location of the partial character string in the second character string, and a sequence number representing the number of matches in the second character string.
[0020]
Since it is configured as described above, it is possible to use not only a method of calculating the similarity by adding the weights of the matched n-grams, but also using many similarities calculation methods.
[0021]
According to a fifth aspect of the present invention, in the character string similarity calculation device for calculating the similarity between two character strings, a partial character string cut out from the first character string is selected based on the effect on the similarity calculation. A partial character string selection unit that performs matching, a matching information collection unit that collects matching information between the second character string, a weight by matching the partial character strings based on the matching information, and summing the weights A character string similarity calculating device comprising: a similarity calculating unit that calculates a degree.
[0022]
Since it is configured as described above, it is possible to realize an apparatus that shortens the calculation time while maintaining a search accuracy substantially equal to a search accuracy by a method that does not select.
[0023]
According to a sixth aspect of the present invention, in the character string similarity calculation device for calculating the similarity between two character strings, a partial character string cut out from the first character string is selected based on the effect on the similarity calculation. A partial character string selection unit, a matching information collection unit that collects matching information with a second character string for the selected partial character string, and a partial character string included in the matching information, Character string similarity, comprising: a matching partial character string selection unit that selects a partial character string that matches the appearance order; and a similarity calculation unit that sums the weights assigned to the matching partial character strings It is a calculation device.
[0024]
Since it is configured as described above, it is possible to realize a device in which partial character strings for calculating weights are limited and the calculation speed can be further increased while the search accuracy is maintained.
[0025]
The invention according to claim 7 is a character string similarity calculation program for calculating the similarity between two character strings, wherein a partial character string obtained by cutting the computer from the first character string is changed to the second character string. Sorting means for sorting based on the number of appearances; Matching information collecting means for collecting matching information with the second character string for the sorted partial character string; and weights of partial character strings matched based on the matching information A character string similarity calculation program that functions as similarity calculation means for calculating similarity by calculating the sum of the weights.
[0026]
With such a configuration, it is possible to cause the computer to function as a means for shortening the calculation time while maintaining a search accuracy substantially equal to a search accuracy by a method that does not select.
[0027]
The invention according to claim 8 is a character string similarity calculation program for calculating the similarity between two character strings, and the computer has an effect on the similarity calculation for a partial character string cut out from the first character string. The sorting means for sorting on the basis of the second character string, as the matching information with the second character string, for the sorted partial character string, the length of the partial character string, the appearance location of the partial character string in the first character string, the second A match information collecting means for collecting the appearance location of the partial character string in the character string, a sequence number representing the number of matches in the second character string, and a partial character string included in the match information, It is a character string similarity calculation program that functions as a similarity calculation unit that calculates a similarity based on a weight assigned to a partial character string that matches the order of appearance in the character string.
[0028]
With this configuration, the computer can function as a means for further speeding up the calculation while limiting the partial character string for calculating the weight and maintaining the search accuracy.
[0029]
The invention according to claim 9 is a computer-readable recording medium in which the program according to claim 7 or 8 is recorded.
[0030]
Since it is configured as described above, the recorded function can be realized at a necessary place.
[0031]
DETAILED DESCRIPTION OF THE INVENTION
The present invention relates to a method for calculating the similarity between character strings using n-grams, and an apparatus, program, and recording medium for realizing the method. Although it is assumed that the similarity between the input character string and a plurality of documents registered in the database is calculated, other applications are possible. When determining the matching part between character strings, it is not a method of obtaining n-grams common to the input character strings and the documents for all the documents in the database. A method of efficiently searching a document containing n-grams from the database by using a suffix file is used.
[0032]
The n-grams cut out from the input character string are considered to contain many n-grams that have little influence on the similarity calculation. The number of partial character strings cut out from the input character string is, for example, m-1 when the length of the input character string is m and the partial character string is limited to bigram which is two characters. Therefore, if the matching information is obtained for all the extracted bigrams without selecting the partial character strings, the calculation time increases as the input character string length m increases. Therefore, in the present invention, n-grams are selected from the extracted n-grams by estimating the effect on the calculation of the degree of similarity, and only a fixed number of n-grams are applied to the collection of matching information.
[0033]
Regarding the selection of partial character strings, it is necessary to select a partial character string having a large weight added to the degree of similarity in order to improve the search accuracy. Since the weight is determined by the number of documents in the database including the partial character string (hereinafter referred to as df), it is natural to select based on the value of df. However, in one aspect of the present invention, it is recommended to select a partial character string based on the number of n-grams appearing in the document in the database (hereinafter referred to as the frequency of appearance or tf) instead of df. Is done.
[0034]
tf can be calculated directly, and even if the value increases, the calculation time hardly changes. To calculate df, after calculating tf, the duplication in the document must be recalculated. When the value of is large, the calculation time becomes long. On the other hand, there is a large correlation between tf and df, and it can be expected that the use of tf instead of df will not affect the accuracy of the search.
[0035]
The collection of coincidence information is performed for each selected n-gram by the following method.
First, n-gram weights are calculated. Next, a document including the n-gram is obtained from the entire document database, the location of the n-gram in the document, the location of the n-gram in the input character string, the length of the n-gram, and the number of matches in the document The sequence number representing n and the weight of the n-gram are recorded as coincidence information.
[0036]
Usually, the obtained matching information is added and weighted as it is without recording and managing, and the similarity is calculated. In the present invention, by recording and managing this, the weight of the matched n-gram is obtained. In addition to the method of calculating the similarity by addition, the method can be applied to many methods of calculating the similarity while maintaining high speed.
[0037]
The similarity between the input character string and the document in the database is calculated by adding the weights assigned to the matched n-grams. However, even when a matching n-gram appears more than once in the same document, the weight is added only once. That is, just because a matching n-gram appears twice does not mean that the added weight is doubled. Instead, the weight of the n-gram stored as the matching information varies depending on each document according to the number of times the n-gram appears in the document (hereinafter referred to as dtf), and the greater the value of dtf, the greater the weight given. It is getting bigger. This weight is also defined for each n-gram even when dtf = 0, and a weight of 0 or less is added to all documents not including the n-gram. This is based on the idea that not including n-grams indicates that they are not similar, and it can be said that the added weight is obtained by quantifying the degree of dissimilarity. Accordingly, the value indicating the similarity is a real value including a negative value, and the similarity with the input character string increases as the value on the number line increases.
[0038]
The method of calculating the similarity is not limited to the above-described method of adding the weights attached to the matched n-grams, and other similarity calculation methods can be applied. One example is a method in which only n-grams that match the order of appearance in each character string are selected from the matched n-grams, and the sum of the weights assigned to these n-grams is used as the similarity.
[0039]
A method for calculating the similarity in consideration of the appearance order of character strings has been proposed. However, since they calculate weights for all matching partial character strings, the amount of calculation increases compared to the method of adding weights of matched n-grams, and the calculation time increases as the length of the character string to be handled increases. growing.
[0040]
In the present invention, a weight is given to the selected n-grams, and the weight is added to the similarity only when the n-grams match. Therefore, it is only necessary to consider the matching and appearance order for the selected partial character strings, so that the calculation can be performed faster than the conventional methods, and the calculation time can be increased even if the length of the character string to be handled increases. The effect is small. In this method, the similarity is calculated using dynamic programming (hereinafter referred to as DP) in order to efficiently find the combination having the highest similarity among the matching n-gram combinations. Hereinafter, a method of obtaining n-gram DP similarity using DP will be described.
[0041]
α, β, γ, and δ are character strings having a length of 0 or more, ξ, ζ, and η are character strings having a length of 1 or more, and “” is an empty character string. A character string (for example, α) in which a plurality of character strings (for example, ξ and γ) are connected is expressed by continuously writing the symbol of the character string as an element. (For example, α = ξγ).
[0042]
The n-gram DP similarity SimDP is obtained by recursively applying the following expression according to the matching pattern related to the character string portion of the argument. First, if both are empty strings,
SimDP ("", "") = 0 (1)
And Otherwise,
SimDP (α, β) = MAX (SimDPs (α, β), SimDPg (α, β)) (2)
And
Here, SimDPs (α, β) is a character string recorded in the list related to α and β in the matching information management table, and α = ξγ, β = ξδ,
SimDPs (α, β) = MAX (Score (ξ) + SimDP (γ, δ)) (3)
Is calculated for all ξ. When there is no such character string ξ,
SimDPs (α, β) = 0.0 (4)
And Score (ξ) is a function that returns the weight of ξ recorded in the coincidence information management table.
[0043]
Further, if ζ and η are the maximum character strings satisfying α = ζγ and β = ηδ and having no common part with the character strings recorded in the list related to α and β in the matching information management table, SimDPg (α , β) is
SimDPg (α, β) = MAX (SimDP (α, δ), SimDP (γ, β), SimDP (γ, δ)) (5)
Sought by. This formula is obtained by removing one of ζ or η or both ζ and η from two character strings (ζγ and ηδ) (α and δ, γ and β, γ and δ). This means that the highest degree of similarity is adopted.
[0044]
By applying the above formula recursively, a partial character string that matches the order of each of the two character strings can be obtained from the partial character strings recorded in the match information management table, and the degree of similarity is maximum. It becomes.
[0045]
The n-gram DP similarity shown above is weighted in consideration of matching not the character (character string of length 1) but the n-gram (character string of length n), as in equation (3). Thus, the similarity calculation method takes account of the continuity of characters. In addition, the matching between character strings is not considered for all partial character strings, but is limited to n-grams recorded in the matching information management table, and recorded in the matching information management table as shown in Equation (5). It is not necessary to determine whether or not there is a match for a portion that is not related to the partial character string that has been set, and can be removed. As a result, no matter how large the length of the character string for which similarity is to be calculated, only the portion related to the partial character string recorded in the match information management table needs to be considered. Compared with the conventional method that considers matching between character strings, the similarity can be calculated at high speed.
[0046]
(First embodiment)
First, an embodiment of a method for calculating the similarity between character strings using n-grams will be described. FIG. 1 shows an example of a document retrieval apparatus that retrieves a document having the highest similarity to an input character string based on the selected partial character string. The document search apparatus includes a document database 10, a character string input unit 11, a partial character string selection unit 13, a matching information collection unit 14, a similarity calculation unit 15, a similarity calculation control unit 16, and a search result output unit 12. It is configured.
[0047]
In the document database 10, a plurality of documents 10a, 10b, ..., 10c to be searched are registered. For the search, keywords, words, phrases, sentences, sentences, and the like are input (hereinafter referred to as search sentences representatively). The character string input unit 11 gives the search sentence as the character string X to the partial character string selection unit 13.
[0048]
The partial character string selection unit 13 cuts out the partial character string from the character string X given from the character string input unit 11, calculates the appearance frequency tf, extracts a constant number from the calculated small tf, and extracts the partial character string. Register in the management table T1. The partial character string selection unit 13 includes a partial character string cutout control unit 31, a partial character string cutout unit 32, a character string appearance frequency calculation unit 33, and a partial character string registration unit 34. The partial character string cutout control unit 31 controls which partial character string the partial character string cutout unit 32 cuts out. The partial character string cutout unit 32 cuts out the partial character string x from the character string X. The character string appearance frequency calculation unit 33 calculates the appearance frequency tf of the partial character string x in the document database 10. The partial character string registration unit 34 selects a fixed number of cut partial character strings in ascending order of the value of tf and registers them in the partial character string management table T1.
[0049]
The coincidence information collecting unit 14 calculates the weight given to the character string for each partial character string registered in the partial character string management table T1, detects the appearance location of each partial character string in the document database, and uses it as the coincidence information. The appearance location in the document, the appearance location in the input character string, the length of the character string of the partial character string, the sequence number indicating the number of matching in the document, and the weight of the character string are recorded in the match information management table T2. . The match information collection unit 14 includes a match information collection control unit 41, a character string appearance location search unit 42, a character string weight calculation unit 43, a match information registration control unit 44, and a match information registration unit 45. The coincidence information collection control unit 41 extracts the partial character strings a registered in the partial character string management table T1 one by one and gives them to the character string appearance location search unit 42 and the character string weight calculation unit 43. The character string appearance location search unit 42 sets the appearance number, the length of the partial character string a, and the sequence number representing the number of matches in the document for all the appearance locations in the document database of the given partial character string a. Ask. The character string weight calculation unit 43 calculates the weight given to the given character string a. The matching information registration control unit 44 selects the appearance location of the partial character string a one by one, the appearance location in the document, the appearance location in the input character string, the length of the partial character string a, the sequence number in the document, the partial character string A match with the weight of a is given to the matching information registration unit 45. The match information registration unit 45 registers each received set as match information in the corresponding document number list of the match information management table T2.
[0050]
The similarity calculation control unit 16 extracts a list relating to one document Y from the coincidence information management table T2 and provides the list to the similarity calculation unit 15.
[0051]
The similarity calculation unit 15 calculates the similarity between X and Y from the given list of matching information. The similarity calculation unit 15 includes a character string weight addition control unit 51 and a character string weight addition unit 52. The character string weight addition control unit 51 selects one match information from the list of match information, and if the sequence number of the match information is 1, gives the character string weight score to the character string weight adder 52. The character string weight addition unit 52 adds the given weight to the similarity Sim (X, Y) between X and Y.
[0052]
The search result output unit 12 selects and outputs a document having the highest similarity. At this time, documents having similarities of a certain value or higher, and documents from higher ranks to a certain rank may be output together.
[0053]
(Second embodiment)
An embodiment in which the text search according to the present invention is implemented by software will be described below.
[0054]
FIG. 2 shows an example of a computer system used for executing a text search. The computer system includes a display 101, a printer 102, a keyboard 103, a floppy (R) disk device 104, a CD-ROM (Compact Disk-Read Only Memory) device 105, a read-only memory (hereinafter referred to as ROM) 106, A readable / writable random access memory (RAM) 107, a magnetic disk device 108, a central processing unit (CPU) 109, a communication interface 110, and a bus 111 for connecting them. Has been. The floppy (R) disk device 104 reads and writes the floppy (R) disk 112, and the CD-ROM device 104 reads the CD-ROM 113. The computer system is connected to the communication network 114 by the communication interface 110.
[0055]
A text search program for implementing the present invention is stored in the ROM 106. Alternatively, the sentence search program may be stored in the floppy (R) disk 112, the CD-ROM 113, or the magnetic disk device 108 and transferred to the RAM 107, and then executed by the CPU 109. The CPU 109 executes a text search program using the RAM 107 as a work area. If necessary, the magnetic disk device 108 may be used as a work area. An instruction to execute the text search program is given from the keyboard 103, and the execution result is output to the display 101 or the printer 102. It goes without saying that the execution of the text search program may be instructed from the floppy (R) disk 112 and the execution result may be written to the floppy (R) disk 112.
[0056]
The document database is stored in the floppy (R) disk 112, the CD-ROM 113, or the magnetic disk 108. It may be transferred to the RAM 107 for high-speed access. When transferring to the RAM 107, it may be converted into a format that can be easily processed. In addition, a text search program, a document database, or an execution instruction may be input to the computer system via the network 114, or an execution result may be output from the computer system via the network 114. Of course.
[0057]
Further, the present invention is not limited to those shown in the figure, and it goes without saying that various modifications can be made to various embodiments such as input and output to the computer system using various recording media, input means, and output means. Yes. Of course, the recording medium, input means, and output means may be accessed via a communication network in addition to those directly accessed by the computer system.
[0058]
FIG. 3 to FIG. 7 show the processing flow of the document search program based on the character string similarity calculated by selecting n-grams to be calculated.
[0059]
FIG. 3 shows a processing flow in which a document database is searched based on a search sentence, and a document having a high similarity is selected and output.
[0060]
First, in step S11 (hereinafter abbreviated as S11), in order to prepare for efficiently calculating the number of appearances of a certain character string, all the documents included in the document database are integrated to create a suffix file. See S. Yamamoto and KW Church, Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in a Corpus, In proceeding of 6 ^th Workshop on Very Large Corpora, Ed. Eugene Charniak, Motreal, pp 28-37, 1998.
[0061]
By using a suffix file, the number of times a character string appears in the document database can be obtained at high speed. The suffix file is implemented by rearranging character strings of parts that can occur in all documents in the order of character codes and attaching serial numbers (suffixes). The number of times a character string appears in the document database can be obtained by calculating how many character strings in the suffix file match the character string.
[0062]
Specifically, first, the minimum value min and the maximum value max of the matching character string suffix are obtained by the binary search method. If there is no matching character string, the number of appearances in the document database is zero. If min and max are obtained, the number of times tf at which the character string appears is obtained by tf = max−min + 1.
[0063]
The documents in the document database are distinguished from each other by the document number, and the document number is assigned to the partial character string registered in the suffix file. As a result, a document including a certain partial character string can be efficiently searched. Also, df can be calculated by counting the number of duplicate document numbers and subtracting that number from tf.
[0064]
In step S12, the search sentence is read into the character string X.
[0065]
In S13, the partial character string cut out from the character string X is selected based on the appearance frequency tf in the document database, and recorded in the partial character string management table as a pair with tf. The process performed in S13 will be described later with reference to FIG.
[0066]
In S14, match information is collected for each partial character string recorded in the partial character string management table and recorded in the match information management table. In the match information management table, a list of match information is recorded for each document number. The process performed in S14 will be described later with reference to FIG.
[0067]
In S15, a list of one document Y is extracted from the match information management table.
[0068]
In step S16, the similarity between X and Y is calculated from the extracted list. The process performed in S16 will be described later with reference to FIG.
[0069]
In S17, the obtained similarity and the document number are paired and registered in the document management table.
[0070]
In S18, it is determined whether the similarity is calculated for all the lists recorded in the match information management table. If the similarity is not yet calculated for all the lists, a list for which the similarity is not yet calculated is selected and extracted in S15, and the processes up to S17 are repeated. If all the lists have been calculated, the registered tables are rearranged in descending order of similarity in S19.
[0071]
In S20, a process of outputting a document having a high similarity is performed. Various modes are possible such as only one document to be output, a predetermined plural number, or all documents having a predetermined similarity or higher.
[0072]
FIG. 4 shows a flow of processing in which a partial character string is cut out from the character string X from which the search text is read, a partial character string used for collecting matching information is selected based on the appearance frequency, and recorded in the partial character string management table. .
[0073]
First, in S31, a variable num_substring representing the number of partial character strings recorded in the partial character string management table and a variable j representing the length of the partial character string to be extracted are initialized. MinNgramLength is a parameter that determines the minimum length of the partial character string to be cut out.
[0074]
In step S32, one partial character string having a length j is cut out from the character string X, and the appearance frequency tf in the document database is calculated.
[0075]
In S33, it is determined whether or not the value of tf of the extracted partial character string is 0. If tf = 0, the partial character string does not exist in the document database, so it is inappropriate to use it for collecting matching information. Therefore, the process of S34 is skipped and the process proceeds to S35. If tf ≠ 0, the process proceeds to S34.
[0076]
In S34, the extracted partial character string is recorded in the partial character string management table together with tf, and 1 is added to the value of num_substring.
[0077]
In S35, it is determined whether tf has been calculated for all partial character strings of length j cut out from X. If all the partial character strings of length j have not been calculated yet, a partial character string of length j that has not yet been calculated is selected in S32, tf is calculated, and the processing up to S34 is repeated. If all the partial character strings having the length j are calculated, 1 is added to j in S36.
[0078]
In S37, it is determined whether the value of the length j of the partial character string cut out from X is larger than the parameter MaxNgramLength that determines the maximum length of the partial character string to be cut out. If the value of j is less than or equal to MaxNgramLength, the process returns to S32, and the process up to S36 is repeated for the partial character string of length j. If the value of j is greater than MaxNgramLength, the calculation of tf has been completed for all partial character strings whose length is greater than or equal to MinNgramLength and less than or equal to MaxNgramLength. Therefore, the process proceeds to S38 and the part recorded in the partial character string management table The character strings are rearranged in ascending order of tf.
[0079]
In S39, it is determined whether or not the number num_substring of partial character strings registered in the partial character string management table is larger than the parameter SubStringLimit that determines the upper limit value of the number of partial character strings used for collecting matching information. If num_substring is larger than SubStringLimit, the process proceeds to S40, and SubStringLimit partial character strings are extracted in ascending order of tf, and these partial character strings are newly recorded in the partial character string management table. If num_substring is less than or equal to SubStringLimit, S40 is skipped and the process proceeds to S41.
[0080]
S41 is a process of returning the recorded partial character string management table.
[0081]
FIG. 5 shows a flow of processing for collecting matching information between each partial character string recorded in the partial character string management table and each document in the document database and recording the information in the matching information management table.
[0082]
First, in S51, the variable p0fit_sum is initialized to zero. The variable p0fit_sum is a variable used for speeding up the calculation effort when calculating the weight of n-grams with matching similarities, and is an offset of the similarity regarding the entire document in the document database.
[0083]
In S52, one partial character string is selected from the partial character string management table and read into a.
[0084]
In S53, p0fit, p1fit, p2fit, p3fit, and p4fit are calculated, and p0fit is added to p0fit_sum. p0fit, p1fit, p2fit, p3fit, and p4fit are documents in which a did not appear in a document, appeared once, appeared twice, appeared three times, appeared four times or more, respectively. Is the weight of a. The calculation method of p0fit, p1fit, p2fit, p3fit, and p4fit will be described later with reference to FIG.
[0085]
In S54, all the places where a appears in the document database are obtained and rearranged in the order of the places where they appear.
[0086]
In S55, the document number of the document including a is obtained for each occurrence location of a. At this time, since a is arranged in the order of appearance, the obtained document numbers are also arranged in ascending order.
[0087]
In S56, one appearance location a is selected in the order of the appearance location.
[0088]
In S57, it is determined whether or not the appearance location of the selected “a” is the foremost occurrence location in the document including it. That is, if the document at the selected appearance location is different from the document at the previous appearance location, it is the first appearance location, and if it is the same, it is the second or subsequent appearance location. If it is the first appearance place, the process proceeds to S58, where the number of occurrences dtf of a in the document is calculated, and the weight of a in the document is determined. Also, sequence_num = 1. The sequence_num is a sequence number indicating the number a of the occurrence location in the document.
[0089]
In S59, the sequence number sequence_num in the document, the occurrence location of a in the input character string X (hereinafter referred to as startX), the occurrence location of a in the document (hereinafter referred to as startdoc), the length of a (hereinafter referred to as termlength), the a A weight (hereinafter, score) is paired and recorded in the match information management table, and 1 is added to sequence_num. However, the value recorded in score is not the weight of a as it is, but the value obtained by subtracting p0fit from the weight of a. This is because, when the similarity is calculated by adding the weights of the matched n-grams, instead of adding p0fit of each substring to the similarity of the document that does not include, for each selected substring, This is a contrivance to reduce the calculation effort by adding a value obtained by subtracting p0fit from the weights of the matched n-grams and finally adding a weight offset p0fit_sum to all similarities.
[0090]
In S60, the values of sequence_num and tf are compared, and it is determined whether or not the coincidence information has been recorded for all appearance locations of a. If there is coincidence information that is not recorded, the next a appearance location is selected in S56, and the processing up to S59 is repeated. If coincidence information is recorded for all the appearance locations of a, the process proceeds to S61.
[0091]
In S61, it is determined whether or not the matching information has been collected for all the partial character strings in the partial character string management table. If there is a partial character string for which no matching information has been collected, in S52, the partial character string not yet selected is read into a, and the processing up to S60 is repeated. If the collection of matching information has been completed for all the partial character strings, the obtained matching information management table is returned in S62.
[0092]
FIG. 6 is a processing flow for obtaining the similarity between the input sentence X and the document Y by adding the weights of the matched character strings using the list extracted from the matching information management table.
[0093]
First, in S71, the similarity between X and Y (hereinafter, sim) is initialized to zero.
[0094]
In S72, one of the lists related to Y recorded in the coincidence information management table is selected and read into I.
[0095]
In S73, it is determined whether the sequence_num of the read I is 1. This is a process for not adding the score of the same partial character string redundantly to sim. If sequence_num is not 1, S74 is skipped and the process proceeds to S75. If sequence_num = 1, the score of I is added to sim in S74.
[0096]
In S75, it is determined whether or not all the matching information recorded in the matching information list regarding Y has been checked. If all the matching information is checked, the offset p0fit_sum of the weight of the entire document is added to sim in S76. If there is matching information that has not been checked yet, the matching information that has not been checked yet is selected and read into I in S72, and the processing up to S74 is repeated.
[0097]
S77 is processing for returning the obtained sim as the similarity between X and Y.
[0098]
FIG. 7 shows a calculation processing flow of p0fit, p1fit, p2fit, p3fit, and p4fit in S53 of FIG.
[0099]
First, in S81, p0fit, p1fit, p2fit, p3fit, and p4fit are all initialized to zero.
[0100]
In S82, df of the partial character string a is calculated, and in S83, idf is calculated from df and the total number N of documents in the document database. This idf is a value based on the amount of information in the field of information theory, and a method using this value as the weight of the partial character string is well known.
[0101]
In S84, a threshold value tf_threshold for determining whether or not the partial character string a is a partial character string effective for search is calculated.
[0102]
In S85, it is determined whether or not the partial character string a is valid for the search from the values of tf and df. If tf / df> tf_threshold, it is determined that the partial character string is valid for search, and the process proceeds to S86. Otherwise, it is determined that the search is not effective, S86 and S87 are skipped, and p0fit, p1fit, p2fit, p3fit and p4fit are returned in S88. That is, 0 is returned for all values of p0fit, p1fit, p2fit, p3fit, and p4fit.
[0103]
In S86, p0fit, p1fit, p2fit, p3fit, and p4fit are calculated.
[0104]
The functions MAX and MIN in S87 are functions that return the maximum value or the minimum value of the numerical value given to the argument, respectively. With this function, the values of p0fit, p1fit, p2fit, p3fit, and p4fit are within the range of LB to UB. Restricted. LB and UB are parameters that limit the distribution of p0fit, p1fit, p2fit, p3fit, and p4fit. S88 is a process of returning p0fit, p1fit, p2fit, p3fit, and p4fit.
[0105]
As can be seen from the above description, the weights p0fit, p1fit, p2fit, p3fit, and p4fit are obtained as functions of tf, df, and idf. The coefficients used in S84 and S86 are based on the theory disclosed in Christopher D. Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing, The MIT Press, Cambridge, Massachusetts, pp. 529-574, 1999. It was determined by obtaining the observation value of document data. These coefficients are not limited to the numerical values shown, but are allowed to be appropriate values according to the purpose.
[0106]
As the calculation processing flow of p0fit, p1fit, p2fit, p3fit, and p4fit in S53 of FIG. 5, the flow shown in FIG. 15 can be applied instead of the flow shown in FIG. The calculation processing flow in FIG. 15 will be described below.
[0107]
First, in S181, p0fit, p1fit, p2fit, p3fit, and p4fit are all initialized to zero.
[0108]
In S182, the df of the partial character string a and the number of documents in the document database in which the partial character string a appears twice or more (hereinafter referred to as df2) are calculated. In step S183, idf is calculated from df and the total number N of documents in the document database.
[0109]
In S184, a threshold df2_threshold for determining whether or not the partial character string a is a partial character string effective for search is set to 0.22.
[0110]
In S185, it is determined whether or not the partial character string a is valid for the search from the values of df and df2. If df2 / df> df2_threshold, it is determined that the partial character string is valid for the search, and the process proceeds to S186. Otherwise, it is determined that the search is not effective, S186 and S187 are skipped, and p0fit, p1fit, p2fit, p3fit and p4fit are returned in S188. That is, 0 is returned for all values of p0fit, p1fit, p2fit, p3fit, and p4fit.
[0111]
In S186, p0fit, p1fit, p2fit, p3fit, and p4fit are calculated.
[0112]
S187 restricts the values of p0fit, p1fit, p2fit, p3fit, and p4fit to the range of LB to UB in the same manner as S87 in FIG. LB and UB are parameters that limit the distribution of p0fit, p1fit, p2fit, p3fit, and p4fit. S188 is processing for returning p0fit, p1fit, p2fit, p3fit, and p4fit.
[0113]
As can be seen from the above description, in the calculation processing flow of FIG. 15, df2 is used instead of tf in the calculation processing flow of FIG. 7 described above as a criterion for determining whether the partial character string a is valid for the search. ing. df2 / df represents the degree of appearance of a partial character string, that is, the degree to which a certain partial character string appears only in a specific document. By using this information, the partial character string is selected. To improve search accuracy.
[0114]
The threshold value df2_threshold in S183 and the coefficient used in S186 are not limited to the numerical values shown, but can be set to appropriate values according to the purpose.
[0115]
FIG. 8 shows a configuration diagram of the matching information management table. The match information management table is configured by a list of match information for each document number. In FIG. 8, match information 1 and match information 5 are recorded in document number 0002, match information 2 is recorded in document number 0100, match information 3 and match information 6 are recorded, and match information 4 and match information 7 are recorded in document number 0111 as a list. ing. Each match information includes the sequence number sequence_num in the document of the partial character string, the occurrence location of the partial character string in the input character string X (startX), the occurrence location of the partial character string in the document (startdoc), and the partial character string The length (termlength) and the weight (score) attached to the partial character string are stored.
[0116]
When the matching information 8 related to the document number 0002 is newly obtained, as shown in FIG. 8, the pointer pointing to the head of the list that has been pointing to the matching information 5 so far points to the matching information 8. A pointer to 5 is set, and the match information 8 is recorded at the top of the list of the document number 0002.
[0117]
(Third embodiment)
Next, FIG. 9 shows an embodiment of a document search apparatus that searches for a document having the highest similarity with the input character string based on the n-gram DP similarity. The document search apparatus includes a document database 10, a character string input unit 11, a partial character string selection unit 13 (internal illustration is omitted), a matching information collection unit 14 (internal illustration is omitted), a similarity calculation unit 17, and a similarity The degree calculation control unit 18, the recursive execution control unit 19, and the search result output unit 12 are configured.
[0118]
The document database 10, the character string input unit 11, the partial character string selection unit 13, the matching information collection unit 14, and the search result output unit 12 have the same functions and configurations as the parts denoted by the same reference numerals in the first embodiment. Omitted.
[0119]
The similarity calculation control unit 18 takes out a list related to a certain document Y from the coincidence information management table T2, and provides the similarity calculation unit 17 together with the character strings X and Y.
[0120]
The similarity calculation unit 17 calculates the similarity between X and Y based on the formula (1) or the formula (2) from the given list of matching information. In the middle of calculating the similarity, it is necessary to similarly determine the similarity for a part of character strings. This is implemented by repeatedly using the similarity calculation unit 17 by the recursive execution control unit 19. The similarity calculation unit 17 includes a matched character string similarity calculation unit 61, an arbitrary character string similarity calculation unit 62, and a maximum value selection unit 63. The matched character string similarity calculation unit 61 calculates SimDPs (α, β) in Expression (3). The arbitrary character string similarity calculation unit 62 calculates SimDPg (α, β) in equation (5). The maximum value selection unit 63 calculates SimDP (α, β) in Expression (2) by performing the function MAX on these. When both the character strings α and β received by the similarity calculation unit 17 are empty characters, the recursive execution control unit 19 sets SimDP (α, β) = 0.0. At this time, the matched character string similarity calculation unit 61, the arbitrary character string similarity calculation unit 62, and the maximum value selection unit 63 are not operated. Needless to say, this value of SimDP (α, β) = 0.0 implements the equation (1).
[0121]
The matched character string similarity calculation unit 61 is implemented by a character string separation control unit 71, a character string separation unit 72, a similarity calculation unit 73, an addition unit 74, and a maximum value selection unit 75, and SimDPs of Expression (3) (α, β) is calculated. SimDPs (α, β) is calculated only when the leading character string in which α and β match is the character string recorded in the match information management table T2.
[0122]
First, in the character strings α (= ξγ) and β (= ξδ) received by the matching character string similarity calculation unit 61, the character string separation control unit 71 does not have a matching character string ξ, that is, a matching character. When the column ξ is an empty character string, SimDPs (α, β) = 0.0 is set according to Equation (1).
[0123]
Next, in the character strings α (= ξγ) and β (= ξδ) received by the matching character string similarity calculation unit 61, the character string separation control unit 71 determines that all the ξ , The character string separation unit 72, the similarity calculation unit 73, and the addition unit 74 are operated to calculate Score (ξ) + SimDP (γ, δ) included in Equation (3). Then, the maximum value is selected by the maximum value selection unit 75. As a result, SimDP shown in Equation (3) _s (α, β) is obtained.
[0124]
The character string separating unit 72 separates the character string α into ξ and γ and the character string β into ξ and δ, and then gives the weight Score (ξ) of ξ to the adding unit 74 with reference to the matching information management table T2. γ and δ are given to the similarity calculation unit 73. The similarity calculation unit 73 calculates SimDP (γ, δ) in Expression (3). The similarity calculation unit 73 is actually implemented by applying the similarity calculation unit 16 to γ and δ by the recursive execution control unit 17. The adding unit 74 performs addition of Expression (3).
[0125]
The arbitrary character string similarity calculation unit 62 is implemented by the similarity calculation units 81 to 83 and the maximum value selection unit 84, and calculates SimDPg (α, β) in Expression (2). The arbitrary character string similarity calculation unit 62 is executed when the leading character strings are different or when the leading character strings match but are not registered in the matching information management table. Corresponding to each case regarding the presence or absence of the first character ζ, η of the received character strings α (= ζγ), β (= ηδ), the similarity calculation units 81, 82, 83 respectively Find SimDP (α, δ), SimDP (γ, β), SimDP (γ, δ). The similarity calculation units 81 to 83 are actually implemented by applying the similarity calculation unit 17 to each of α and δ, γ and β, and γ and δ by the recursive execution control unit 19. . The maximum value selection unit 84 performs the function MAX of Expression (5).
[0126]
(Fourth embodiment)
A processing flow for obtaining the n-gram DP similarity between the character string X and the character string Y by software is shown in FIGS. This process can be used instead of the process described in FIG. 6 as the internal process of S16 of FIG. In execution, the computer system shown in the second embodiment is used.
[0127]
First, in S91, in each of X and Y, among the first character and last character of the partial character string recorded in the list, the position of the first character in the forefront, minX, minY, and the position of the last character in the last maxX and maxY are obtained, an array X_index of length maxX-minX + 1 and an array Y_index of length maxY-minY + 1 are prepared, and all elements are initialized to -1. These arrays correspond to each character from X minX to maxX and each character from Y minY to maxY.
[0128]
In the processing from S92 to S94, the array X_index corresponding to the character corresponding to the first character or the last character of each partial character string recorded in the list, with each character from minX to maxX in X and each character from minY to maxY in Y. , 0 is assigned to the Y_index element.
[0129]
In the processing from S95 to S99, serial numbers 0, 1, 2,..., X_index_num-1 are assigned in order from the front to i where X_index [i] = 0, and the numbers are substituted into X_index [i]. Therefore, X_index_num is the number of i for which X_index [i] ≠ −1.
[0130]
In S100 to S104, the same processing is performed on Y_index. A serial number such as 0, 1, 2,..., Y_index_num-1 is assigned to j where Y_index [j] = 0 in order from the front, and the number is substituted into Y_index [j]. Therefore, Y_index_num is the number of j where Y_index [j] ≠ −1.
[0131]
In S105, the matching information of X and Y recorded in the list is first rearranged in the order in which the partial character strings in X appear. Next, in the order in which the partial character strings appear in X, they are rearranged in the order in which the partial character strings appear in Y, and the number of matching information is read into m.
[0132]
Next, as a preparation for efficiently obtaining the similarity by DP, a score table scoretable of (X_index_num + 2) rows (Y_index_num + 2) columns is created, and all elements of the table are initialized to zero. In this table, the vertical direction is the character corresponding to the first character or the last character of the partial character string recorded in the list in the character string X, and the horizontal direction is the partial character recorded in the list in the character string Y. Corresponds to the first or last character in the column.
[0133]
In S106, k and i are initialized to k = 1 and i = 0.
[0134]
In S107, j is initialized to j = 0. The variable k indicates that attention is currently focused on the k-th matching information from the beginning, and i and j are the first character or the last character of the partial character string recorded in the list in X and Y, respectively. , A variable that indicates which character is focused on. In S108, scoretable [i] [j] is assigned to currentscore as the current score.
[0135]
Here, for convenience of explanation, the kth matching information from the top of the list is represented as startX (k), startdoc (k), termlength (k), and score (k), respectively.
[0136]
In S109, it is determined whether or not the location indicated by i and j matches the location where the partial character string of the kth matching information from the front appears. If they match, the process proceeds to S110, and if they do not match, the process proceeds to S114.
[0137]
In S110, a line number and a column number corresponding to the final character of the matched partial character string are obtained in the score table, and are substituted into target_i and target_j, respectively.
[0138]
In S111, in scoretable [target_i] [target_j], the score obtained at the current stage is compared with the score obtained by adding the weight score (k) of the substring matching currentscore. If currentscore + score (k) is larger than target_i] [target_j], currentscore + score (k) is substituted into scoretable [target_i] [target_j] in S112. Otherwise, skip S112 and go to S113.
[0139]
In S113, 1 is added to the value of k, and attention is paid to the next matching information in the list, and the process returns to S109. In S109, if i and j are also used as the next match information, the process up to S113 is repeated, and if not, the process proceeds to S114.
[0140]
In the processing from S114 to S119, the current score currentscore is compared with the right, lower, and lower right scores of the score table, and if the currentscore is larger, the currentscore is substituted.
[0141]
In S120, it is determined whether j has reached the right end of the score table. If it has not come to the right end, 1 is added to j in S121, the process returns to S108, and the processes up to S119 are repeated. If it has come to the right end, it will progress to S122.
[0142]
In S122, it is determined whether i has reached the lower end of the score table. If it has not come to the lower end, 1 is added to i in S123, the process returns to S107, and the processes up to S120 are repeated. If it has reached the lower end, the process proceeds to S124, and scoretable [X_index_num + 1] [Y_index_num + 1] is returned as the similarity between X and Y.
[0143]
【The invention's effect】
According to the first aspect of the present invention, the character string that is effective in calculating the similarity to the document in the document database is selected from the partial character strings cut out from the input character string to obtain the similarity. be able to. That is, the similarity can be obtained by limiting to character strings that are highly effective for searching.
[0144]
According to the second aspect of the present invention, it is possible to obtain the similarity by paying attention to the partial character strings that match the order in each of the two character strings. That is, the similarity considering the appearance order of the character strings is obtained.
[0145]
According to the third aspect of the present invention, it is possible to obtain the similarity by sorting the character strings in descending order of the effect on the similarity calculation. As a result, the determined similarity becomes a more appropriate value.
[0146]
According to the fourth aspect of the present invention, not only a method of adding weights of matched partial character strings but also other similarity calculation methods can be combined. This further improves the search accuracy.
[0147]
According to the fifth aspect of the present invention, a character string that is effective in calculating similarity to a document in the document database is selected from the partial character strings cut out from the input character string, and the search is effective. It is possible to realize an apparatus that obtains similarity by limiting to a high character string.
[0148]
According to the sixth aspect of the present invention, it is possible to realize an apparatus that finds the degree of similarity that matches the order in each of the two character strings and that considers the appearance order of the character strings by paying attention to the common partial character strings.
[0149]
According to the seventh aspect of the present invention, a character string that has an effect on the similarity calculation with the document in the document database is selected from the partial character strings cut out from the input character string, and the character that is highly effective for the search. It is possible to provide a program for obtaining the similarity only for the columns.
[0150]
According to the eighth aspect of the invention, it is possible to provide a program that finds the degree of similarity that matches the order in each of the two character strings and that considers the appearance order of the character strings by paying attention to the common partial character strings.
[0151]
According to the ninth aspect of the present invention, the effects described above can be realized on various computers.
[0152]
Next, the search performance regarding the first and second embodiments will be described. An evaluation experiment was conducted to confirm the effect of selecting a partial character string cut out from the input character string and the effect of using tf instead of df as a selection criterion for the partial character string.
[0153]
FIG. 12 shows the relationship between the number of selected partial character strings and search accuracy when tf and df are used as selection criteria, and FIG. 13 shows the relationship between the number of selected partial character strings and search time. A value called 11pt average accuracy was used for the search accuracy. The 11 pt average accuracy is an evaluation index obtained by assigning 1 to 11 points in 0.1 increments to the recall (0 to 1) and averaging each value. For details, see G Salton and M. J MacGill, Introduction to Modern Information Retrieval, p174-181, MacGraw-Hill Book Co., New York, 1983.
[0154]
The length of the partial character string cut out from the input character string is 2, that is, only bigram. Also, the parameters LB and UB used in S87 of FIG. 7 are set to LB = 0 and UB = idf, respectively. The used document data is representative similar information search test data called NTCIR1 test collection, which includes about 300,000 documents and 53 search sentences.
[0155]
12 and 13, the horizontal axis indicates the number of selected partial character strings, that is, the value of SubStringLimit, the vertical axis in FIG. 12 indicates search accuracy, and the vertical axis in FIG. 13 searches for 53 search sentences. It shows the time taken to do.
[0156]
Referring to FIG. 12, it can be confirmed that the search accuracy is improved by increasing the SubStringLimit when the value of SubStringLimit is small for both tf and df, but the search accuracy is improved even if the value of SubStringLimit is increased to a certain extent. It can be confirmed that there is not much change. For this reason, if SubStringLimit is set to an appropriate value, it is possible to suppress a decrease in search accuracy even if partial character strings are selected. In addition, when tf and df are compared, it can be confirmed that the search accuracy based on the selection based on df is higher, but if the value of SubStringLimit increases to some extent, even when the selection is performed based on tf, It can be confirmed that it has almost the same search accuracy.
[0157]
Referring to FIG. 13, both tf and df, the search time is shortened as the value of SubStringLimit decreases, and the effect of speeding up by selecting partial character strings can be confirmed. Further, when tf and df are compared, it can be confirmed that the search time using tf is shorter, and that the difference becomes larger as the value of SubStringLimit is smaller. If the value of SubStringLimit is set to a value indicating high search accuracy for both tf and df, SubStringLimit = 22, the search time at that time is 264.6 seconds for tf and 367.1 seconds for df. The tf is about 1.4 times faster. Thus, the search time can be improved by the present invention.
[0158]
Next, search performance when the methods according to the third and fourth embodiments are applied to the calculation of the similarity in S16 of FIG. 3 will be described. The search accuracy is compared between the case of using the addition method of FIG. 6 and the method of DP of FIGS. 10 and 11 as the similarity calculation method, and the results are shown in FIG. The search accuracy was 11 pt average accuracy. The length of the partial character string cut out from the input character string is 2, that is, only bigram, and the parameters LB and UB used in S87 of FIG. 7 are LB = 0 and UB = idf, respectively. The document data used is the NTCIR1 test collection.
[0159]
The horizontal axis of FIG. 14 indicates the number of selected partial character strings, that is, the value of SubStringLimit, and the vertical axis indicates the search accuracy. As shown in FIG. 14, when SubStringLimit is 15 or less, there is no big difference in search accuracy. However, when it exceeds 15, the method using DP is more accurate than the method using addition. I can confirm. Further, when the search time was measured, it was confirmed that the DP method requires approximately twice as much search time as the addition method, and is several tens of times faster than the existing DP method.
[0160]
Thus, according to the present invention, by applying the DP to the collected match information, the search accuracy can be improved as compared with the addition method, and the similarity can be calculated at a higher speed than the existing DP method. it can.
[Brief description of the drawings]
FIG. 1 is a diagram showing an embodiment of a document search apparatus according to the present invention.
FIG. 2 is a diagram showing an embodiment of a computer system for performing a text search according to the present invention.
FIG. 3 is a flowchart of a text search process according to an embodiment of the present invention.
FIG. 4 is a flowchart of processing for selecting a partial character string cut out from a character string.
FIG. 5 is a flowchart of a process for collecting and recording coincidence information.
FIG. 6 is a flowchart of processing for obtaining similarity by addition from coincidence information.
FIG. 7 is a flowchart of processing for obtaining a weight of a partial character string.
FIG. 8 is a configuration diagram of a matching information management table.
FIG. 9 is a diagram showing another embodiment of the document search apparatus according to the present invention.
FIG. 10 is a flowchart of the first half of processing for obtaining n-gram DP similarity from coincidence information.
FIG. 11 is a flowchart of the latter half of the process of obtaining n-gram DP similarity from coincidence information.
FIG. 12 is a diagram showing the relationship between the number of selected partial character strings and search accuracy when tf and df are used as selection criteria.
FIG. 13 is a diagram showing the relationship between the number of selected partial character strings and search time when tf and df are used as selection criteria.
FIG. 14 is a diagram showing a comparison of search accuracy between a similarity calculation method using DP and a similarity calculation method by addition in the present invention.
FIG. 15 is a flowchart of another process for obtaining the weight of a partial character string.
[Explanation of symbols]
10: Document database
10a, 10b, 10c: Document
11: Character string input part
12: Search result output part
13: Partial character string selector
14: Matching information collection unit
15, 17, 73, 81, 82, 83: similarity calculation unit
16, 18: Similarity calculation control unit
19: Recursive execution control unit
31: Partial character string segmentation control unit
32: Partial character string cutout part
33: Character string appearance frequency calculation unit
34: Partial character string registration part
41: Matching information collection control unit
42: Character string appearance location search part
43: Character string weight calculation unit
44: Matching information registration control unit
45: Match information registration section
51: Character string weight addition control unit
52: Character string weight addition unit
61: Matched character string similarity calculation unit
62: Arbitrary character string similarity calculation unit
63, 75, 84: Maximum value selection section
71: Character string separation control unit
72: Character string separator
74: Adder
101: Display
102: Printer
103: Keyboard
104: floppy (R) disk device
105: CD-ROM device
106: Read-only memory (ROM)
107: Random access memory (RAM)
108: Magnetic disk device
109: Central processing unit (CPU)
110: Communication interface
111: Bus
112: Floppy (R) disk
113: CD-ROM
114: Communication network

Claims (9)

A character string similarity calculation device for calculating a similarity between an input character string and a character string in a document included in a document database,
The document database includes at least one document;
Cutting means for sequentially cutting out a partial character string of a predetermined length from the input character string;
Appearance frequency calculating means for calculating the appearance frequency in the document database for each of the partial character strings cut out sequentially,
Wherein based on the appearance frequency for each of the partial character string, sequentially cut out the partial character string selection means for said frequency is selected a predetermined number of partial strings of the relatively small as sorting character strings of When,
For each partial character string included in the selected character string group, the appearance location in the document included in the document database is searched, and the appearance frequency of each partial character string included in the selected character string group in each document is determined. Search means to output ;
For each partial character string included in the selected character string group, one function corresponding to the appearance frequency of each partial character string in each document among a plurality of functions defined in advance in association with the value of the appearance frequency in the document A weight calculating means for calculating a weight in the document using
Similarity calculation means for calculating the similarity by adding the weights associated with each of the partial character strings included in the selected character string group appearing in the document for the document included in the document database; A character string similarity calculation device. The weight calculation means has a ratio of the number of documents including two or more partial character strings to the number of documents including at least one partial character string for each of the partial character strings included in the selected character string group. , it is determined whether more than a predetermined threshold value, the weight of the partial character string is set to zero if the ratio is less than a predetermined threshold value, the string similarity calculating apparatus according to claim 1 . The weight calculation means, for each of the partial character strings included in the selected character string group, the ratio of the appearance frequency of the partial character string in the document database with respect to the number of documents including at least one partial character string, determining whether more than a predetermined threshold value, the weight of the partial character string is set to zero if the ratio is less than a predetermined threshold value, the string similarity calculating apparatus according to claim 1. 4. The character string similarity calculation device according to claim 3 , wherein the predetermined threshold value is calculated based on the number of documents including at least one corresponding partial character string and the total number of documents included in the document database. . Each of the plurality of predefined functions is a function including a value idf with an information amount as a background,
The value idf with the amount of information in the background is represented by the following equation:
idf = log ₂ (N / df)
However, N is the number of documents included in the document database, df is the number of documents that contain the substring corresponding among the documents included in the document database, according to any one of claims 1-4 Character string similarity calculation device. Wherein among the documents included in the document database, the result the degree of similarity calculated selects and outputs a relatively high document further comprising an output unit, a character string according to any one of claims 1 to 5 Similarity calculation device. A character string similarity calculation program that causes a computer to function as a character string similarity calculation device that calculates a similarity between an input character string and a character string in a document included in a document database,
The document database includes at least one document;
The program is a computer,
Cutting means for sequentially cutting out a partial character string of a predetermined length from the input character string;
Appearance frequency calculating means for calculating the appearance frequency in the document database for each of the partial character strings cut out sequentially,
Wherein based on the appearance frequency for each of the partial character string, sequentially cut out the partial character string selection means for said frequency is selected a predetermined number of partial strings of the relatively small as sorting character strings of When,
For each partial character string included in the selected character string group, the appearance location in the document included in the document database is searched, and the appearance frequency of each partial character string included in the selected character string group in each document is determined. Search means to output ;
For each partial character string included in the selected character string group, one function corresponding to the appearance frequency of each partial character string in each document among a plurality of functions defined in advance in association with the value of the appearance frequency in the document A weight calculating means for calculating a weight in the document using
As a similarity calculation means for calculating the similarity by adding the weights associated with each of the partial character strings included in the selected character string group appearing in the document for the document included in the document database. A character string similarity calculation program that functions. A computer-readable recording medium on which the character string similarity calculation program according to claim 7 is recorded. A character string similarity calculation method for calculating a similarity between an input character string and a character string in a document included in a document database using a computer including a processing device,
The document database includes at least one document;
The character string similarity calculation method is:
The processor sequentially cuts out a partial character string of a predetermined length from the input character string;
The processing device calculates the appearance frequency in the document database for each of the partial character strings sequentially cut out;
The processing device selects a predetermined number of partial character strings out of the partial character strings that are sequentially cut out based on the appearance frequency for each of the partial character strings, the partial character strings having a relatively low appearance frequency. As a step to sort as
For each partial character string included in the selected character string group, the processing device searches for an appearance location in the document included in the document database, and each of the partial character strings included in the selected character string group. Outputting the appearance frequency in the document;
For each partial character string included in the selected character string group, the processing device sets an appearance frequency of each partial character string in each document among a plurality of functions defined in advance in association with an appearance frequency value in the document. Calculating weights in the document using one corresponding function ;
Calculating the similarity by adding the weight associated with each of the partial character strings included in the selected character string group appearing in the document for the document included in the document database. String similarity calculation method.

Images