JP2003288366A - Similar text retrieval device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine how much a plurality of texts agree with each other even when they are not perfectly agree with each other at a high speed. <P>SOLUTION: This device is characterized in that it is provided with an input interface means 101 for inputting a text, a pre-processing means 103 for performing a pre-processing to the inputted text, an N-gram forming means 104 for forming the N-gram element to the text, a similarity calculation means 105 for calculating the matching degree of N-gram element for a plurality of texts, and an output interface means 108 for outputting the calculation result of the similarity calculation means 105. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a similar text search apparatus, and more particularly to a method for quickly determining how similar texts are, even if a plurality of texts do not completely match.

[0002]

2. Description of the Related Art An exact match search for keywords is performed as a search device for texts, but this only outputs texts that exactly match a search key entered by a searcher. However, "fuzzy search" is required to judge that even incomplete matches such as "text" and "text" match. For example W
Regarding the expressions regarding Windows (registered trademark), various expressions such as Windows-NT and WIN NT4 are made by an individual depending on the version, but it is necessary to determine that these are similar. There are many things.

[0003]

The conventional search device outputs only the text that exactly matches the search key input by the searcher, and it is considered that high-precision fuzzy search takes time. The search system was not well developed.

Therefore, the object of the present invention is to achieve the above-mentioned "Win.
An object of the present invention is to provide a device for high-speed search for a keyword having a variation in notation, such as "dows-NT" or "WIN NT4".

[0005]

FIG. 1 shows the principle of the present invention. In FIG. 1, 1 is a similar text search device, 101
Is an input interface means, 103 is a preprocessing means, 10
4 is an N-gram conversion unit, 105 is a similarity calculation unit, and 108.
Is an output interface means.

The above object of the present invention is achieved by the following items (1) to (5).

(1) Input interface means 101 for inputting text, preprocessing means 103 for preprocessing input text, N-gram conversion means 104 for creating N-gram elements for text, and a plurality of texts A similar text search device comprising: a similarity calculation means 105 for calculating the degree of coincidence of N-gram elements relating to the above and an output interface means 108 for outputting the calculation result of the similarity calculation means 105.

(2) An input interface means for inputting text, a text database means for holding text to be a similarity calculation object, a preprocessing means for preprocessing text, and an N-gram element for text. The N-gram converting means to be created, the similarity calculating means for calculating the degree of coincidence of N-gram elements regarding a plurality of texts, the sorting means for outputting the operation result in the order of the highest degree of coincidence, and the sorting result of the sorting means A similar text search device comprising output interface means for outputting.

(3) The input interface means for inputting text, the text database means for holding the text to be the similarity calculation object, the preprocessing means for preprocessing the text, and the value of N for the text are A plurality of N-gram converting means for creating a plurality of different types of N-gram elements, a similarity degree calculating means for calculating the degree of similarity of the different plurality of types of N-gram elements based on the frequency of each N-gram element, and each N-gram element. A similarity adding means for adding the values calculated by the frequency of the gram element, a sorting means for outputting the outputs of the similarity adding means in descending order, and an output interface means for outputting the sorting result of the sorting means. A similar text search device characterized by being provided.

(4) An input interface means for inputting text, a text database means for holding text to be a similarity calculation object, a preprocessing means for preprocessing text, and an N-gram element for text. Regarding N-gram forming means for creating, index database means for holding N-gram elements created for the text held in the text database means, access means for this index database means, and a plurality of texts It is characterized by further comprising a similarity calculation means for calculating the degree of coincidence of the N-gram element, a sorting means for outputting the calculation result in the order of the highest degree of coincidence, and an output interface means for outputting the sorting result of the sorting means. Similar text search device.

(5) The input interface means for inputting a text, the text database means for holding the text to be the similarity calculation object, the preprocessing means for preprocessing the text, and the value of N for the text are A plurality of N-gram conversion means for creating different types of N-gram elements, and an index database means for indexing different types of N-gram elements created for the text stored in the text database means; , An access means for the index database means, a similarity calculation means for calculating the degree of similarity of the N-gram elements of the different plural types by the frequency of each N-gram element, and a degree of similarity by the frequency of each N-gram element. The similarity adding means for adding the calculated value and this similarity addition And sorting means for outputting the descending order output means, similar text search apparatus characterized by comprising an output interface means for outputting the sort result of the sorting means.

As a result, the following operational effects are exhibited.

(1) Since N-gram elements are created for each text and the matching is performed, the text matching can be searched in a form that absorbs the blurring of the expression, so that the fuzzy search can be accurately executed.

(2) Since one of the texts to be compared is held in the text database in advance, it is not necessary to input all the texts to be compared at each search, and similar texts can be searched at high speed.

(3) Since one of the texts is held in the database, a plurality of types of N-gram elements having different N values are created, and the degree of similarity is calculated according to the frequency.
For example, in the case of N = 2, that is, in the case of a 2-gram element, even if the apparent similarity is increased by matching the particle part, this can be suppressed in the case of a 3-gram element, and the similarity determination The speed and accuracy of the results can be improved.

(4) Since the N-gram element of the text stored in advance in the text database is created and stored in the index database, it is stored in this index database when the texts are compared. It is possible to quickly calculate the similarity to the input text using the N-gram element.

(5) A plurality of types of N-gram elements having different N values are created from the input text, and a plurality of types of N-gram elements having different N are also created for the text held in the text database. Since it is stored in the index database, the similarity depending on the frequency of N-gram elements can be performed at high speed, and the accuracy of the similarity can be improved.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described.

A. First Embodiment of the Present Invention A first embodiment of the present invention will be described with reference to FIG.
FIG. 1A shows a first embodiment of the present invention, and FIG. 1B is an operation explanatory diagram of the N-gram conversion device.

In FIG. 1, a similar text retrieval device 1
Are input interface means 101 and preprocessing means 10.
3, N-gram conversion means 104, similarity calculation means 105, and output interface means 108.

The input interface means 101 is used for inputting the text whose similarity is searched, and the texts "Win NT4" and "Windows-NT" are now input.
These texts are input when the similarity of is calculated. In order to perform this input, a terminal device such as a personal computer (hereinafter referred to as a personal computer) is connected.

When the text is alphabetic, the preprocessing means 103 unifies them with capital letters, removes blanks, removes hyphens, removes punctuation marks, parentheses, etc., and converts half-width characters into full-width characters. Is.

The N-gram converting means 104 converts the text into N-grams, that is, divides the text into N-character elements. For example, the operation of converting WINNT4 into 2 grams (divided into two character elements) will be described with reference to FIG.

First, take the first two characters WI of WINNT4.

Next, two characters are taken by shifting one character from the beginning. As a result, IN can be obtained.

Then, one character is further shifted to take two characters. As a result, NN can be obtained.

By repeating such processing, W
INT4 is connected to WI, IN, N as shown in FIG.
It can be divided into N, NT, and T4 as a 2-gram element (bigram element).

The similarity calculating means 105 calculates the similarity by comparing the coincidence and non-coincidence of bigram elements of two texts. For example, two texts whose similarity is to be measured are divided into bigram elements, and the number of common sets is divided by the number of unions to calculate the similarity. As shown in FIG. 1B, WINNT4 and WI
When obtaining the similarity with NDOWNSTNT (registered trademark),
The common set is 3 (WI, IN, NT), and the union is 10 (W
I, IN, NN, NT, T4, ND, DO, OW, W
S, SN), and the similarity between the two texts in this case is 0.3.

When the lengths of the two texts are extremely different, there is also a method of dividing the number of common sets by the number of bigram elements of the shorter text. In this example, the similarity of the shorter text WINNT4 is that the number of common sets is 3,
Since the number of bigram elements in the shorter text is 5, the similarity is 0.6.

The output interface means 108 outputs the similarity calculated by the similarity calculating means 105 to, for example, the terminal device which has input the text to the input interface means 101.

For the operation of FIG. 1, the text "WinNT
A case where the similarity between “4” and “Windows-NT” is obtained will be described.

(1) First, the user inputs the texts "WinNT4" and "Windows-" from a terminal device (not shown).
Enter "NT". The pre-processing unit 103 pre-processes these texts and processes them as WINNT4 and WINDOW.
It is sent to the N-gram converting means 104 as SNT.

(2) In this example, an example in which the N-gram converting means 104 operates as the 2-gram converting means will be described. N
The gram conversion means 104 first sets WINNT4 in FIG.
The resulting WI, I was fragmented into 2 gram elements as shown in
N, NN, NT, T4 are sent to the similarity calculation means 105, and then WINDOWSNT is set to 2 as shown in FIG. 1 (B).
Fragmented into gram elements and obtained WI, IN, ND, D
The O, OW, WS, SN, and NT are sent to the similarity calculation means 105. If there are multiple identical 2-gram elements with the same text, only one element is left
Delete the gram element. The same applies to N (N ≠ 2) grammarization.

(3) In the similarity calculating means 105, a common set (WI, IN, NT) and a union (WI, IN, NN, NT, T4, ND, DO, O) are created from these two-gram elements.
W, WS, SN). And the number of elements in the common set is 3
Is quotient with the number of elements of the union, and the obtained value 0.3 is output and displayed on the terminal device to which the text is input via the output interface 108. Thereby, the user recognizes that the similarity between the two texts is 0.3.

B. Second Embodiment of the Present Invention A second embodiment of the present invention will be described with reference to FIGS. FIG. 2 shows a second embodiment of the present invention, and FIG. 3 is an operation explanatory diagram thereof.

In FIG. 2, a similar text search device 2
Is to create a plurality of types of N-gram elements having different values of N from the text, add the calculated values of similarity to the frequency of each N-gram, and recognize the similarity from the addition result. . For example, two types of text, a 2-gram element and a 3-gram element, are created, the similarity calculated by the 2-gram element and the similarity calculated by the 3-gram element are added, and the similarity is calculated by the addition result. recognize.

As a result, for example, when the text is a sentence, the degree of similarity increases when two-letter particles such as "kara", "yori", and "wa" match in the 2-gram element. Since such a match can be prevented with the 3-gram element, the similarity can be calculated accurately. In the following description, the case of converting into 2 grams and 3 grams will be described.

(1) First, the user inputs the texts "WinNT4" and "Windows-N" from a terminal device (not shown).
Enter "T". These texts are input to the preprocessing unit 103 via the input interface unit 101.
The pre-processing means 103 pre-processes these texts and sends them to the N-gram converting means 104 as WINNT4 and WINDOWSNT.

(2) In the N-gram conversion means 104, these are first fragmented into 2-gram elements, and the one shown in FIG. 3B is output to the similarity calculation means 105 to calculate the similarity between the 2-gram elements. The calculation is performed (3/10 in this example), and then fragmented into 3 gram elements, and the one shown in FIG. 3C is output to the similarity calculation means 105 to calculate the similarity in the 3 gram elements. (1/10 in this example).

(3) Then, the degree of similarity in the 2-gram element and the degree of similarity in the 3-gram element are sent to the degree-of-similarity adding means 106 to obtain the sum of these, which is then passed through the output interface means 108. , The text is output and displayed on the terminal device to which the text is input.

Since the degree of similarity changes depending on the value of N and the value of a plurality of types N, the criteria of similarity and dissimilarity differ depending on each case.

In the above case, N-gram conversion means and similarity calculation means may be provided for each of N types.
You may provide one by one. If a plurality is provided, the speed can be increased, and if only one is provided, cost can be saved.

C. Third embodiment of the present invention A third embodiment of the present invention will be described based on FIG.

In FIG. 4, the similar text search device 3 is
A plurality of texts whose similarity is to be calculated are stored in the text database 109 in advance. Thus, when calculating the similarity, it is not necessary to input a plurality of texts from a terminal device (not shown), and only one text needs to be input, so that the input cost is significantly reduced and the similarity can be calculated at high speed. can do. Since a plurality of texts are stored in the text database 109, the texts having a high degree of similarity are sequentially output via the sorting means 107.

(1) First, the text database 109
In advance, a plurality of texts T ₁ , T ₂ , T ₃ ... For which the similarity is calculated are stored from an input means (not shown).

(2) The user inputs the text T ₀ from the terminal device (not shown). This text T ₀ is sent to the preprocessing means 103 via the input interface means 102, the preprocessing is performed, and the N-gram converting means 10 is executed.
4 is fragmented into N-grams, for example, 2-gram elements, and sent to the similarity calculation means 105.

(3) In the text database 109, the text T ₁ is read out first, and the preprocessing means 10 is read.
3 is pre-processed, and the N-gram conversion means 104 fragments the same as the text T ₀ into 2-gram elements, and the similarity calculation means 105 fragments the similarity S between the texts T ₀ and T _1.
1 is calculated and sent to the sorting means 107.

(4) Next, the text database 109
From the text T₂Is read out and the similarity is calculated in the same way.
Text T by means 105₀And T₂Similarity S with₂But
It is calculated and sent to the sorting means 107. in this way
Text database 109 to text T₃,
T_Four... is read and the text T₀Similarity with
S ₃, S_Four... is calculated and sent to the sorting means 107.
To be done.

(5) After the similarities have been calculated for all the texts in the text database 109 in this way, the sorting means 107 sorts the similarities in descending order, and all of them or in advance. A predetermined number is output, and these are output to the terminal device of the user who has input the text T ₀ via the output interface unit 108.

In the above description, the preprocessing means and N-gram converting means for the text T ₀ input by the user are
The example in which the preprocessing means and the N-gram converting means for the texts T ₁ , T ₂ ... Read out from the text database 109 are used separately, but of course the same ones may be used. Good.

D. Fourth Embodiment of the Present Invention A fourth embodiment of the present invention is shown in FIG. 5, in the similar text search device 4, the text T ₀ input by the user and the texts T ₁ , T ₂ ... Stored in the text database 109 are respectively converted into a plurality of types of N-gram elements having different N values. Create, calculate the similarity,
The degree of similarity is recognized by adding the degrees of similarity.

The operation of FIG. 5 will be described.

(1) First, the text database 109
In advance, a plurality of texts T ₁ , T ₂ , T ₃ ... Which are the objects of similarity calculation are stored from an input means (not shown).

(2) The user inputs the text T ₀ from the terminal device (not shown). This text T ₀ is sent to the preprocessing means 103 via the input interface means 102, the preprocessing is performed, and the N-gram converting means 10 is executed.
4, 104 are fragmented into a plurality of types of N-gram elements having different values of N, for example, a 2-gram element and a 3-gram element,
It is sent to the similarity calculation devices 105 and 105, respectively.

(3) In the text database 109, the text T ₁ is read out first, and the preprocessing means 10 is read.
3 is pre-processed, and the N-gram converting means 104, 104 fragment the same into the 2-gram element and the 3-gram element as the text T _0, and the fragment is sent to the similarity calculating means 105, 105, and the similarity calculating means 105, 105. The similarity S ₁₂ and S ₁₃ between the texts T ₀ and T ₁ are calculated on the basis of the 2-gram element and the 3-gram element, and these are calculated by the similarity adding means 10
6 and the sum (S ₁₂ + S ₁₃ ) is obtained and sent to the sorting means 107.

(4) Next, the text database 109
The text T ₂ is read from the text T ₂ , and the similarity calculation means 105, 105 calculates the similarities S ₂₂ and S ₂₃ based on the 2-gram element and the 3-gram element of the texts T ₀ and T _{2 in} the same manner. The sum (S ₂₂ + S ₂₃ ) is sent to the degree adding means 106 and is then sent to the sorting means 107.

(5) In this way, the texts T ₃ , T _4, ... Are read from the text database 109 and the similarity (S ₃₂ , S) based on the 2-gram element and the 3-gram element with the text T ₀ is read. ₃₃ ), (S ₄₂ , S ₄₃ ) ...
Are calculated, and these are sent to the similarity adding means 106 to obtain their sums (S ₃₂ + S ₃₃ ), (S ₄₂ + S ₄₃ ), and sent to the sorting means 107.

(6) After the sum of the similarities of the 2-gram element and the 3-gram element is calculated for all the texts in the text database 109, the sorting means 1
07 is sorted in descending order of the value of the sum of these similarities and outputs all or a predetermined number, and the terminal device of the user who inputs the text T ₀ via the output interface means 108. These are output to.

In the above description, the preprocessing means and N-gram converting means for the text T ₀ input by the user are
The example in which the preprocessing means and the N-gram converting means for the texts T ₁ , T _2, ... Read out from the text database 109 are used separately has been described, but the same may be used. . The same N-gram conversion means and similarity calculation means may be used.

E. Fifth Embodiment of the Invention A fifth embodiment of the invention will be described with reference to FIGS. 6 to 9. FIG. 6 shows a fifth embodiment of the present invention, FIG. 7 is an explanatory diagram of the index database, FIG. 8 is an explanatory diagram of an operation when data is registered in the index database, and FIG. 9 is a search / similarity degree. It is an operation explanatory view at the time of calculation.

In FIG. 6, in the similar text search apparatus 5, the relationship between the N-gram elements of the texts T ₁ , T ₂ ... Which are the search targets in the index database 111 and the text is, for example, FIG. 7D. As shown in FIG. 4, the data is stored in a table.

Texts to be searched T ₁ , T ₂ , T ₃
However, as shown in FIG. 7 (A), WindowsNT, Wi
Ndows2000, when WinME, as shown in FIG. 7 (B), for example, be 2 g of, as shown in (D), each 2 g element WI, IN · · · and a text name T _1, T ₂ thereof belonging , T ₃ are stored. In the text T ₂ , two 00 elements are created as 2-gram elements, but only one element is left for the same element.
It becomes something like. And t ₁ is the number of 2-gram elements of the text T ₁ (in this case t ₁ = 8), t ₂ is the text T
The number of ₂ 2 grams elements (t ₂ = 9 in this case), t ₃ Number 2 grams elements of text T ₃ are (in this case t ₃ =
4) is shown.

Then, the text T ₀ (W shown in FIG.
inNT4), the 2-gram element of the text T ₀ is collated with the 2-gram elements of the texts T ₁ , T ₂ and T ₃ stored in the index database 111, and the number of matches is obtained. At this time, the 2-gram elements existing only in the text T ₀ , NN and T4 are indexed.
It is registered in the database 111 and prepared for collation with other text. This makes the text T ₀ and T ₁ 3/10, T
₀ and T ₂ are 2/12, T ₀ and T ₃ can be obtained similarity of 2/7. Thus, the text T ₁ ,
2-gram elements of T ₂ and T ₃ and their number (also called frequency)
Since t ₁ , t ₂ , and t ₃ are registered, the similarity can be calculated at high speed.

In the index database 111, as shown in FIG. 7C, the texts T ₁ , T ₂ , T _{3 are written.}
N-gram elements (..., 2-gram elements in this example) may be stored.

The operation of FIG. 6 will be described for the case of registering data in the index database 111 shown in FIG. 8 and the case of similarity calculation shown in FIG.

(1) Regarding data registration, S1. First, in the text database 109, a plurality of texts T ₁ , T ₂ , T ₃ ... As shown in FIG. Then, the first text T ₁ obtained from the list is sent to the pre-processing means 103, and the pre-processing such as lower case lettering, blanks, and deletion of hyphens is performed for keyword cleaning.

S2. The text thus preprocessed is sent to the N-gram conversion means 104, and is fragmented into, for example, 2-gram elements.

S3. When the same 2-gram element exists in the same text, duplicate 2-gram elements are deleted to make one.

S4. The 2-gram element of the text T ₁ thus obtained is converted by the database access interface means 110 into the text name T ₁ and the 2-gram element number t.
_It is stored in the index database 111 together with ₁ . In this way, similar processing is performed for the other texts T ₂ and T ₃ obtained from the list of the text database 109, and the index database 111 is created as shown in FIG. 7D.

(2) Calculation of similarity S10. The user selects a terminal device (not shown) from FIG.
Input the text T ₀ shown in (A). The input text T ₀ (input keyword) is sent to the preprocessing unit 103 via the input interface unit 102 to be preprocessed and the input keyword is cleaned.

S11. The text T ₀ preprocessed in this way is processed by the N-gram conversion means 104.
Similar to ₁ , T ₂ and T ₃ , it is fragmented into 2 gram elements as shown in FIG.

S12. In this case, if there are duplicate 2-gram elements, only one of them is left and the other duplicate 2-gram elements are deleted.

S13. When the 2-gram element of the input text T ₀ thus obtained is input from the N-gram converting means 104 to the similarity calculating means 105, the similarity calculating means 1
05 is a database access interface means 1
The record of the text T ₁ shown in FIG. 7D is acquired from the index database 111 via 10.

S14. Then, the degree of similarity is calculated. This gives a similarity S ₁ = 3/10 between the texts T ₀ and T ₁ .

S15. In this way, the similarities S ₂ and S ₃ between all the other texts T ₂ and T ₃ entered in the list and the text T ₀ are calculated, and S ₂ = 2/12 and S ₃ = 2/7
Is obtained.

S16. These similarities S ₁ , S ₂ , S ₃
Is sent to the sorting means 107. The sorting means 107 sorts these similarities in the descending order and outputs them to the terminal device of the user who has input the text T ₀ via the output interface means 108.

In the above description, the preprocessing means and N-gram conversion means for the text T ₀ input by the user are read out from the text database 109.
_Although an example in which a pretreatment means for ₁ , T _2, ... And another means for N-gram conversion are used has been described, these may be the same.

F. Sixth Embodiment of the Present Invention A sixth embodiment of the present invention will be described with reference to FIG. 10, in the text similarity search device 6, the text T ₀ input by the user and the texts T ₁ and T ₂ stored in the text database 109.
.. creates a plurality of types of N-gram elements each having a different N value, calculates the degree of similarity, and adds the degrees of similarity to recognize the degree of similarity. Also, a plurality of index databases 111 are prepared for a plurality of types of N-gram elements having different N values.

(1) First, the text database 109
In FIG. 7, a plurality of texts T ₁ , T ₂ and T ₃ which are the objects of similarity calculation are stored in advance from the input means (not shown) as shown in FIG. 7A. Then the first text T
_After sending ₁ to the pre-processing means 103 to perform pre-processing,
A plurality of types of N-gram elements having different values of N, for example, two-gram elements and three-gram elements, are fragmented by the N-gram converting means 104, 104, and a plurality of types of different values of N are respectively passed through the database access interface means 110. It is stored in the index databases 111, 111. Similar processing is performed on the texts T ₂ and T ₃ , and the texts are stored as shown in FIG.

(2) The user inputs the text T ₀ from the terminal device (not shown). This text T ₀ is sent to the preprocessing means 103 via the input interface means 102, the preprocessing is performed, and the N-gram converting means 10 is executed.
4, 104 are fragmented into a plurality of types of N-gram elements having different values of N, for example, a 2-gram element and a 3-gram element,
It is sent to the similarity calculation devices 105 and 105, respectively.

(3) Index database 111,
In 111, first, the 2-gram element and 3-gram element for the text T ₁ are read out, and the 2-gram element and 3-gram element for the text T ₀ and the similarity calculation means 105,
In 105, the similarity between two-gram elements and between three-gram elements is calculated, and the similarity S ₁₂ between two-gram elements of the texts T ₀ and T ₁ and the similarity S ₁₃ between three-gram elements are calculated. The sum (S ₁₂ + S ₁₃ ) is obtained by sending to the degree adding means 106, and sent to the sorting means 107.

(4) Next, the index database 11
The 2-gram element and 3-gram element corresponding to the text T ₂ are read out from Nos. ₁ and 111, and similarly, the similarity calculation means 105, 105 calculates the similarity S based on the 2-gram element and 3-gram element of the texts T ₀ and T _{2. 22} and S ₂₃ are calculated, and these are sent to the similarity adding means 106 to obtain the sum (S ₂₂ + S ₂₃ ) and sent to the sorting means 107. In this way, the similarities S ₃₂ and S ₃₃ are calculated based on the 2-gram element and the 3-gram element of the texts T ₀ and T ₃ and the sum (S ₃₂₎ is calculated by the similarity adding means 106.
+ S ₃₃ ) is obtained and sent to the sorting means 107.

(5) The sorting means 107 sums the similarities sent from the similarity adding means 106 (S ₁₂ + S ₁₃ ), (S
₂₂ + S ₂₃ ), (S ₃₂ + S ₃₃ ) are sorted in descending order, and the output interface means 108 is sorted in the sorted order.
The text T ₀ is output to the terminal device of the user who has input the text.

In the above description, the preprocessing means and N-gram conversion means for the text T ₀ input by the user are
Although an example in which different preprocessing means and N-gram conversion means are used for the texts T ₁ , T _2, ... Read from the text database 109 has been described, the same may be used. The N-gram conversion means and the similarity calculation means index database may also be the same. Those having the same function may use the same one.

In the above description, an example in which the text is described by a program name or the like has been described, but the present invention is not limited to this. The same applies to Japanese documents. For example, in the case of a book search system, even if the correct title of a book is forgotten, a title candidate can be obtained by inputting a famous document that is a part of it.

By performing a fuzzy search with respect to a specific search target, it is possible to perform a search that absorbs fluctuations in input or input errors. By putting preprocessed data to be searched in the index database, it is possible to increase the response speed after the searcher inputs.

When the lengths of the two texts are extremely different, the common N-gram element can be quoted by the number of N-gram elements of the shorter text to obtain the similarity. As a result, the similarity calculation can be performed faster than the similarity calculation based on the length of the longest subsequence that has been conventionally performed.

In order to further increase the speed when the present invention is applied to a search engine, the pre-processing, N-gram conversion, deletion of duplicate N-gram elements, etc. are performed on the text stored in advance as data. By storing, it is possible to increase the speed.

The embodiments of the present invention will be additionally described below.

(Supplementary Note 1) Input interface means for inputting text, preprocessing means for performing preprocessing on the input text, N-gram conversion means for creating N-gram elements for the text, and N for a plurality of texts.
A similar text search device comprising: a similarity calculation means for calculating the degree of coincidence of gram elements; and an output interface means for outputting the calculation result of the similarity calculation means.

(Supplementary Note 2) Input interface means for inputting text, preprocessing means for preprocessing input text, and N-gram conversion means for creating N-gram elements having different N values for text. A similar text retrieval device comprising: a similarity calculation means for calculating the degree of coincidence of N-gram elements relating to a plurality of texts based on the frequency of N-grams; and an output interface means for outputting the calculation result of the similarity calculation means. .

(Supplementary Note 3) Input interface means for inputting text, text database means for holding text to be a similarity calculation object, preprocessing means for preprocessing text, and N-gram element for text , An N-gram conversion means for creating the N-gram element, a similarity calculation means for calculating the degree of coincidence of N-gram elements for a plurality of texts, a sorting means for outputting the calculation result in the descending order of the degree of coincidence, and a sorting result of the sorting means. A similar text search device comprising output interface means for outputting

(Supplementary Note 4) Input interface means for inputting text, text / database means for holding text to be a similarity calculation object, preprocessing means for preprocessing text, and N value for text Multiple N's that create multiple types of N-gram elements with different
The grammarizing means, the similarity calculating means for calculating the degree of similarity for each of the plurality of different N-gram elements by the frequency of each N-gram element, and the value for calculating the degree of similarity by the frequency of each N-gram element are added. A similar text search device comprising: a similarity adding means, a sorting means for outputting the outputs of the similarity adding means in descending order, and an output interface means for outputting a sorting result of the sorting means.

(Supplementary Note 5) Input interface means for inputting text, text database means for holding text to be subjected to similarity calculation, preprocessing means for preprocessing text, and N-gram element for text , An index database means for index-holding the N-gram elements created for the text held in the text database means, an access means for the index database means, and a plurality of texts. The similarity calculation means for calculating the degree of coincidence of the N-gram element, the sorting means for outputting the calculation result in the descending order of the degree of coincidence, and the output interface means for outputting the sorting result of the sorting means. And similar text search device.

(Supplementary Note 6) Input interface means for inputting text, text / database means for holding text to be subjected to similarity calculation, preprocessing means for preprocessing text, and N value for text Multiple N's that create multiple types of N-gram elements with different
Different types of N created for the grammarizing means and the text held in the text database means
Index database means for holding the gram element as an index, access means for the index database means, and similarity calculating means for calculating the degree of similarity of the N-gram elements of the different plurality of types by the frequency of each N-gram element. , A similarity adding means for adding a value of which the similarity is calculated according to the frequency of each N-gram element, a sorting means for outputting the outputs of the similarity adding means in descending order, and a sorting result of the sorting means. A similar text search device comprising output interface means.

[0096]

According to the present invention, the following effects can be obtained.

(1) Since each N-gram element is created for each text and the matching is performed, the text matching can be searched in a form that absorbs the blurring of the expression, so that the fuzzy search can be accurately executed.

(2) Since one of the texts to be compared is held in the text database in advance, it is not necessary to input all the texts to be compared each time the search is performed, and similar texts can be searched at high speed.

(3) One of the texts is held in the database, a plurality of types of N-gram elements having different N values are created, and the degree of similarity is calculated according to the frequency.
For example, in the case of N = 2, that is, in the case of a 2-gram element, even if the apparent similarity is increased by matching the particle part, this can be suppressed in the case of a 3-gram element, and the similarity determination The speed and accuracy of the results can be improved.

(4) Since the N-gram element of the text stored in advance in the text database is created and stored in the index database, it is stored in this index database when the texts are compared. It is possible to quickly calculate the similarity to the input text using the N-gram element.

(5) A plurality of types of N-gram elements having different N values are created for the input text, and a plurality of types of N-gram elements having different N are also created for the text held in the text database. Since it is stored in the index database, the similarity depending on the frequency of N-gram elements can be performed at high speed, and the accuracy of the similarity can be improved.

[Brief description of drawings]

FIG. 1 is a first embodiment of the present invention.

FIG. 2 is a second embodiment of the present invention.

FIG. 3 is an operation explanatory diagram of the second embodiment of the present invention.

FIG. 4 is a third embodiment of the present invention.

FIG. 5 is a fourth embodiment of the present invention.

FIG. 6 is a fifth embodiment of the present invention.

FIG. 7 is an explanatory diagram of an index database.

FIG. 8 is an operation explanatory diagram when registering data in an index database.

FIG. 9 is an explanatory diagram of an operation at the time of searching and calculating a similarity.

FIG. 10 is a sixth embodiment of the present invention.

[Explanation of symbols]

1-6 Similar text search device 101, 102 input interface means 103 pretreatment means 104 N-gram conversion means 105 similarity calculation means 106 similarity adder 107 sorting means 108 output interface means 109 Text Database 110 Database Access Interface Means 111 Index Database

Claims (5)

[Claims] 1. Input interface means for inputting text, preprocessing means for preprocessing input text, N-gram conversion means for creating N-gram elements for text, and N-gram elements for a plurality of texts. 2. A similar text search device comprising: a similarity calculation means for calculating the degree of coincidence and an output interface means for outputting the calculation result of the similarity calculation means. 2. An input interface means for inputting text, and a text for holding text to be a similarity calculation target.
Database means, preprocessing means for preprocessing text, N-gram conversion means for creating N-gram elements for text, similarity operation means for calculating the degree of coincidence of N-gram elements for a plurality of texts, A similar text search device comprising: sorting means for outputting a calculation result in descending order of frequency and output interface means for outputting a sorting result of the sorting means. 3. An input interface means for inputting text, and a text for holding text to be a similarity calculation target.
The database means, the preprocessing means for performing preprocessing on the text, the plurality of N-gram converting means for creating a plurality of types of N-gram elements having different N values for the text, and the different plurality of types of N-gram elements, respectively. The similarity calculation means for calculating the similarity according to the frequency of N-gram elements, the similarity addition means for adding the values calculated for the similarity with the frequency of each N-gram element, and the output of the similarity addition means are large. A similar text search device comprising a sorting means for outputting in order and an output interface means for outputting a sorting result of the sorting means. 4. An input interface means for inputting text, and a text for holding text to be a similarity calculation target.
Database means, preprocessing means for preprocessing text, N-gram conversion means for creating N-gram elements for text, and N-gram element creation for N-gram elements created for text held in the text database means. Index database means, access means for the index database means, similarity calculating means for calculating the degree of coincidence of N-gram elements relating to a plurality of texts, and sorting means for outputting operation results in descending order of the degree of coincidence And a similar text search device comprising output interface means for outputting the sorting result of the sorting means. 5. An input interface means for inputting text, and a text for holding text to be a similarity calculation target.
The database means, the preprocessing means for preprocessing the text, the plurality of N-gram converting means for creating a plurality of types of N-gram elements having different N values for the text, and the text held in the text database means A plurality of different types of N-gram elements that are stored in the index database means, an access means for accessing the index database means, and a plurality of different types of N-gram elements according to the frequency of each N-gram element. A similarity calculation means for calculating the similarity, a similarity addition means for adding the values calculated by the frequencies of the respective N-gram elements, and a sorting means for outputting the outputs of the similarity addition means in descending order. , Output interface means for outputting the sorting result of this sorting means Similar text search apparatus, characterized in that Bei was.