JPH0668159A - Retrieval device - Google Patents

Abstract

PURPOSE:To reduce the capacity of a table without reducing an indexing efficiency by using 2-character concatenation selected based on the frequency of appearance for the table for retrieval pre-processing especially with respect to the retrieval device based on a full-text search retrieving document data on request from lots of document data without provision of retrieval index information. CONSTITUTION:The device is provided with a 2-character concatenation extract means 1 extracting 2-character concatenation from a retrieval object document and a spreadsheet generating means 4 selecting the 2-character concatenation to generate a spreadsheet based on the means 2 extracting appearance frequency information of 2-character concatenation for each file, and 2-character concatenation is extracted from a requested retrieval character string similarly, 2-character concatenation is selected based on the frequency of appearance, a retrieval entry and retrieval content 6 for the selected 2-character concatenation is used and a spreadsheet retrieval means 7 are used to obtain an object document 8, a detailed retrieval means 9 is used for the object document 8 to retrieve it and then the result of retrieval is obtained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a search device based on a full-text search method for extracting requested document data without using index information which is manually assigned to a large amount of document data in an information processing device. It is about.

[0002]

2. Description of the Related Art In recent years, word processors and personal computers have become widespread due to the development of computer peripheral technology, and a large amount of document data has come to be used at work and at home. In order to organize and effectively use such a large amount of document data, a large-capacity data storage device and a high-speed search machine have been researched and developed. However, in the conventional search machine, it is necessary to add index information to the original data in advance for the search, and if the index amount is manually increased as the amount of data increases, a great deal of labor is required. There is also an attempt to automatically extract the index on a computer, but the natural language processing technology that is the basis of the process of extracting the keyword that will be the index from the document is not yet completed, and it is the current situation that it is not fully automated. Is. On the other hand, a full-text search method has been researched and developed as a method for searching without adding index information. This full-text search method has a drawback in that the search speed is slower than the search method using the index information for search. As a method for solving this drawback, for example, a text search machine for full-text search (electronic information IEICE Technical Report / Data Engineering 89
-38), and a method for speeding up full-text search by encoding including the attributes / positions of constituent characters (IEICE Technical Report / Data Engineering 90-24). Further, as a pre-process for searching the text, a method of speeding up the search by using two-character concatenation to narrow down search target documents has been proposed.

[0003]

However, when all two-character concatenations existing in a sentence are used, if a table is created for two-character concatenations that appear very frequently in each document, the search target is narrowed down. The efficiency will not increase and the capacity of table data will increase. For example, two-character concatenation "Aru" and "Suru" that are often used in Japanese documents.
If "ru.", "Kore", etc. are present in most of the document files to be searched, even if they are used for creating a table, the capacity of the table itself is increased without playing the role of narrowing down.

According to the present invention, the appearance frequency of two-character concatenation is analyzed for each document, and two-character concatenation is selected from the result to create a table, thereby reducing the volume of table data without greatly reducing the narrowing rate. It is an object of the present invention to provide a search device capable of achieving the above.

[0005]

In order to solve the problems, a retrieval apparatus of the present invention extracts a two-character concatenation from a Japanese character code string of a document to be retrieved, and the two-character concatenation is each document file. Frequency information that appears in the, and the first character of the two-character concatenation selected based on the frequency information is used as an entry and the remaining one character and the two-character concatenation are extracted. Table is created, two-character concatenation is selected from the search character string input at the time of the search request based on the appearance frequency, the first character of the selected two-character concatenation is the search entry, and the second character is the search content. As a result, the search contents are searched from the created table using the search entry to obtain the corresponding recording position outline information,
After the search target documents are narrowed down, a detailed search is performed for the search target documents and the document data satisfying the search request can be output.

[0006]

According to the present invention, as described above, the two-character concatenation is extracted from the document data to be searched, the frequency information that the two-character concatenation appears in each document file is obtained, and based on the frequency information. Since the table is created by selecting the two-character concatenation, the capacity of the table can be reduced compared to the case of creating the table for all the two-character concatenations that appear, and the selection efficiency can be reduced by selecting the most frequent ones. Nor. If the table capacity can be reduced, the table search time can be shortened,
It is possible to increase the storage capacity of the document data to be searched.

Therefore, the storage medium for full-text search can be effectively used without lowering the search efficiency.

[0008]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a functional block diagram of a search method according to the present invention. In FIG. 1, the two-character concatenation extraction means 1 from the Japanese character code string of the document to be searched sequentially outputs two.
Character concatenation is extracted, and the frequency of appearance of the two character concatenation existing in each document file is extracted using the second frequency information extraction means. The remaining 2 characters are selected after selecting the 2-character concatenation based on the appearance frequency and deleting the duplicate 2-character concatenation in the same document.
1 character of concatenated character as an entry and 1 remaining character and 5
A table is obtained by the table creating means 4 from the recording position outline information about the search target document in which the two-character concatenation is extracted. Similarly, the two-character concatenation extraction means of 1 is sequentially extracted from the requested search character string, and the two-character concatenation is selected from the document to be searched based on the per-file frequency information obtained by using the two frequency information extraction means. Then, the 6 search entries and the search contents are obtained. The table is searched using the table search means 7 according to the search entry and the search contents, the recording position outline information of the corresponding search candidate document is obtained, and the requested search character string is used in the search candidate document by the detailed search means 9 You can search for and get search results.

FIG. 2 is a specific example of table creation using frequency information according to the present invention. As a Japanese character string, for example, "..., comparing Tokyo, the capital of Japan, with Kyoto, the ancient capital,
... "is obtained by a two-character concatenation extraction means such as" Japan "," book "," neck of ... "" De "," Aru "... The frequency information extraction means obtains as frequency information the number of files in which the appearing two-character concatenation exists in all files of the target sentence. Based on the frequency information, select only those whose appearance frequency is lower than the set threshold value, obtain entries and contents, and create a table. The higher the appearance frequency is, the more the two-character concatenation exists between the files. Therefore, the effect of narrowing down using the file information decreases in proportion to the appearance frequency, and the capacity of the table increases. Especially, regarding the two-character concatenation existing in all the files, the effect of narrowing is completely eliminated, and the ratio occupied as a table element is large. From the above, it is possible to create a two-character concatenation only when the frequency of appearance is below the threshold.
It is possible to reduce the capacity of the table without significantly reducing the narrowing rate, and it is possible to reduce the time required for table search.

FIG. 3 is a specific example in which the appearance frequency of two-character concatenation included in a patent document is graphed. The target data is
Patent document 190 files (Average file size 52 [Kby
te] and the total file capacity 10 [Mbyte]) are used. The horizontal axis represents the number of files, and the vertical axis represents the number of types of two-character concatenation that appears for that number of files. The number of types of appearance with respect to the number of files of two-character concatenation decreases exponentially as the number of files increases, but conversely increases near the total number of files. This tendency is almost the same regardless of the target document data. The data show that unless the two-character concatenation that is close to the total number of files, that is, the appearance frequency is selected, the reduction rate can be reduced by 10% to 20% with almost no reduction in the narrowing rate.

In the present invention, if the configuration is such that the one with a high appearance frequency is not selected, it is not possible to distinguish the one that does not actually exist from the one that is not selected because the appearance frequency is high. Therefore, in order to distinguish those that have appeared but are not selected by the frequency information, it is possible to register the special code in the table only once for each two-character concatenation.
As a result, if there is something that does not actually exist in the search request character string, the result of the search does not require the detailed search time because the candidate document file does not exist before the detailed search. Can be displayed.

The two-character concatenation having a high appearance frequency is divided into a part that depends on the target document data and a part that is frequently used in a character string of a Japanese sentence. For example,
If the document data is a patent, it depends on the document 2
The words such as "invention", "apparatus", "method", "effect", and the like used for items as character concatenation are given. The most frequently used Japanese words are particles and auxiliary verbs such as “suru”, “naruru”, “mono”, and “ar”. When the table for the frequency is prepared in advance, the analysis result focusing only on the latter will be used.

As a characteristic of Japanese, punctuation marks ",""."
It is difficult to search for special symbols such as and words with only hiragana. Hiragana such as particles, auxiliary verbs, and verbs often come before punctuation marks, and because they have no meaning in themselves, they are rarely included in the constituent elements of the request search character string as full-text search. In addition, a character string that consists of only hiragana is often used as a phonetic kana,
There is almost no problem if the system is configured to search after converting kanji. From the above, it is possible to adopt a configuration in which two-character concatenation including punctuation marks or two-character concatenation starting with hiragana is not selected in the search target document character string. The frequency of occurrence of hiragana is around 40% in normal documents, so if you do not select two-character concatenation that starts with hiragana, the table capacity will be deleted by around 40% and a significant capacity reduction can be realized. . It should be noted that it is better to treat hiragana in the same way as other characters in situations where it is frequently used as a search word that includes hiragana, as it affects the search time. It is easy and does not exclude them from the scope of the present invention.

[0015]

As described above, according to the present invention, the following effects can be obtained.

If two-character concatenation that does not exceed the preset threshold value at the time of creating the table is selected using the appearance frequency information for each file of two-character concatenation, only those that have a narrowing-down effect by the table will be deleted. Since the capacity of the table to be deleted is large, it is possible to significantly reduce the capacity of the table without substantially reducing the narrowing rate by the table at the time of retrieval.

[Brief description of drawings]

FIG. 1 is a functional block diagram of a search method according to the present invention.

FIG. 2 is a diagram showing a specific example of creating a table using frequency information according to the present invention.

FIG. 3 is a graph showing a specific example of a two-character continuous appearance frequency in the present invention.

[Explanation of symbols]

 1 2 Character concatenation extraction means 2 File-by-file appearance frequency information extraction means 3 2 Character concatenation selection means 4 Table creation means 5 Recording position summary information 6 Search entry and search content 7 Table search means 8 Search candidate document 9 Detailed search means

Claims (3)

[Claims] 1. A two-character concatenation extraction means for extracting an arbitrary two-character concatenation from a Japanese character code string of a search target document in a storage medium, and frequency information for obtaining the frequency of the two-character concatenation appearing in each document file. Extraction means, means for selecting a two-character concatenation whose appearance frequency does not exceed a preset threshold value in the two-character concatenation, and the remaining one character and two characters with the first character of the selected two-character concatenation as an entry. A table creating means for creating a table containing the recording position outline information of the document to be searched for which the character connection is extracted, and connecting the two characters from the search character string input at the time of the search request based on the appearance frequency. By selecting the first character of the selected two-character concatenation as the search entry and the second character as the search content, the search content is searched from the created table by using the search entry, and the corresponding recording position outline information is searched. A search device characterized by performing a detailed search for a search target document after acquiring information and narrowing down the search target document. 2. The search device according to claim 1, wherein a character combination that forms a two-character concatenation that includes punctuation marks "." And "," is not selected when a table is created. 3. The retrieval apparatus according to claim 1, wherein character combinations that form a two-character concatenation that do not begin with hiragana are not selected when creating a table.