JP2001344282A - Method and device for document retrieval - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such problem that the conventional full text search system requires much processing time because an entire text file is scanned from the beginning character by character and this system requires much time when being applied to a large- scale database. SOLUTION: When a document is registered in a document database, a text character string of the registered document is divided for every character type such as Hiragana (cursive form of Japanese syllabary) or KANJI (Chinese character), and the inclusion relations between divided partial character strings are examined to generate a condensed text consisting of a set of partial character strings where character strings included in other character strings are excluded, and a character component table is generated where characters appearing in the condensed text are registered without duplication, and the condensed text and the character component table are registered in the document database besides the text of the registration object document. At the time of retrieval, the character component table is referred to extract a document including characters of a designated keyword, and next, the condensed text of the extracted document is referred to extract only documents corresponding to the condensed text including the partial character string of the designated keywords, and texts of extracted documents are referred to extract only a document meeting a retrieval condition given between keywords.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a search technique for searching a document database by specifying a character string for the entire text of a document, and more particularly to a high-speed search process using auxiliary files. Related to search technology.

[0002]

2. Description of the Related Art In a conventional document retrieval system, a method is employed in which a word (referred to as a keyword) representing the contents of a document to be registered is indexed. However, in this method, an indexer called an indexer had to read the document one by one, understand the contents, and assign appropriate keywords. In order to avoid the troublesome work at the time of registration, there has been proposed a method of registering all words appearing in the text as keywords in an index file as disclosed in JP-A-63-198124. However, with the above method, it is difficult to determine the smallest unit word that has meaning when creating an index file, and the analysis of a sentence fails due to inadequate word dictionary or grammatical rules. Is not extracted as a keyword. In order to solve this problem, a document is directly registered in a computer as character-coded text at the time of retrieval,
At the time of retrieval, a full-text search has been proposed that reads the contents of all documents in a text database and searches for documents containing a given keyword (hereinafter referred to as a search term to distinguish it from control keywords in conventional systems). ing.

This full-text search method is described in "Information Processing Society of Japan Vol. 89, No. 66, Informatics Fundamentals 14-7 Text Database Management System SIG"
As described at the beginning of the second section of "MA and its applications (1989. 7.27)", a major feature is that the entire text file is scanned one character at a time from the beginning. By doing so, even if there is no index file in which a document identifier or the like corresponding to the keyword is described, it is possible to search using the text body of the text database as a clue. That is, a character string search can be performed on the entire text data using a given search term, and a document in which the search term is described can be output as a search result.

[0004]

The conventional full-text search method has a problem that it takes a long processing time to scan the entire text file one character at a time from the beginning, and takes a long time when applied to a large-scale database. As seen in the second section of the document, even with a general-purpose large computer, only a search processing speed of about 2 MB / s can be realized. Even at this speed, if the database is several megabytes, the search time will be within the practical range. However, a practical-scale database such as an office requires a capacity of several hundred megabytes, and in this case, a sufficient search response cannot be obtained.

An object of the present invention is to provide a high-speed document search method and apparatus.

[0006]

In order to solve the above-mentioned problems, a document search method including the following processing steps is used, and an apparatus for implementing the method is configured. (1) Step of storing the text itself (2) Decomposing the stored text into partial character strings at the word level, examining the mutual inclusion of the character strings between the decomposed partial character strings, and including them in other partial character strings Step of creating a condensed text composed of a set of partial character strings excluding character strings to be used (3) Step of generating a character component table in which characters used in the text are collected without duplication (4) Given search term Is decomposed at the character level, and a character component table search step is performed to extract a document containing all the characters constituting the search term. (5) Referring to the condensed text corresponding to the document extracted in the character component table, Step of Condensed Text Search for Extracting Documents Containing Search Terms (6) If a given search condition expression specifies a positional relationship in the text between a plurality of search terms, the condensed text extracted Sentence Corresponding to reference the text data includes a given search term, step text search yet for extracting satisfy the search conditions of the positional relationship or the like which is applied between the search terms

As described above, the character component table search and the condensed text search are provided by providing a hierarchical pre-search means for narrowing down the text component table search and the condensed text search and finally performing the text search. Documents that do not satisfy the conditional expression can be truncated before referring to the text body to reduce the amount of searching for the text body to be searched. That is, it is possible to reduce the entire search processing time by reducing the text search processing time, which accounts for a large proportion of the search processing time. For example, given a conditional expression that specifies the positional relationship in the body of two search terms, "Search for documents in which the" image "and" processing "are in the same sentence (sentence) in the body" In the conventional method, the search processing speed is 2 MB /
Assuming s, it takes 250 seconds, or about 4 minutes, to search through the full 500 MB full text. However, in the case of hierarchical pre-search, in a typical case, the character component table accounts for 10% of the entire database,
If you can narrow down to 30%, the capacity of the condensed text is 30% of the text
In the case of, the capacity of the character component table is so small that it can be ignored from the viewpoint of the whole database. Therefore, the capacity of the condensed text to be searched is 15 MB, and the search amount of the text data is 1% of the total database size, that is, 5 MB. Even at a search speed of 2 MB / s, the search process can be completed in 10 seconds. As described above, in the “hierarchical pre-search method”, two-stage pre-searches of “character component table” and “condensed text” are performed in advance, and sieved at “character level” and “word level” respectively. By narrowing down the number of documents to be subjected to the time-consuming text search to a minimum in advance, the search document capacity can be reduced, and therefore, a very high-speed full-text search can be equivalently realized.
If the conditional expression is an AND, OR, or NOT condition in a single search term or a plurality of search terms, the result of the condensed text search can be used as the final search result. Because the words in the condensed text are
This is because there is no need to search the text again because it is always present in the text. As described above, the search at the "word level" can completely omit a text search that requires a long processing time, so that the entire search processing time can be further reduced. According to the full-text search method including the above processing steps, the amount of direct text search can be reduced in advance, so that high-speed full-text search can be performed.

[0007]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described below with reference to FIG. This device has a display 1
00, keyboard 101, central control unit CPU102,
Character component table 105, condensed text 104, text storage file 110, floppy disk driver 10
6, the main memory 200.

In the main memory 200, a text registration program 201, a condensed text creation registration program 202, a character component table creation registration program 203, a character component table search program 204, a condensed text search program 205, a text search program 206, a hierarchical type A search control program 207 is stored, and a data area 208 is secured. These programs are executed by the CPU 102. At the time of document registration, the CPU 102 reads document data from a floppy disk inserted into the floppy disk driver 106 by a command input from the keyboard 101, executes the text registration program 201 and sets the read document data as the text 103. It is stored in the file 110. Next, the CPU 102 executes the condensed text creation / registration program 202, divides the text 103 into partial character strings at the word level, checks the mutual inclusion of the character strings among the divided partial character strings, A condensed text composed of a set of partial character strings excluding the character strings included in the column is created, and the
Store to 0. Finally, the CPU 102 executes the character component table creation registration program 203 to create a character component table in which characters used in the text 103 are collected without duplication.
This is stored in the file 110 as the character component table 105. At the time of the search, a search condition expression input from the keyboard 101 is sent to the CPU 102. The CPU 102 first executes the hierarchical search control program 207, and sequentially executes the character component table search program 204, the condensed text search program 205, and the text search program 206 based on the control. That is, in the character component table search, the search term in the input search condition expression is decomposed at the character level, and a document including all the characters constituting the search term is extracted. Then, referring to the condensed text corresponding to the document extracted in the character component table, a document including the given search term is extracted. If only a single search term or a logical relationship between multiple search terms is specified in the given search condition expression, but not a positional relationship in the text, Ends the search here,
The result of the condensed text search is output as a search result. In other cases, that is, when a given search condition expression specifies a positional relationship in the text between multiple search terms, refer to the text data corresponding to the document extracted by the condensed text search. Then, those that include the given search terms and satisfy the search conditions such as the positional relationship given between the search terms are extracted and output as the search results. The above is the outline of the full-text search device according to the first embodiment of the present invention.

Hereinafter, registration and search methods of a character component table search, a condensed text search, and a hierarchical presearch method for performing text search and refinement, which are features of the present invention, will be briefly described. First, the "condensed text" and the "character component table" are created automatically when a document is registered. This processing content,
As shown in FIG. In this figure, when the document to be registered is entered,
First, it is stored as "text" as it is.

Next, a "condensed text" is created from the "text". The “condensed text” is created by dividing a character string from the “text” for each character type (Kanji, Hiragana, Katakana, English, etc.) and eliminating duplicate words that appear repeatedly.
If the text is document 1 "Search technique for fuzzy search ...", "search" is truncated as a duplicate word, and "fuzzy" and "search technique" and "for"
Will remain as "condensed text". Also, "text"
To create a "character component table". Here, characters appearing in the “body” are represented by 1-bit information. In the example of the document 1, "1" is set because there are "A" and "I", and "0" is set because there is no "U". Similarly, “1” is set for “detect” and “cord”, respectively. Similarly, in the corresponding character portion of the character component table, "1" is set when the character is present in "body", and "0" is set when it does not exist. In this way, a "condensed text" and a "character component table" are automatically created when a document is registered, and preparation for hierarchical presearch is made. At the time of retrieval, these auxiliary files are referred to in the reverse order of registration, as shown in FIG. First, in the character component table search, the character component table is referred to, and a character component table corresponding to all the characters in the search term and a character portion where “1” is set in a corresponding character portion is selected. Second
In the condensed text search, the condensed text of the document selected in the character component table is referred to, and a document including the search term given in the conditional expression is selected. Finally, in the text search, the search term whose appearance position in the text matches the conditional expression is selected. In the example of this figure, search [4C] understanding,
An example is shown in which a search is performed using the conditional expression “Search for a character in which“ search ”and“ understanding ”appear close to each other within four characters in the text”. As a result, a document in which “search” and “understanding” of document 4 are separated by 4 characters in the text is extracted.

Hereinafter, a method of creating a character type division / duplicate word elimination type condensed text and a character code dependent type character component table used in this embodiment, and a method of controlling a hierarchical presearch using these will be described in detail. First, a description will be given of a method of creating a character type division / duplicate word elimination type condensed text used in the present embodiment. As shown in FIG. 4, first, a character string is divided from the body text according to the character type. The character type at this time is
Kanji, Hiragana, Katakana, English letters, numbers, symbols, etc. The character string of the body is divided for each character string composed of a series of these single character types. Next, for each of the divided character strings, when the other partial character strings in the same document completely include the partial character strings, the character strings are excluded from the target of the condensed text as duplicate character strings. For example, since the partial character string “search” is completely included in another partial character string “intelligent search technology” in the same document, this “search” is not registered in the condensed text. However, in the condensed text search, even if the character string “search” is not registered in the condensed text, “intelligent search technology”
Will be hit as a substring.

As described above, the overlapping registration of the partial character strings is eliminated, and a separator is inserted between the character strings in the obtained partial character strings for each document as shown in FIG. In this drawing, symbols ',' are used as separators. In FIGS. 2 and 3, this separator is represented by the symbol "|", but this separator does not need to be particularly represented as a character, and a special code not assigned to a character can be used.

Next, a method of creating a character code dependent type character component table used in this embodiment will be described. As shown in FIG. 6, the character code dependent type character component table is a character component table that determines a bit position at which 1 is set as bit information indicating presence by a character code. In the figure, the shift JIS code is described as an example. In the figure, (XXXX) H is a character code displayed in hexadecimal. For example, to indicate that the character string “search” exists in the text of document 1, (8C9F) H, (8DF
5) Set 1 to H-th. The bit position corresponding to this character in the bit list is called the entry number of the character component table. For example, the entry number of “check” is (8C9
F) 35999 if H or decimal notation is displayed. The control and search operation of the hierarchical pre-search using the above character component table and the condensed text will be described. First, the search terms in the search condition expression are decomposed into character units, and a character component table search is performed. Here, a document having a bit list in which the positions of the entry numbers in the bit list corresponding to the character codes constituting the given search term are all 1 is obtained. For example, if the character string “search” is given as a search term, the document 1 in which the (8C9F) H and (8DF5) H-th bits of the bit list corresponding to “search” and “search” are all 1 2,3
4,. . . Is the search result of the character component table search.

That is, as shown in FIG. 7, a bit list 701 of the entry number of (8C9F) H indicating "detection"
AND operation is performed for each bit between the bit list 702 of the entry number of (8DF5) H indicating the 'search'
A bit AND operation result 703 is obtained. This bit AND
The document number corresponding to the bit position of 1 in the bit list of the calculation result 703 indicates a hit document as a search result of the character component table search. That is,
A document including all of the search and the search will be extracted. If the search term is composed of only one character such as "lake", the result of the character component table search can be output here and the search can be terminated. Next, a search is performed on the condensed text of the document extracted by the character component table search. Here, as shown in FIG. 5, the content of the condensed text registered for each document is scanned, and a document containing a given search term as a word is extracted. That is, a document in which the two characters “search” and “search” appear consecutively as “search” is extracted. In other words, even if the words “search” and “search” are included, words that appear as different words, such as “detect” and “search”, are discarded here. For this purpose, the condensed text for each document narrowed down by the character component table search is searched while scanning one character at a time in the same manner as the text data. At this time, only the condensed text corresponding to the document number obtained as a result of the character component table search is scanned. For example, the result of the character component table search is document number 1,
2, 3, 4,. . . Then, in the condensed text search, the document numbers 1, 2, 3, 4,. . . Scan the condensed text of. Then, a document in which the search term actually exists in the condensed text is output as a search result of the condensed text search. As described above, in the “hierarchical pre-search method”, two-stage pre-searches of “character component table” and “condensed text” are performed in advance, and sieved at “character level” and “word level” respectively. By narrowing down the number of documents targeted for time-consuming text search to a minimum in advance,
Since the search document capacity can be reduced, a very high speed full text search can be equivalently realized. That is, in the character component table search, since the character component table indicates the presence of a character by one-bit information, the data capacity to be searched can be extremely reduced, and as a result, the search time can be reduced to a short time. Becomes further,
By taking the logical product of the bit list for each character constituting the keyword, documents that are not related to the keyword can be largely cut off, and the subsequent target documents can be significantly narrowed down. Also, in the condensed text search, the search processing time can be shortened by the smaller amount of data than by directly scanning the text.

Next, a second embodiment of the present invention will be described.
In this embodiment, even when multiple search terms are specified,
An object of the present invention is to provide a full-text search method capable of efficiently performing a hierarchical pre-search. For example, when a conditional expression "" search "AND" understand "" is given, a character component table is searched as a first step. Here, a document containing all the characters is searched for each given search term, and then a document that satisfies the given conditions between the search terms is output. "" Search "AND
In the case of the conditional expression “understand”, a document that includes the two characters “search” and that includes the two characters “understand” is searched. That is, "('test' AND 'search') AND ('
'Solution') "
AND 'solution', that is, a document containing the above four characters at the same time is searched. Next, the condensed text of the document narrowed down as a result of the character component table search is searched. Here, only documents in which the specified keyword appears as a word are extracted. That is, a document that includes both “search” and “understanding” at the same time is searched. As in the case of this example, if the relationship between the search terms is only a logical condition such as "AND" or "OR" and no other condition for specifying the positional relationship between the keywords is specified, The search is terminated, and the result of the condensed text search is output as the final search result. If the position condition is specified, the text of the document extracted by the condensed text search is searched to extract a document that satisfies the specified condition, and this is output as the final search condition. The above is the description of the search operation in this embodiment. In this way, by taking the logical product between the search terms in the character component table search and the condensed text search, even if multiple search terms are specified, hierarchical pre-search can be performed efficiently, and high-speed full text search can be performed. A search can be realized.

As a third embodiment, the search control of the hierarchical presearch will now be described in more detail. FIG. 8 illustrates a procedure of controlling the hierarchical pre-search at this time with reference to a PAD diagram. Here, a description will be given of an example of a search formula of “search for a document containing either“ computer ”or“ intelligent interface ””, that is, ““ computer ”OR“ intelligent interface ””. First, step 80
At 00, a character component table search is performed. Here, a document containing all the characters is searched for each given search term, and then a document that satisfies the compound condition given between the search terms is output. In this example, as shown in FIG. 9, for each of the three characters constituting the “computer”, a bit AND operation is performed between the corresponding entry numbers in the character component table, and then the “intelligent interface” is similarly constructed. The bit AND operation between the corresponding entry numbers in the character component table is performed for each of the nine characters to be performed, and the bit AND operation result for the "computer" created previously and the OR operation of the bit string are performed. That is,
"('Total'AND'Calculation'AND'machine') OR
('Knowledge'AND'AND'I'AND'N'A
ND'TA'AND'F'AND 'E'AND' ー '
AND's') "is executed.
As a result, a document including all three characters constituting the “computer” or a document including all nine characters constituting the “intelligent interface” is extracted. If the number of the results of the character component table search described above is 0, a search result of 0 is output as shown in FIG. 8, and the search is terminated. Also, when the search term is composed of only one character such as “lake”, the result of the character component table search is output here and the search is terminated. If the search term is composed of a plurality of characters and the number of results of the character component table search is not 0, a condensed text search is performed next. The content registered in the condensed text is a character string divided for each character type. For example, if the character type is different in the middle, such as "intelligent interface", the condensed text is decomposed into partial character strings, and a separator is inserted at the division point, such as "intelligent interface". Therefore, in the case of a search term composed of different character types, such as "intelligent interface", there is no corresponding character string even if the condensed text is searched as it is. Therefore, a search term is checked before entering the condensed text search, and a search term composed of different character types is divided for each character type. A search term that has been subjected to the process of splitting by character type in this way is distinguished from the original search term, and is called a split search term. In the condensed text search, for example, a search is performed using a split search term such as "computer", "intelligent", or "interface". However, with respect to the divided search term, a search is performed between terms having the same division source under an AND condition. For example, in the case of a conditional expression ““ computer ”OR“ intelligent interface ””, in the condensed text search, “computer” OR
("Intelligent" AND "interface")
The search is performed as a conditional expression such as “Search for a document in which“ intelligent ”and“ interface ”exist in the same document or“ computer ”exists”. If the result of the condensed text search is 0, a search result of 0 is output and the search is terminated. Further, the text search is performed only when there is a designation of a neighborhood condition or a context condition, or when there is a search term to be divided such as “intelligent interface”, that is, when the search term and the divided search term are different. If not, the hierarchical pre-search is terminated here, and the result of the condensed text is output as a search result. here,
The context condition is, for example, a conditional expression such as ““ computer ”[S]“ intelligent interface ””, which means that ““ computer ”and“ intelligent interface ”are in the same sentence (sentence). Look for things. " Alternatively, the neighborhood condition is, for example, ““ computer ”[10C]
It is described as "intelligent interface".
This is because "computer" and "intelligent interface"
Find documents that appear close to each other within characters. " That is, both the context condition and the neighborhood condition are search conditions that specify the positional relationship of search terms that appear in the document. If a search condition that specifies the positional relationship of the search terms appearing in the text is given, or if a search term whose character type changes in the middle of the condensed text in the middle of the separator is given, the condensed text The text data corresponding to the search result is referred to, the search term that appears in the text according to the given condition is output as the search result, and the search is terminated. In this way, even when the search terms are composed of different character types, or when there is a condition specification regarding the appearance position in the text between search terms, efficient hierarchical pre-search is performed and high-speed full text A search can be realized.

Next, a fourth embodiment of the present invention will be described. In the present embodiment, the capacity of the character component table in the first embodiment is reduced to make it compact. The character code dependent character component table used in the first embodiment is easy to process, but has a problem that the character component table becomes large because the bit list per document of the character component table is long.
In addition, there is a problem that there are many useless portions because the entry number is assigned even though there is no corresponding character code. For example, in the case of shift JIS, between (0000) H and (8140) H and from (A000) H to (E04
0) During H, that is, from 0th to 33087th and 4
There is no corresponding character code for the entry numbers from 0960 to 57408. Nevertheless, in order to determine the entry number by the character code, it is necessary to have all of this part as a table entry. The character code is once converted to eliminate useless portions in the bit list, and a character component table is created so that bit positions can be used without gaps from the 0th position. The details of the embodiment using the character code conversion type character component table will be described below.

The following formula is given as an example of a conversion formula into a character code for creating a character code conversion type character component table. FIG. 10 shows a corresponding PAD diagram. if
SJIS <(A000) H then
SCODE = SJIS-(8040) H
else SCODE = SJ
IS− (C040) H SCODE = SC
ODE-(SCODE / 256) x 64
・ ・ ・ ・ ・ ・ ・ (4-1)
Expression (However, since a small value portion of the normal character code is often used as a control code, in this expression, (814
0) H is not set and (8040) H is given some margin. Further, the decimal part of the calculation result of (SCODE / 256) is rounded down, and the rounded-down result is multiplied by 64. In the expression, SJIS indicates the original shift JIS code, and SCODE indicates the character code after conversion. K
Shift JIS for EIS codes and other code systems
Since the correspondence with the code is taken, SCODE is
Conversion to is possible. The expression (4-1) means a conversion as shown in FIG. 11 when represented in a character code table. That is, (8) between (0000) H and (FFFF) H
140) H- (9FFC) H and (E040) H-
The character code is converted so that the character codes distributed and arranged with (FEFC) H are arranged without gaps from (0000) H. By performing code conversion using this equation (4-1), the length of the bit list can be extremely reduced as shown in FIG. 12, and the overall capacity of the character component table can be reduced.

The control of the hierarchical presearch is the same as in the first embodiment. That is, using the control procedure of FIG. 8 as it is, first, a character component table search is performed using the characters in the search term, and second, a condensed text search is performed using the search term. If no context condition is specified, the search result is output here and the search ends. Thirdly, if a context condition is specified, a text search is performed and the result is output. However, in the case of a character component table search, all the input search terms are (4-
The character code conversion is performed based on the expression (1). The fourth embodiment using the character code conversion type character component table has been described above. According to the present embodiment, by converting the character code and creating a character component table in which bit positions are arranged without gaps from the 0th position, entries in the character component table to which no character is allocated can be eliminated. The capacity of the character component table can be made very small.

Next, a fifth embodiment of the present invention will be described. In the present embodiment, the capacity of the character component table in the fourth embodiment is further reduced by using a hashing method. In order to further reduce the capacity of the character component table of the fourth embodiment, in this embodiment, a plurality of characters are assigned to one entry number in the bit list. That is, a method is used in which a character in a search term is associated with a bit position in a bit list using a hash function. For example, the following equation can be used as the hash function. h (SCO
DE) = mod (SCODE, N)
・ ・ ・ ・ ・ (5-1) In the formula, SCODE is (4-
This is a character code converted from shift JIS by the expression (1). mod is a function that outputs a remainder obtained by dividing the first argument by the second argument. N is an arbitrary integer value. For example, if 512 is used as N, “A” is the entry number 4
80 and 'ma' are entry numbers 193.

FIG. 13 shows an example of the character component table created in this way. In this case, N was set to 512,
It can be seen that only 512 bits are required to register one document. At the time of search, an entry number is obtained for each character of a given search term using the hash function of equation (5-1) in the same manner as at the time of registration, and the bit position of the corresponding character component table is referenced. For example, in the case of a character string “ambiguity”, as shown in FIG.
A document in which all bits at positions 80, 482, and 193 are 1 is set as a search result of the character component table search. Next, the condensed text of the document obtained by the character component table search is searched.

The control procedure of the condensed text search and the text search will be described below with reference to FIG. In the first embodiment, if the search term consists of one character after the character component table search, the result of the character component table search is output as the search result, and the hierarchical presearch is terminated. However,
In the character component table search of the character component table used in the present embodiment, since there is a possibility that search noise may occur, it is necessary to continue the hierarchical pre-search up to the condensed text search. For example, hiragana 'wa' (shift JIS code (82C
D) H) is the entry number 13 in the expression (5-1), but the kanji character “ship” (shift JIS code (8ACD)
H) has the same entry number 13. This means that when "ship" is given as a search term, all documents containing "ha" are also searched as a result of the character component table search. Therefore, the condensed text is further scanned to extract a document that actually contains the kanji “ship”, and this is output as a search result. The fifth embodiment has been described above. In the present embodiment, by assigning a plurality of characters to one entry of the character component table by using the hash function, the effect of significantly reducing the capacity of the character component table can be obtained.

Next, a sixth embodiment will be described. In the case of simple hashing as in the fifth embodiment, there is a possibility that a character that easily appears in a document, such as hiragana, and a character that rarely appears, such as JIS second-level kanji, have the same entry number. Come out. For example, “Hiragana” and “Kanji” have the same entry number 13, and if “Ship” is given as a search term, all documents containing “H” will be hit as a result of the character component table search. become. Since Hiragana 'ha' is a character that is very frequently used in Japanese documents, almost all documents are hit in the character component table search. As described above, when the narrowing rate in the character component table search is reduced, the amount of documents to be scanned for the condensed text also increases, so that the entire search processing time increases.

In order to prevent such a reduction in the narrowing rate, it is necessary to determine the hash function in consideration of the frequency of use of characters. The character component table used in this embodiment is called a character type hashing type character component table. In order to create a character type hashing type character component table, for example, as shown in FIG. 15, an entry area of the character component table is allocated to each character type, and a hash function is formed in the area by means of a character code. In order to realize such a hash function, a character type may be determined based on a character code and then returned using a mod function, or may be realized based on a correspondence table between character codes and entry numbers. An example of this hash function is shown in a PAD diagram in FIG. In this embodiment, hiragana,
The number of katakana and alphabetic entries is 20 each, the number of symbol entries is 10, the number of numeric entries is 10, JIS
The number of entries of the first level is 370, and the number of entries of the JIS second level is 61. First, the character type of the input search term is determined based on the character code, and the entry portion assigned to the character component table for each character type is looped back using the mod function.

That is, if SCODE is in the range of (01DF) H to (0231) H, it is a hiragana character string. Therefore, the SCODE is modulo 20 and this is used as the entry number. If SCODE is in the range of (0240) H to (0296) H, it is a katakana character string. Therefore, modulate the SCODE by 20 and add a value obtained by adding 20 which is the head of the katakana hashing area. Let it be an entry number. If SCODE is in the range of (01A0) H to (01DA) H, it is an alphabetic character string. Therefore, modulate the SCODE by 20 and add a value obtained by adding 40 which is the head of the hashing area of the alphabetic character to this. Let it be an entry number. SCODE changes from (018F) H to (0
198) If it is in the range of H, since it is a numeric character string, its SCODE is modulo 10 and the value obtained by adding 70 which is the head of the numeric hashing area is set as the entry number. SCODE changes from (065F) H to (123
2) If it is in the range of H, it is a JIS first-level kanji character string, so its SCODE is mod 370, and
A value obtained by adding 80, which is the head of the hashing area of the JIS first-level kanji character string, to this is set as the entry number. SC
If the ODE is in the range from (125F) H to (1FDE) H, it is a JIS second-level kanji character string.
MOD is taken as 61, and a value obtained by adding 450, which is the head of the hashing area of the JIS second-level kanji character string, to the entry is used as the entry number. In cases other than the above SCODE, it is regarded as a character string of a symbol or other character type, and the SCODE is modulated by 10, and a value obtained by adding 60, which is the head of the hashing area of the symbol, to the entry number.

The control procedure of the hierarchical presearch using the character type hashing type character component table is the same as that of the fifth embodiment. That is, first, a character component table search is performed using the characters in the search term, and second, a condensed text search is performed using the search term. If no context condition or the like is specified, the search is terminated here. If not, a third text search is performed and the result is output. As described above, according to the present embodiment, by using the character type hashing type character component table in which the entry number of the character component table is made to correspond to each character type in consideration of the use frequency, the character component table search can be performed. Since the noise can be reduced, the amount of scanning of the document in the condensed text is reduced, and a high-speed full text search can be performed accordingly.

Next, as a seventh embodiment, a hierarchical pre-search using a frequency information hashing type character component table which can further improve the narrowing rate in the character component table search and reduce the scan amount of the condensed text can be performed. The control method will be described. To create a frequency information hashing type character component table, the use frequency of characters in a document registered in a database is checked, and a hash function is determined based on the frequency information. For a character with a high frequency, another character is prevented from entering the same entry as much as possible, and a hash function is adjusted so that a character with a low frequency includes a plurality of characters in the same entry. By doing so, an averagely stable narrowing-down rate can be obtained by the character component table search. Specifically, as shown in FIG. 17, based on the SCODE obtained by the equation (4-1), the number of documents using the corresponding character in the database once is checked and sorted in the order of frequency. Next, from the most frequent one, N
Take only t. Then, leaving only the entry having the highest frequency in the frequency number distribution within Nt, the other entries are sequentially assigned entry numbers of Nt or more. In order to assign an entry number equal to or greater than Nt, an entry having a higher frequency than Nt is assigned so that the (Nt + 1) th entry is the Nt entry and the (Nt + 2) th entry is the (Nt-1) th entry. Go. In the assignment process, no other entry is assigned above the entry having the highest frequency. The allocated entries are in the form of a table as shown in FIG.
The hash function is configured by referring to the stored table. That is, it can be seen that the character “check” having the SCODE of (095F) H is the entry number 231.

The control procedure of the hierarchical presearch is the same as that of the fifth embodiment. That is, using the control procedure of FIG. 14 as it is, first, a character component table search is performed using the characters in the search term, and second, a condensed text search is performed using the search term. If no context condition or the like is specified, the search is terminated here. If not, a third text search is performed and the result is output. As described above, according to the present embodiment, a character component table is created based on the frequency distribution of characters actually used in the database, so that a high narrowing-down rate can always be obtained stably in the character component table search. Therefore, a short search processing time can be stably obtained without depending on the search term.

In the above, five embodiments have been described with respect to different embodiments of the character component table. An embodiment with a different condensed text will now be described. Although the process of creating the condensed text used in the first embodiment is simple, as shown in FIG. 4, even a character string such as "for" which is not originally used in the search remains in the condensed text. become. This leads to a reduction in the compression rate of the condensed text. That is, the amount of condensed text to be scanned at the time of retrieval increases, so that the retrieval processing time increases. The main factor that reduces the compression ratio of the condensed text is that a string of adjuncts such as "for" that has no meaning in itself is registered in the condensed text. .

Therefore, as an eighth embodiment, a hierarchical pre-search using a condensed text from which a series of attached words unnecessary for this search are removed will be described. This condensed text is divided into character types
This is called a deduplication / adjunct removal type condensed text. As shown in FIG. 19, the method of creating the condensed body text is to separate the character string from the text character string of the body body into partial character strings, remove duplicate words from the partial character strings, and then remove attached words. Character type division and deduplication are the same as those in the first embodiment. Adjunct removal is performed on de-duplicated hiragana character strings. The analysis for removing the attached word is performed based on the basic word dictionary and the connection rules between words as shown in FIG. The basic word dictionary contains
As shown in FIG. 21, a verb composed of only hiragana, a demonstrative pronoun, an adjective, an adjective verb, an adverb, a conjunction, a particle, an auxiliary verb, and a conjugation ending of these parts of speech are registered together with part of speech information. In the example of this figure, the verbs <a> and <
<Naru>, <Hitaru>, etc. are registered together with their endings. In the connection rules, each word registered in the basic word dictionary is registered with which other words can be connected. For example, FIG.
As shown in the table, <verb-adjunct form> has <noun-koto>
Are connected, and it is registered that <noun-> can be connected to <noun-to>. Using such a basic word dictionary and the connection rules, it is determined whether or not the partial character string of Hiragana is composed of an accessory word, and whether or not the character string is registered in the condensed text is determined. For example, the substring "for" is <particle-><noun-for><
Since it can be analyzed as a connected character string such as a particle->, it is determined and removed as a character string composed only of attached words. On the other hand, the character string “ambiguity” cannot be analyzed as an adjunct word, so that it is registered in the condensed text without excluding it.

As described above, the compression rate of the condensed text can be increased by analyzing the attached words to eliminate the hiragana character string and deleting unnecessary information that is not used for search. Basic word dictionaries and connection rules used for analysis are fundamentally different from conventional keyword dictionaries in which the number of registered words increases with the times, and they are universal and need not be updated once created There is an advantage.
In order to exclude only those that can be analyzed as adjuncts, even if a new word composed of hiragana that does not exist in the dictionary appears, it will always remain in the condensed text.

Next, the control of the hierarchical pre-search method using the condensed text of the character type division / duplication elimination / adjunct word removal type will be described. In the condensed text of the character type division / duplication elimination / adjunct removal type, there are cases where the hiragana character string is analyzed as an adjunct and not registered in the condensed text. For this reason, when an attempt is made to search for a specific hiragana character string, the search may be missed in the condensed text search. For example, the character string “vertigo”
It can be analyzed that the verb inflected form ending "me" is connected to the auxiliary verb "mai". A specific example is "no admission". However, even if "vertigo" is used as a noun, it is deleted from the condensed text as a result of the attached word removal processing. Therefore, in such a case, if the condensed text is searched for with “vertigo”, there is a possibility that the search may be omitted. Therefore, it is necessary to check whether the search term is an original word in the condensed text or a word that may have been removed in the process of creating the condensed text, and then search. To check whether the search term is a word to be registered in the condensed text, the algorithm for removing attached words used when creating the condensed text is applied as it is. In this example, when a search term “vertigo” is given, this can be determined to be a series of attached words.

The above search control procedure will be described with reference to FIG. First, a character component table search is performed. If the number of results is 0, 0 is output as a search result, and the search process ends. As described in the first embodiment, in the method not using the hash function, the search result of the character component table can be output as the final search result only when the search term is one character. That is, when using the character component table described in the first and fourth embodiments, it is checked whether or not the search term is one character, and if it is one character, the result of the character component table search is output as the search result. , End the process. When the character component table based on the hash function described in the fifth, sixth, and seventh embodiments is used, the following condensed text search is always performed without checking whether or not this search term is one character. Thereafter, similarly to the first embodiment, a divided search term is generated.

Next, an attached word analysis is performed for each of the divided search terms. If any of the split search terms is determined to be an adjunct, the split search term may have been deleted from the condensed text, so the condensed text search is not performed and the search is performed based on the result of the character component table search. Search the text directly. On the other hand, as a result of the attached word analysis, if it is determined that all of the divided search terms are not attached words, a condensed text search is performed as in the first embodiment. If no neighborhood condition or context condition is specified, or if the divided search term is the same as the original search term, the result of this condensed text search is output as the final search result, and the search is terminated. If the neighborhood condition or the context condition is specified, or if the divided search term and the original search term are different, then a text search is executed, and the result is used as the final search result output. As described above, according to the present embodiment, the condensed text is analyzed by analyzing the hiragana character string, and using the condensed text of the character type division / duplication elimination / adjunct word removal type in which unnecessary attached word sequences are removed from the condensed text. The compression ratio can be improved and the search processing time can be reduced.

Next, as a ninth embodiment, a description will be given of a hierarchical type presearch using a condensed text in which character type division / duplication elimination / Hiragana character string elimination is performed, excluding all Hiragana character strings. The condensed text described in the eighth embodiment certainly increases the compression ratio, but may erroneously analyze the attached words. For example, besides the example of the character string “vertigo” used in the eighth embodiment, there are rare cases where it is essentially impossible to correctly determine which of the adjuncts is just by adjunct analysis.
For example, the substring “Toshino” in the document “Operating in this application ...” is used to mean “to this to” or “to , It is difficult to determine whether or not leverage means a mechanical lever. When used in the latter sense,
When the search term “leverage” is specified, “leverage” is not determined to be an adjunct, so that the condensed text is searched. On the other hand, in the creation of the condensed text, "Shiteko" is analyzed as an adjunct word and is deleted from the condensed text. In order to correct the incompleteness of the attached word analysis, the ninth embodiment implements a hierarchical pre-search by a simple determination method of whether or not it is a hiragana character string. FIG. 24 shows a method of creating the condensed text. In this method, after character type division, Hiragana is removed to eliminate duplicate registration.

A control procedure of the hierarchical pre-search using the character type division / duplication elimination / Hiragana character string removal type condensed text will be described with reference to FIG. First, a character component table search is performed as in the eighth embodiment. Thereafter, a split search term is generated. Next, it is checked whether each of the divided search terms is a hiragana character string. If at least one of the divided search terms has a hiragana character string, the text is directly searched based on the result of the character component table search without performing the condensed text search. On the other hand, if there is no hiragana character string in the divided search term, a condensed text search is performed in the same way as in the first embodiment, and the neighborhood and context conditions are specified, or the divided search term is different from the original search term Has
Continue the search process until the text search. As described above, according to the present embodiment, an accurate full-text search with no search omission can be realized even for a hiragana character string by using a condensed text in which all the hiragana character strings are excluded.

Next, a tenth embodiment of the present invention will be described. In the ninth embodiment, when a search term of Hiragana is given, it is necessary to directly refer to the text. Therefore, more search time is required. Therefore, a tenth embodiment will be described as a method for performing a full-text search at a high speed even when a hiragana search term is given. In the present embodiment, in addition to the condensed text used in the ninth embodiment, a condensed text in which the removed hiragana character strings are registered in the ninth embodiment is separately created. As shown in FIG. 26, after character type division and elimination of duplicate registration, it is determined whether or not the remaining partial character string is a Hiragana character string, a character string other than the Hiragana character string is registered as a condensed text A, and the Hiragana character string is condensed text. Register as B. In this way, when a search term of only Hiragana is given, the condensed text B can be searched, so that the search time can be reduced. FIG. 27 shows the procedure of actual hierarchical presearch search control.
Shown in First, a character component table search is performed as in the eighth embodiment. If the search result is 0, the search ends here. Thereafter, a split search term is generated. Next, the divided search terms are classified into terms consisting of hiragana character strings and terms consisting of other character strings. Thereafter, if there is a divided search term consisting of a character string other than Hiragana, the condensed A is searched. Next, if there is a Hiragana split search term, the condensed B is searched. Thereafter, similarly to the first embodiment, when there is a designation of a neighborhood or a context condition, or when the divided search term is different from the original search term, the search processing is continued until the text search. In this way, by storing separately the condensed text of only Hiragana and the condensed text of other than Hiragana, the condensed text can be used effectively regardless of the search term of any character type, and high-speed full-text search is always performed. realizable.

Next, an eleventh embodiment will be described.
The present embodiment is based on a method of using an independent condensed text for each character type in order to increase the compression rate of the condensed text. The condensed text used in this embodiment is referred to as a character type division / duplication elimination / character type registration type condensed text. Figure 2 shows how to create a condensed text body with this character type division / duplication elimination / character type registration.
As shown in FIG. 8, after character type division and duplicate registration elimination, the character type of the remaining partial character string is determined, and the Hiragana condensed text H, Katakana condensed text I, Kanji condensed text J, English condensed text K, and numeric condensed text It is classified and registered as a text L, a symbol or other character type condensed text M. By doing so, for example, when a search is performed using a kanji search term, only the condensed text J of the kanji character type needs to be searched, so that the search time can be further reduced. A specific hierarchical pre-search control procedure will be described with reference to FIG. First,
A character component table search is performed as in the eighth embodiment. If the number of search results is 0, the search ends here. Thereafter, a split search term is generated. Next, the divided search terms are classified for each character type. Thereafter, a condensed text to be searched is selected according to the character type of the decomposed search term, such as condensed H when there is a hiragana split search term, condensed I when there is a katakana split search term, and so on. Thereafter, similarly to the first embodiment, when there is a designation of a neighborhood or a context condition, or when the divided search term is different from the original search term, the search processing is continued until the text search. In this way, by separating the condensed text file for each character type and reducing the volume of each condensed text, full-text search with single character type search terms such as kanji only, katakana only, or hiragana only can be performed at high speed. The effect that it can be performed is obtained.

Next, a twelfth embodiment will be described with reference to FIGS. This embodiment is described in Japanese Patent Application No.
The present invention is realized by using the document search device proposed in 193015. The main configuration of this device is a keyboard 3001, a search expression analysis program 3002, a bit search processor 3007a, a string search engine 3
006, Complex Condition Determination Microprocessor 3045
a, search result storage memory 3046, display 302
0, a semiconductor memory device 3010a, a RAM disk device 3010b, a collective magnetic disk 3010c, and a search execution control program 3008. A character component table is stored in the semiconductor memory device 3010a.
Reference numeral 010b denotes a condensed text, a collective magnetic disk device 301
0c stores the text. However, the character component table and the condensed text are stored on the collective magnetic disk 3010c, and the semiconductor memory device 3010a and the RAM disk device 301, respectively, at the start of operation of this device.
0b.

The procedure of the hierarchical presearch control is the same as that described in the embodiment. The difference from the previous embodiments is that the character component table is stored in a semiconductor memory, the condensed text is stored on a RAM disk, and the text is stored on a collective magnetic disk. That is, a string search engine dedicated to search is used. The procedure of the search process will be described below.

The search condition expression input from the keyboard 3001 is a search machine control microprocessor MPU03.
050 is analyzed by the search formula analysis program 3002. In other words, the search formula analysis program 3002 separates the keyword portions constituting the search condition formula into the decoding condition description portion which describes the inclusion condition and the placement condition. The inclusion condition is described as a logical condition, and the arrangement condition is described as a neighborhood condition or a context condition. After separation and extraction,
The keyword part is also passed to the synonym expansion program 3003 on the MPU03050, and the decoding condition description part is passed to the decoding condition analysis program 3041. In the synonym expansion program 3003, the synonym of the input keyword is obtained with reference to the built-in synonym dictionary. Then, the keyword group that has undergone synonym expansion is passed to the variant notation expansion program 3004. In the case of the example in this figure, “computer”, “computer”,
“COMPUTER” or the like is generated. In the different notation development program 3004, a different notation development process is performed on the keyword group input here. In the case of the example of this figure, “computer” is changed from “computer” to “C”
"Computer" or the like is generated from "OMPUTER". The keyword group thus expanded into the synonym and the different notation is then processed by the microprocessor M for automaton generation.
Automaton generation program 3 on PU13005a
005. Automaton generation program 30
At step 05, an automaton for collectively collating the keywords sent from the different notation development program 3004 is generated, and the search engine 300 is used as the identification code information of the keyword to be collated with the state transition table.
Set to 6. The search engine 3006 is a high-speed multiple character matching circuit based on the finite automaton method. Also,
The keyword group expanded in different notations by the different notation expansion program 3004 is passed to the bit search program 3007 on the bit search microprocessor MPU33007a together with the corresponding keywords.

On the other hand, neighborhood conditions, context conditions, AND, O
Logical conditions such as R are obtained from the search expression analysis program 3002,
The complex condition analysis program 3041, the neighborhood condition analysis program 3042, the context condition analysis program 3043, and the logical condition analysis program 3044 are sent to the complex condition determination program 3045. Necessary search information is bit search program 3007, string search engine 3
006, after being sent to the complex condition determination program 3045, the search control execution program 3008 first activates the bit search program 3007. The bit search program 3007 reads a character component table stored in the semiconductor memory device 3010a and performs a character component table search. The result of the character component table search is stored in the search result storage memory 3046.

After the character component table search is completed, the search execution control program 3008 executes the search result storage memory 3046
If the search result is 0, 0 is output as the search result and the search process is interrupted. If the search result is not 0, the string search engine 3006 is activated, and at the same time, the condensed text of the document hit by the result of the character component table search stored in the search result storage memory 3046 is read from the RAM disk device 2910b. The text is sent to the string search engine 3006, and the condensed text search is executed. As a result, the condition determination as to whether or not the number of cases is 0 is performed by the search execution control program 3008. In the string search engine 3006, the RAM disk device 3
The condensed text read from 010b is searched by the divided search term. The comparison result is a compound condition determination program 30
45. Complex condition determination program 3045
Then, the logical condition given between the search terms is determined, and the document number of the document that satisfies the condition is determined.
The data is sequentially stored in 46.

After the condensed text search is completed, the search execution control program 3008 refers to the search result storage memory 3046 again. If the number of results is 0, 0 is output as the search result and the search is terminated. . When the search result document number is read from the search result storage memory only when the search condition is not zero and the neighborhood, the context condition is set, or the divided search term is different from the search term,
The corresponding text is stored in the collective magnetic disk drive 3010.
c, and sends it to the string search engine 3006, this time executing a text search. If the neighborhood and context conditions are not set and the divided search terms are equal to the search terms, the number of search results stored in the search result storage memory is output, and the search is terminated.

In the string search engine 3006,
The text read from the collective magnetic disk device 3010c is scanned to perform text search. The results are sequentially sent to the complex condition determination program 3045. The compound condition determination program 3045 determines not only the logical conditions given between the search terms, but also the neighborhood and the context conditions, and sequentially stores the document numbers of the documents meeting the conditions in the search result storage memory 3046. When the text search is executed, after the text search is completed, the search execution control program 3008 refers to the search result storage memory 3046, outputs the number of search results, and ends the search. In this way, large-volume text data is stored on the magnetic disk,
By storing in semiconductor memory or RAM disk,
High-speed full-text search can be realized even for a large-scale database.

Next, a description will be given of a thirteenth embodiment in which the condensed text is stored on a magnetic disk. When the condensed text is stored on the magnetic disk, processing can be performed at a higher speed than by performing a normal hierarchical presearch using the same configuration by optimizing the control procedure of the hierarchical presearch. Hereinafter, the procedure of this control will be described. A magnetic disk usually has a magnetic head that moves mechanically.
For this reason, there is a feature that it is faster to collectively read out information (referred to as sequential access) than to read out information on the disk step by step (referred to as skip access). Now, assuming that the read speed of skip access is Vskip MB / s and the read speed of sequential access is Vseq MB / s, the number of documents in all databases is Na, the number of results in the character component table search is Nc, and Assuming that the capacity is uniform, Nc> (Vskip / Vseq) · Na
... (12-1), the processing time is shorter when searching for all the condensed texts by sequential access than when performing skip access based on the result of the character component table search. Therefore, as shown in FIG. 32, after the character component table search, the number of results is determined in the hierarchical pre-search control program, and when the number of hits satisfying the expression (12-1) is reached, the result of the character component table search is obtained. Is ignored, and the condensed text is searched for all databases. With the above method, it is not necessary to use a large-capacity RAM disk to store the condensed text on the magnetic disk.
A relatively high-speed full-text search can be realized by a low-cost document search device.

Next, a description will be given of a fourteenth embodiment in which the condensed text is stored on a magnetic disk. When the neighborhood and context conditions are specified, if the result of the character component table search is very small, it is better to search the text directly based on the result of the character component table search without performing the condensed text search. Be shorter. Now, the search speed of the condensed text is Vsr MB /
s, the search speed of the text is VtxMB / s, the number of results in the character component table is Nc, and the number of results in the condensed text is Ns
r, where Qsr is the data capacity per case of the condensed text and Qtx is the data capacity per case of the text, NcQsr / Vsr + NsrQtx / Vtx> NcQtx / Vtx When formula (13-1) holds, Performing a text search directly without a text search shortens the search time. Nsr is not known until the condensed text is actually searched, but a constant is set in advance to determine whether to perform the condensed text search. For example, assuming that the number of documents in the entire database is Na, Nsr = αNa (0 <α <1)... (13-2) By transforming expression (13-1), Nc <αNa (Qtx / Vtx) / (Qtx / Vtx-Qsr / Vsr) In the case of Expression (13-3), the text search is directly performed. A value is set in advance before the search using α as a threshold value, and after the character component table search, it is determined whether or not to perform a condensed text search according to the equation (13-3). By performing this control, a high-speed full-text search can be realized under the specification of the neighborhood and context conditions. In the foregoing, the thirteenth and fourteenth embodiments for realizing a full-text search with a low-cost system configuration of the twelfth embodiment have been described.

In addition, the hierarchical pre-search can be realized by a control method of directly executing the text search from the character component table search without using the condensed text at all and omitting the step of the condensed text search. According to this method, it takes a little time to search because the amount of text to be scanned increases, but it does not require the use of an expensive RAM disk, and does not require the capacity of a magnetic disk for storing condensed text. Document retrieval device can be realized. Also, without using the character component table, the entire text search on the RAM disk or magnetic disk is directly searched, and the text search is performed only when there is a search condition specification of the positional relationship between search terms such as neighborhood and context conditions. Hierarchical pre-search can also be realized by a control method. According to this method, the search time is slightly increased because the amount of search for the condensed text increases, but a semiconductor memory for storing the character component table is not required, so that a low-priced document search device can be realized.

Alternatively, as shown in FIG. 33, the character component table in the bit list format used in the embodiments up to this point is a format in which characters appearing in a document are listed, that is, one character is represented as one bit. Instead, a character component table stored as it is as the character code itself can be used.
Alternatively, at this time, the capacity of the character component table is reduced by associating a plurality of characters with one character entry by using the hash function described in the fifth, sixth, and seventh embodiments. You can also. A character component table search using a character component table that stores character codes in this way is realized by reading data from the file character by character and determining whether or not the corresponding character exists, as in the case of a condensed text or text search. it can. In this way, by using a character component table that collects only the characters used in the text, the data structure can be simplified, and the condensed text,
Since the same scan type search as the text search can be used, the effect that the search processing method can be simplified can be obtained.

Furthermore, even in a configuration in which the character component table is also stored on the magnetic disk, hierarchical presearch can be realized. When the character component table is stored in the magnetic disk, the bit operation process is performed while sequentially reading the bit list of the characters used in the search term from the magnetic disk in the character component table search. Alternatively, if the above character code is used directly as a character component table,
While sequentially reading the character component table, a document including all the corresponding characters is selected. According to the method of storing the character component table on the magnetic disk, it is not necessary to use a semiconductor memory, so that a lower-priced document search device can be realized.

[0051]

According to the present invention, a useless text search can be performed by using a text component table and a condensed text to hierarchically filter out documents that are not related to a search term input at a text level and a word level. Since it can be omitted, it becomes a technology for realizing high-speed document search full-text search, and it becomes possible to perform document search with a practical response speed even in a large-scale document database.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a first embodiment of the present invention.

FIG. 2 is a diagram showing a registration process for hierarchical presearch, which is a feature of the present invention.

FIG. 3 is a diagram illustrating a search process of a hierarchical presearch, which is a feature of the present invention.

FIG. 4 is a diagram showing an example of creating a condensed text.

FIG. 5 is a diagram showing a storage form of a condensed text.

FIG. 6 is a diagram showing an outline of a character component table.

FIG. 7 is a diagram showing an outline of a character component table search.

FIG. 8 is a diagram illustrating a processing procedure of a hierarchical presearch.

FIG. 9 is a diagram showing a character component table search process according to the third embodiment.

FIG. 10 is a PAD diagram showing a code conversion process of a character component table used in the fourth embodiment.

FIG. 11 is a diagram illustrating an outline of code conversion of a character component table used in the fourth embodiment.

FIG. 12 is a diagram showing an outline of a character component table used in a fourth embodiment.

FIG. 13 is a diagram showing an outline of a character component table used in the fifth embodiment.

FIG. 14 is a diagram showing a processing procedure of a hierarchical presearch used in the fifth embodiment.

FIG. 15 is a diagram showing an outline of a character component table used in the sixth embodiment.

FIG. 16 is a diagram showing a processing procedure of a hierarchical presearch used in the sixth embodiment.

FIG. 17 is a diagram showing an outline of a method of creating a character component table used in the seventh embodiment.

FIG. 18 is a diagram showing an outline of a character code-entry number correspondence table used in a hash function for a character component table used in the seventh embodiment.

FIG. 19 is a diagram showing a method of creating a condensed text used in the eighth embodiment.

FIG. 20 is a diagram illustrating a method of processing a hiragana character string for a condensed text used in the eighth embodiment.

FIG. 21 is a diagram showing a basic word dictionary for attached word analysis used in the eighth embodiment.

FIG. 22 is a diagram showing connection rules for attached word analysis used in the eighth embodiment.

FIG. 23 is a diagram showing a processing procedure of a hierarchical presearch used in the eighth embodiment.

FIG. 24 is a diagram showing a method of creating a condensed text used in the ninth embodiment.

FIG. 25 is a diagram showing a processing procedure of hierarchical presearch used in the ninth embodiment.

FIG. 26 is a diagram illustrating a method of creating a condensed text used in the tenth embodiment.

FIG. 27 is a diagram illustrating a processing procedure of hierarchical presearch used in the tenth embodiment.

FIG. 28 is a diagram illustrating a method of creating a condensed text used in the eleventh embodiment.

FIG. 29 is a diagram illustrating a processing procedure of hierarchical presearch used in the eleventh embodiment.

FIG. 30 is a diagram showing a part of the configuration of a twelfth embodiment.

FIG. 31 is a diagram showing the remaining part of the configuration of the twelfth embodiment.

FIG. 32 is a diagram illustrating a processing procedure of hierarchical presearch used in the twelfth embodiment.

FIG. 33 is a diagram showing an outline of a character component table stored as characters.

[Explanation of symbols]

 103: Text 104: Condensed text 105: Character component table 201: Text registration program 202: Condensed text creation registration program 203: Character component table creation registration program 204: Character component table creation search program 205: Condensed text creation creation program 206: Text Search program

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kanji Kato 1-280 Higashi Koikebo, Kokubunji-shi, Tokyo Inside the Central Research Laboratory, Hitachi, Ltd. (72) Inventor Hisamitsu Kawaguchi 1-280 Higashi Koikebo, Kokubunji-shi, Tokyo Hitachi, Ltd. Central Research Laboratory (72) Inventor, Naoki Minegishi 890 Kashimada, Saiwai-ku, Kawasaki-shi, Kanagawa Prefecture F-term (reference) 5B075 ND03 NR02 NR14 NS10 PP02 PQ02 QM01

Claims (4)

[Claims] 1. A document retrieval apparatus for retrieving a document including a keyword specified by a searcher from a document database storing document information as character code data.
When registering a document in the document database, the body character string of the registered document is divided into character types such as hiragana, kanji, and alphanumeric characters, and the character string inclusion relationship between the divided partial character strings is determined. Means for creating a condensed text composed of a set of partial character strings excluding character strings included in other character strings, and a means for generating a character component table in which characters appearing in the condensed text are registered without duplication, A means for storing the condensed text and the character component table together with the text of the document to be registered in the document database, and a document containing all the types of characters that make up the keyword specified by the searcher at the time of search A character component table search means for referring to and extracting, a means for inspecting the number of documents extracted in the character component table search, and a condensed text when the number of documents as a result of the inspection reaches a predetermined number or more. Read all Then, a document corresponding to the condensed text including a partial character string constituting the keyword specified by the searcher is extracted, and if the number of results of the inspection is equal to or less than a predetermined number, the document extracted by the character component table search A condensed text search means for extracting a document corresponding to the condensed text including the partial character string constituting the keyword specified by the searcher by referring to the condensed text of the searcher; and referring to the text of the extracted document by referring to the text of the extracted document. And a text search means for extracting a search condition that satisfies a search condition such as a positional relationship given to the document search device. 2. A document search apparatus for searching for a document including a keyword specified by a searcher in a document database storing document information as character code data.
When registering a document in the document database, the body character string of the registered document is divided into character types such as hiragana, kanji, and alphanumeric characters, and the character string inclusion relationship between the divided partial character strings is determined. Means for creating a condensed text composed of a set of partial character strings excluding character strings included in other character strings, and a means for generating a character component table in which characters appearing in the condensed text are registered without duplication, A means for storing the condensed text and the character component table together with the text of the document to be registered in the document database, and a document containing all the types of characters that make up the keyword specified by the searcher at the time of search A character component table search means for referring to and extracting, a means for inspecting the number of documents extracted by the character component table search, and a condensed text when the number of documents as a result of the inspection reaches a certain number or more. Read all A condensed text search means for extracting a document corresponding to the condensed text including the partial character string constituting the keyword specified by the searcher; and referring to the text of the document extracted by the condensed text search means, A document that satisfies the search conditions such as the positional relationship given between them is extracted, and when the number of results of the character component table search is equal to or less than a certain number, referring to the text of the document extracted by the character component table search And a text search means for extracting a document that includes a specified keyword and satisfies a search condition such as a positional relationship given between the keywords. 3. A document search method for searching for a document including a keyword specified by a searcher in a document database storing document information as character code data.
When registering a document in the document database, the body character string of the registered document is divided into character types such as hiragana, kanji, and alphanumeric characters, and the character string inclusion relationship between the divided partial character strings is determined. Examining and creating a condensed text composed of a set of partial character strings excluding character strings included in other character strings, and creating a character component table in which characters appearing in the condensed text are registered without duplication, A step of storing the condensed text and the character component table together with the text of the registration target document in the document database, and a document including all types of characters constituting the keyword specified by the searcher at the time of search.
A character component table search step of extracting with reference to the character component table, a step of inspecting the number of documents extracted by the character component table search, and, if the number of results of the inspection reaches a predetermined number or more, And reading out all the condensed texts and extracting documents corresponding to the condensed texts including the partial character strings constituting the keyword specified by the searcher. If the number of results of the inspection is less than a predetermined number, the character component table A condensed text search step of extracting a document corresponding to the condensed text including the partial character string constituting the keyword specified by the searcher with reference to the condensed text of the document extracted by the search; A text search step of extracting a document that satisfies a search condition such as a positional relationship given between keywords with reference to a keyword. 4. A document search method for searching for a document including a keyword specified by a searcher in a document database storing document information as character code data.
When registering a document in the document database, the body character string of the registered document is divided into character types such as hiragana, kanji, and alphanumeric characters, and the character string inclusion relationship between the divided partial character strings is determined. Examining and creating a condensed text composed of a set of partial character strings excluding character strings included in other character strings, and creating a character component table in which characters appearing in the condensed text are registered without duplication, A step of storing the condensed text and the character component table together with the text of the registration target document in the document database, and a document including all types of characters constituting the keyword specified by the searcher at the time of search.
A character component table search step for extracting with reference to the character component table, a step of inspecting the number of documents extracted in the character component table search, and, if the number of results of the inspection reaches a certain number or more, A condensed text search step of reading out all the condensed text and extracting a document corresponding to the condensed text including the partial character string constituting the keyword specified by the searcher; and a text of the document extracted by the condensed text search step , A document that satisfies search conditions such as a positional relationship given between keywords is extracted, and if the number of results of the character component table search is less than or equal to a certain number, the document extracted by the character component table search A text search step of extracting a document that includes a specified keyword and satisfies a search condition such as a positional relationship given between keywords by referring to a text of the keyword. Document search method according to claim.