JP3578501B2 - Document search method and apparatus - Google Patents

Description

[0001]
[Industrial applications]
The present invention relates to a document search method and apparatus using an index, and is applied to a database, a document filing system, a DTP (Desk Top Publishing) system, and the like.
[0002]
[Prior art]
In an information processing system, it is one important thing to find all documents including a specific word that a searcher wants to search for, that is, a query word, from a document composed of a collection of character string data stored in a database. Processing.
As a method for searching for such a document, an index search method using an index is well known. This method is described in "Information Search" (Nakahara, IEICE Publishing, 1974) 203-207 (hereinafter referred to as Known Example 1) and "DOCUMENT DATABASE" (G. James, Van Nostrand Reinhold Co., 1985) pp. 87-94.
The index mentioned here is composed of the document number of the document in which the keyword appears. In these index search methods, only by referring to the index of the keyword that matches the query word, the document containing the keyword can be found, so that high-speed search is possible.
[0003]
FIG. 2 shows an example of the index described in the above-mentioned known example 1. The index is configured to store the keyword number and the document number of the document in which the keyword appears, corresponding to the keyword extracted from the document.
In this example, it is assumed that indexes corresponding to the keywords “core”, “disk”, “computer”, and “IR” have been created and stored in files on the magnetic disk.
At the time of retrieval, the index stores the document number of the document in which each keyword appears in the index only when "core", "disk", "computer", and "IR" are designated as the query words. Is read from the file (hereinafter referred to as an index file).
That is, if the query word is "core", document numbers 1 and 4, if the query word is "disk", document number 4; if the query word is "computer", document numbers 1, 2, 4, and 4. When the query word is "IR", a document number is output as a search result like document number 2.
When registering a new document in the database, a keyword that appears in the document is extracted, and the document number of the document in which this keyword appears is additionally registered in the index corresponding to the keyword.
Techniques for extracting keywords from documents as described above include “Trends in Automatic Indexing Research” (Morohashi, IPSJ Journal, Vol. 25, No. 9, 1984) and “DOCUMENT DATABASE” (G. James, Van). Nostrand Reinhold Co., 1985) pp. 87-94.
FIG. 3 shows an example of processing for adding an index corresponding to a keyword extracted using these keyword extraction techniques.
In this example, it is assumed that the document number of the document to be registered is 5, and that “core”, “computer”, and “IR” have been extracted from this document. In this index, a document number 5 is added to each of the indexes corresponding to “core”, “computer”, and “IR”. In this manner, the document registration process is realized by performing the process of adding the index corresponding to the extracted keyword.
[0004]
[Problems to be solved by the invention]
"Information Search" (Nakahara, IEICE Press, 1974) pp. According to H.120-128, a secondary storage device such as a large-capacity and inexpensive magnetic disk, which can perform random access, is generally used as a storage device for storing an index.
In the process of adding an index stored on a magnetic disk, an index corresponding to a keyword to be added is independently accessed. That is, in the index adding process, a plurality of indexes on the magnetic disk are accessed at random (hereinafter, referred to as random access).
In a general operating system of a workstation, access to a magnetic disk is performed in units called logical blocks. Here, it is assumed that the size of the logical block is 8,192 bytes (hereinafter, abbreviated as 8 KB) used in the operating system.
However, writing to the magnetic disk may not be performed in units of 8 KB. For example, assuming that the size of the document number to be written in the index adding process is 4 bytes per one document number, when one document is registered, data smaller than a logical block and less than 8 KB is written. In such a case, the operating system temporarily reads the logical block to be written from the magnetic disk into a buffer area on the main memory of the operating system. Next, the logical block is updated by writing the data to be written to a predetermined location of the logical block in the buffer area. Thereafter, the logical block is written to the magnetic disk again. In this way, writing of less data than the logical blocks to the magnetic disk is realized.
[0005]
An example of the operation of the magnetic disk at this time will be described with reference to FIG.
First, seek processing and rotation waiting processing of the head for positioning the head of the magnetic disk at the head position of the logical block to be read are performed, and then read processing of the logical block to be read is performed.
Thereafter, the logical block is updated (document number adding process), and a seek process and a rotation waiting process of the head for positioning the head of the magnetic disk again at the position of the logical block to be read are performed. Thereafter, a process of writing the logical block to the magnetic disk is performed.
[0006]
The values of the processing times shown in the figure are those of a general 3.5-inch magnetic disk. The average seek time (hereinafter, referred to as the average seek time) is about 14 ms, and the average rotation waiting time ( Hereinafter, it is assumed that the average rotation waiting time is about 17 ms, and the read time and the write time per logical block are about 4 ms. Also, 0.001 ms is assumed as the time for writing a 4-byte document number to a logical block in the buffer area.
In this example, it takes a total of 70 ms to write data less than the logical block to the magnetic disk. According to the time chart of FIG. 11, the time spent for the read process and the write process from the magnetic disk is 8 ms, whereas the time spent for the seek process for positioning the head and the rotation wait process is 62 ms, which is eight times. It turns out that it has been hanging for a long time.
That is, while the processing speed of reading and writing to the magnetic disk is 2 MB / s, the overall effective processing speed including the time of the seek processing and the rotation waiting processing is about 0.13 MB / s, and the reading and writing of the magnetic disk is performed. In addition, writing performance cannot be obtained.
[0007]
For this reason, in index addition processing performed at the time of document registration, random access to the magnetic disk occurs as many times as the number of keywords for which the addition processing has occurred, thereby reducing the effective processing speed of the magnetic disk. It will take time. That is, in the document search method using the index, there is a problem that it takes time to register a document.
An object of the present invention is to speed up the process of adding an index performed at the time of registration and reduce the registration time.
[0008]
[Means for Solving the Problems]
In order to solve the above problems, the present invention provides:
A document search method for extracting a keyword from a document, creating an index based on the keyword, and performing a search by referring to an index corresponding to a keyword that matches the query word at the time of the search, wherein the index is stored in a file on a secondary storage device And the file is divided into a predetermined number of blocks, keywords are assigned to the blocks so that the size of the index stored in each block is substantially equal, and added to the blocks when a document is registered. Alternatively, when the number of indexes in which the update has occurred is equal to or more than a predetermined number, the block is read from the secondary storage device to the main storage, and an index adding process is performed for the keyword corresponding to the index in the block. The block that has been added and processed above is stored in a secondary storage device, and If less than, the index is read from the secondary storage device to the main storage, and for the keyword corresponding to the index, index addition processing is performed on the main storage, and the added index is stored in the secondary storage. It is stored in the device.
In addition, when assigning a block to a keyword, for the keyword assigned to the block, the number of appearing documents of the keyword among all the documents to be indexed is calculated, and the sum of the number of appearing documents calculated for each keyword is calculated as a block. It is calculated for each block, and the block assignment of the keyword is performed so that the total sum of the number of appearing documents in each block is substantially equal in each block.
Further, in a document search device that extracts a keyword from a document, creates an index based on the keyword, and performs a search by referring to an index corresponding to a keyword that matches the query word at the time of search, the index is stored in a secondary storage device. Means for dividing the file into a predetermined number of blocks, allocating means for allocating keywords to the blocks such that the size of the index stored in each block is substantially equal, and Means for determining whether or not the number of indices added or updated in the block is equal to or greater than a predetermined number, and when the determination result is equal to or greater than the predetermined number, reading the block from the secondary storage device into the main memory, Index addition processing is performed on the main memory for the keyword corresponding to the index of, Means for storing the processed block in the secondary storage device, and reading the index from the secondary storage device to the main storage when the number is less than a predetermined number, and adding the index to the keyword corresponding to the index. There is provided a means for storing the index, which has been performed on the storage and additionally processed, in the secondary storage device.
The assigning means calculates the number of appearing documents of the keyword among all the documents to be indexed for the keyword assigned to the block, and calculates the sum of the number of appearing documents calculated for each keyword for each block. , And block assignment of keywords is performed so that the total sum of the number of appearing documents in each block is substantially equal in each block.
[0009]
[Action]
By the above means, keywords can be extracted from all the documents, and the keywords can be assigned to the blocks so that the index size of each block of a predetermined number of blocks on the secondary storage device becomes substantially equal. If the number of relevant extracted keywords is equal to or more than a predetermined number for each block that stores the index, the block is read from the magnetic disk to the memory and the index is added to the relevant extracted keyword. By writing this to the magnetic disk, the number of accesses to the magnetic disk is reduced. If the number is less than the predetermined number, only the index corresponding to the relevant extracted keyword is read into the memory and the index is added, and this is added. By writing to a magnetic disk, It is possible to minimize the access data amount to click, very allows fast indexing addition processing, it is possible to realize a high-speed process of registration in the document database.
[0010]
【Example】
First, the principle of the present invention will be described below.
As an initial setting, a predetermined size is secured on the magnetic disk as an area of an index file for storing an index, and the area is divided into a predetermined number of blocks (not logical blocks). Next, the capacity of the index stored in each block (index size) Sum of Is assigned to each block so that is approximately equal.
When a document is registered, first, a keyword is extracted from the registered document, and the keyword and the document number of the document in which the keyword appears are stored in the main memory.
Next, for each block, it is checked how many keywords assigned to the block have been extracted.
If the number is less than the predetermined number, the index corresponding to the keyword extracted for each block is read from the magnetic disk to the main storage as in the conventional case. Next, the document number of the document in which the keyword appears is added to the end of the index read into the main memory, and the document is stored again on the magnetic disk.
If the number is equal to or more than the predetermined number, first, the corresponding block is read from the magnetic disk to the main storage. Next, only for the extracted keywords assigned to the block, the document of the document in which the keyword appears at the end of the index corresponding to the keyword in the block read into the main memory. Add a number. Thereafter, this block is stored again on the magnetic disk.
This series of processes is performed on all blocks to which the extracted keywords are assigned, thereby performing a process of adding the keywords extracted from the registered document to the index.
[0011]
As described above, for a block including a predetermined number or more of the extracted keywords, the block on the magnetic disk is read into the main storage only once, an additional process for a new keyword is performed, and then this is written to the magnetic disk. Since it is possible to add multiple indexes simply by reading the index from the magnetic disk for each keyword, performing the additional processing for that keyword, and then writing this to the magnetic disk, as in the past, The time required for the index addition process can be reduced.
[0012]
The principle described above will be further described using a specific example.
FIG. 5 shows an example of an index file used in this example.
This index file is obtained by dividing the index file shown in FIG. 2 into two blocks, block 1 and block 2, and storing them on a magnetic disk. In addition, the size of the index contained in the block Sum of The keywords "core", "disk" and "IR" are allocated to block 1 and the keyword "computer" is allocated to block 2 so that the index is stored in the corresponding block, so that the index is stored in the corresponding block.
[0013]
This index is subjected to a search process in the same manner as in the related art without being aware of blocks when searching. Only when “core”, “disk”, “computer”, and “IR” are specified as query words, the document number of the document in which each keyword appears is read from the index file.
That is, if the query word is "core", document numbers 1 and 4, if the query word is "disk", document number 4; if the query word is "computer", document numbers 1, 2, 4, and 4. When the query word is "IR", a document number is output as a search result like document number 2.
[0014]
When registering a new document in the database, processing is performed in consideration of the block as shown in FIG. Hereinafter, the procedure will be described in detail.
First, it is assumed that “core”, “disk”, and “IR” have been extracted as keywords from a document to be registered, and a document number 5 is assumed as the document number of this document.
As for the assignment of the extracted keyword blocks, the keywords “core”, “disk” and “IR” are assigned to block 1 and “computer” is assigned to block 2. Among these keywords, the keywords extracted from the document 5 correspond to the keywords “block”, “core”, “disk” and “IR”.
Therefore, first, block 1 is read into the main memory. Next, 5 which is the document number of the document in which these keywords appear is added to the end of the index corresponding to the keywords "core", "disk" and "IR" stored in block 1, respectively. Thereafter, block 1 stored in the main memory is written to an index file on the magnetic disk.
As for block 2, since no keyword corresponding to the keyword assigned in block 2 has been extracted from document 5, no index addition processing is performed. Thus, the document registration process is performed.
[0015]
FIG. 7 shows a time chart of the index addition processing of the block 1.
In this example, a general 3.5-inch magnetic disk is used as a magnetic disk, 8 KB used in a general operating system of a workstation is used as a logical block size, and the size of a block for storing an index is used. Assume that 8 logical blocks are used as the size.
As the seek time and the rotation waiting time of the magnetic disk, an average seek time and an average rotation waiting time are assumed. Further, the average seek time, the average rotation waiting time, the read time of one logical block, and the write time of one logical block in the magnetic disk are assumed to be about 14 ms, about 17 ms, about 4 ms, and about 4 ms, respectively. Using the above values, the flow of the time chart of this figure will be described.
In this example, first, when reading the block 1 to the buffer area, access to the magnetic disk occurs, and the head of the magnetic disk is positioned at the head of the block 1 by the seek processing and the rotation waiting processing. During this time, an average seek time of 14 ms and an average rotation waiting time of 17 ms are spent.
Next, the logical blocks constituting block 1, in this case eight logical blocks, are read into the buffer area. At this time, 32 ms, which is eight times the read time for one logical block, is consumed.
Here, the document number 5 of the document in which these keywords appear at the end of the index corresponding to the keywords “core”, “disk” and “IR” stored in the block 1 read into the buffer area is shown. Each is added. Therefore, the document number is added to block 1 three times. At this time, 0.001 ms is assumed as a time for writing a 4-byte document number to one index on the buffer area. Here, the addition of the document number three times requires three times as much as 0.003 ms, but is negligibly small compared to other processing times.
After that, the block 1 on which the index addition processing has been performed is written to the magnetic disk. At this time, the head of the magnetic disk is positioned at a predetermined position by the seek processing and the rotation waiting processing. During that time, an average seek time of 14 ms and an average rotation waiting time of 17 ms are spent. After this, the eight logical blocks that make up block 1 are written to the magnetic disk, consuming 32 ms, eight times the writing time for one block.
With the above processing, a total of 126 ms is spent for adding the index in this example. This is an average of 42 ms per keyword.
As in the prior art, when adding processing to the index on the magnetic disk for each keyword is performed, as shown in FIG. 4 described above, it takes 70 ms per keyword, but the index is collectively indexed into blocks as in the present invention. By performing the additional processing, the registration time of the document can be reduced by about 40%.
When a general document is registered, the number of registered keywords is several tens of times that of the present example, so that the effect of the process of adding an index in block units according to the present invention is further enhanced.
[0016]
As described above, according to the present invention, by performing an index addition process on a block including a predetermined number or more of additional registration keywords at a time, a block on a magnetic disk can be read only once into main memory. Since a process of adding a plurality of indexes can be executed, a high-speed registration process to the document database can be realized.
[0017]
Hereinafter, examples of the present invention will be described.
The configuration of a document search system to which the present invention is applied will be described with reference to FIG.
This system includes a display 101, a keyboard 102, a CPU 103, a memory 104, a magnetic disk 105, and a floppy disk drive (FDD) 106.
The display 101, keyboard 102, memory 104, magnetic disk 105, and FDD 106 are accessed by the CPU 103 via a bus. The magnetic disk 105 stores an index file 8000.
The memory 104 is loaded with a system control program 5000, a search interface program 6000, a registration control program 2000, a search control program 3000, a keyword assignment program 2100, an index creation registration program 2200, and an index search program 3100, and a work area 4000 is secured. You.
Documents registered in the document database of the document search system are stored on the floppy disk 107 and accessed by the CPU 103 via the FDD 106.
[0018]
In this system, the system control program 5000 is started by the CPU 103 when the power is turned on, and the registration control program 2000 and the search control program 3000 are started under the control of the system control program 5000.
First, an outline of a document registration process in the present system having such a configuration will be described.
The system control program 5000 activates the registration control program 2000 in accordance with an instruction input from the keyboard 102 by the user.
In the registration control program 2000, before registering a document, the keyword assignment program 2100 is started according to an instruction input by the user from the keyboard 102, and an index file is initialized.
First, in accordance with an instruction input from the keyboard 102 by the user, an index file 8000 for storing an index is secured on the magnetic disk 105 by a predetermined number of blocks, and is divided into a specified number of blocks.
Then, a predetermined number of documents (hereinafter, referred to as seed documents) for assigning a keyword to each block are read from the floppy disk 107 to the work area 4000 of the memory 104 via the FDD 106. For example, when creating 100,000 newspaper article DBs, several hundred to several thousand articles of the same type, ie, newspaper articles, are registered as seed documents.
Next, words required for the search are extracted as keywords from the seed documents read into the work area 4000, and the number of appearing documents of the keywords is calculated.
Keywords are assigned to the respective blocks so that the sum of the number of documents in which the keywords appear is substantially equal between the respective blocks. Thereafter, the registration control program 2000 starts the index creation registration program 2200.
[0019]
The index creation registration program 2200 reads the document to be registered stored on the floppy disk 107 into the work area 4000 of the memory 104 via the FDD 106 in accordance with an instruction input by the user from the keyboard 102.
The words required for the search are extracted as keywords from the registered document, and the keywords and the document numbers or the document numbers are registered in the corresponding blocks of the index file 8000.
[0020]
Next, an outline of a document search operation in the present system will be described. The system control program 5000 activates the search control program 3000 and the search interface program 6000 in accordance with an instruction input from the keyboard 102 by the user.
Thereafter, the query word input by the user from the keyboard 102 is input to the search interface program 6000 and sent to the search control program 3000.
The search control program 3000 activates the index search program 3100 and sends the query to the program.
The index search program 3100 reads out the document number from the index corresponding to the received query word and sends it to the search control program 3000 as a search result.
This search result is sent to the search interface program 6000 and displayed on the display 101 as a search result document number.
[0021]
Next, the configuration of the keyword assignment program 2100 and keyword assignment processing will be described with reference to FIG.
The keyword assignment program 2100 includes an index division step 2105, a repetition step 2110 for the number of seed documents, a seed document reading step 2120, a keyword extraction step 2130, an appearance document count step 2140, and a keyword assignment step 2150.
First, in the index division step 2105, the number of blocks specified by the user is read from the keyboard 102. Next, an area for the specified number of blocks is secured as the index file 8000, and this is equally divided into the specified number of blocks.
Next, in a seed document reading step 2120, one seed document is read via the FDD 106 and stored in the work area 4000.
Further, in a keyword extraction step 2130, a word to be a keyword is extracted from the read seed document, and the extracted keyword is stored in the work area 4000.
Techniques for extracting keywords from this document are described in “Trends in Automatic Indexing Research” for Japanese documents (Morohashi, IPSJ Journal, Vol. 25, No. 9, 1984). About “DOCUMENT DATABASE” (G. James, Van Nostrand Reinhold Co., 1985) pp. 87-94. In the present embodiment, these keyword extraction techniques are used as they are.
In the appearing document number counting step 2140, the number of documents in which the extracted keyword appears is counted, and stored in the work area 4000 in association with the keyword.
In the repetition step 2110 for the number of seed documents, a series of processing from step 2120 to step 2140 is repeated for all seed documents.
After all seed documents have been processed, the keyword assignment step 2150 determines the number of appearing documents. Total The extracted keywords are assigned to each block so that the sum becomes substantially equal, and the block number is stored in the work area 4000 in a form corresponding to the keyword.
[0022]
As described above, by calculating the number of appearing documents of the keyword extracted from the seed document, the index size of the document database can be predicted. That is, the size of the index corresponding to the keyword extracted from the seed document can be calculated by the product of the number of occurrence documents of the keyword extracted from the seed document and the document number size, and the number of registered documents in the document database and the number of seed documents (sampled This is because the index size in the document database can be predicted by multiplying the ratio by the number of seed documents.
In addition, the number of documents registered in the document database increases, and the size of all indexes allocated to blocks is reduced. Total When the sum is not substantially equal among the blocks, a seed document candidate is extracted from all the documents registered in the document database by several percent of the total number of documents by using a method such as random number extraction. Based on this kind of document, keywords are assigned to blocks again.
By re-registering all the documents registered in the document database based on this keyword assignment, the size of the index assigned to the block can be reduced. Total The sum can be made substantially equal between the blocks.
As another method, for a block in which the sum of the sizes of the indexes assigned to the blocks is larger than that of the other blocks, the keywords assigned to the blocks are reassigned to the smaller-sized blocks, so that the sum of the index sizes is reduced. Can be made substantially equal between the blocks.
As an index for assigning a keyword to a block, it is also possible to use the number of appearances of the keyword itself or the number of types of keywords in addition to the number of document appearances of the keyword.
By performing the above processing, keywords can be extracted from all the seed documents, and the keywords can be assigned to the blocks so that the index sizes of the blocks are substantially equal.
[0023]
Further, the keyword assignment processing in the keyword assignment step 2150 will be described in detail with reference to FIG.
The keyword assignment step 2150 includes an extracted keyword sorting step 2152, a block number initial setting step 2153, an extracted keyword repetition step 2154, a block repetition step 2155, a block size determination step 2156, a keyword setting step 2157, a block number counting step 2158, and a jump step 2159. It is composed of
First, in the extracted keyword sorting step 2152, the extracted keywords stored in the work area 4000 are sorted in descending order of the number of documents of the extracted keywords stored corresponding to the extracted keywords.
Next, in the block number initial setting step 2153, 1 which is the first block number is set as the block number to be processed first.
Further, in the block size determination step 2156, assuming that the extracted keyword is allocated to the block, the sum of the sizes of the indexes allocated to this block is calculated, and it is determined whether or not the size exceeds a predetermined block size.
Only when it does not exceed, first, a keyword setting step 2157 is executed, and an extracted keyword is assigned to this block.
Next, a block number counting step 2158 is executed, 1 is added to the block number of the block to be processed, and the block number of the block to be processed next is set.
In this counting up, when the block number has been counted up to the last number, it returns to the first block number of 1.
Further, the repetition processing in the block repetition step 2155 is terminated by the jump step 2159, and the steps after L1 are executed. Here, the extracted keyword repetition step 2154 is executed.
In block repetition step 2155, the processing from step 2156 to step 2159 is repeated for all blocks in order from the block number.
In the extracted keyword repetition step 2154, the processing from step 2155 to step 2159 is repeated for all the extracted keywords.
Through the above series of processing, since the extracted keywords having the largest number of appearing documents can be allocated to each block in order, it is possible to allocate the index size to each block almost equally.
[0024]
Next, the configuration of the index creation registration program 2200 and the document registration process will be described with reference to FIG.
The index creation registration program 2200 includes a document number acquisition step 2205, a document number repetition step 2210, a registered document number reading step 2220, a keyword extraction step 2230, a block number correspondence step 2240, a document number counting step 2245, and a block number repetition step 2250. , An extracted keyword number determination step 2260, a block unit index addition step 2270, a repetition step 2280 for the number of keywords, and a keyword unit index addition step 2290.
First, in the document number acquisition step 2205, the first document number and the number of registered documents of the registered document input by the user from the keyboard 102 are read.
Next, in a registered document reading step 2220, a document to be registered is stored in the read work area 4000 for one document via the FDD 106.
After that, in a keyword extraction step 2230, a keyword word is extracted from the read registered document, as in the keyword extraction step 2130 in the keyword assignment program 2100. This extracted keyword is stored in work area 4000.
In the block number corresponding step 2240, the block number of the block to which the extracted keyword is assigned is obtained by examining the block number stored in the work area 4000 in a form corresponding to the keyword in the keyword assigning step 2150, and the block number is extracted. It is stored in the work area 4000 in association with the keyword. If the extracted keyword is not assigned to any block, the keyword is assigned to the block having the smallest sum of the stored indexes, and the block number is assigned to the work area 4000 in a form corresponding to the extracted keyword. To be stored.
Thereafter, in a document number counting step 2245, the document number is incremented to prepare for the processing of the next registered document.
In the repetition step 2210 for the number of documents, the keyword extraction processing of steps 2220 to 2245 is repeated for the number of registered documents.
After that, in the extracted keyword number determination step 2260, the number of the extracted keywords assigned to the block 1 among the extracted block 1 is counted, and it is determined whether or not the number is equal to or more than a predetermined number N.
The user designates an optimal value for the predetermined number N in consideration of the performance of the used magnetic disk, the used computer, and the like.
If the count number is equal to or greater than the predetermined number N, the next block-based index addition process 2270 is executed. In the block unit index addition processing 2270, the block 1 is read into the work area 4000, and the index addition processing is performed collectively for all the extracted keywords assigned to the block 1.
If the predetermined number N is not reached, a keyword unit index addition step 2290 is executed. In the keyword unit index addition step 2290, only the index corresponding to the extracted keyword existing in the block 1 stored in the index file 8000 is read into the work area 4000, and the index of the document in which the extracted keyword corresponding to the index appears is read. The document number is added to the index and written again to the index file 8000.
In addition, in step 2280, which is repeated by the number of keywords, keyword-based index addition step 2290 is repeatedly executed until index addition processing is completed for all extracted keywords assigned to block 1.
In step 2250, which is repeated as many times as the number of blocks, steps 2260 to 2290 are repeatedly executed for all blocks to perform index addition processing.
In this embodiment, additional registration of a new document is realized in this manner.
[0025]
Further, the above-described block unit index addition processing step 2270 will be described in detail with reference to FIG.
The block unit index addition processing step 2270 includes a block reading step 2272, a keyword number repetition step 2274, an intra-block index addition step 2276, and a block storage step 2278.
First, in the block reading step 2272, the blocks designated by the repetition step 2250 for the number of blocks are read from the index file 8000 and stored in the work area 4000.
Next, in the in-block index adding step 2276, the keyword is added to the end of the index corresponding to the keyword assigned to the update target block read into the work area 4000 in the block reading step 2272 among the extracted keywords. Add the document number of the document that appeared.
Further, in the keyword number repetition step 2274, the index addition processing of step 2276 is repeatedly performed for all the keywords in the read block.
After that, in the block storage step 2278, the above-mentioned block whose index addition processing has been completed is stored in the index file 8000 again.
As described above, in the block unit index addition processing step 2270, the index addition processing is performed in block units.
As described above, by performing the keyword index addition processing collectively in block units, even when the number of additional keywords is large, the index addition processing can be performed in a short time.
[0026]
An example of storing information on keywords stored in the work area by the above-described keyword assignment program 2100 will be described with reference to FIG.
In keyword extraction step 2130, words serving as keywords are extracted from the seed document and stored in work area 4000. As shown in the figure, the keyword number is stored together with the keyword.
In this example, it is assumed that "core", "disk", "computer", and "IR" are extracted from the seed document as the extracted keywords.
In this step, a serial number is assigned as the keyword number in the order in which these extracted keywords were extracted, and stored in the work area 4000 in the form as in this example. In this example, the keywords “core”, “disk”, “computer” and “IR” are assigned keyword numbers of 1, 2, 3 and 4, respectively.
Next, in an appearing document number counting step 2140, the number of appearing documents in the seed document is counted for each keyword. At this time, the number of appearing documents is stored in a form corresponding to the keyword as shown in FIG. In this example, the numbers of documents appearing in the keywords “core”, “disk”, “computer” and “IR” are 2, 1, 3 and 2, respectively.
Thereafter, in a keyword assignment step 2150, keywords are assigned to blocks storing indexes based on the number of appearing documents. The block number assigned here is associated with a keyword as shown in the figure and stored in the work area 4000.
In this example, the block numbers of the blocks to which the keywords “core”, “disk”, “computer”, and “IR” are assigned are 1, 1, 2, and 1, respectively.
By storing information on keywords in the work area 4000 in such a format, it is possible to manage information on keywords extracted from the seed document, so that keyword allocation processing to blocks can be realized.
[0027]
An example of storing information on keywords stored in the work area by the above-described index creation registration program 2200 will be described with reference to FIG. In this example, it is assumed that "core", "disk", and "IR" are extracted as keywords from the registered document of document number 5.
In keyword extraction step 2230, words serving as keywords are extracted from the registered document and stored in work area 4000. At this time, the keyword number corresponding to the extracted keyword and the block number of the allocated block are obtained from the keyword information shown in FIG. At this time, as shown in this figure, the number of the acquired allocation block and the number of the registered document, which is the document appearing the allocation block, are stored in correspondence with the keyword.
In this example, the keywords “core”, “disk”, “computer”, and “IR” store 1, 2, 3, and 4 as keyword numbers, respectively, and the assigned block number and the registered document number respectively. The block number 1 and the document number 5 are stored.
Based on this information, an index addition process is performed in steps 2250 to 2290.
By storing the information on the keywords in the work area 4000 in such a format, the information on the keywords extracted from the registered document can be managed, so that a process of adding an index corresponding to the information can be realized.
[0028]
As described above, according to the present invention, for each block that stores an index, if the number of relevant extracted keywords is equal to or greater than a predetermined number, the block is read from the magnetic disk to the memory and the index for the relevant extracted keyword is determined. Is performed collectively on the additional processing memory, and by writing this to the magnetic disk, the number of accesses to the magnetic disk is reduced. If the number is less than a predetermined number, only the index corresponding to the corresponding extracted keyword is read into the memory. At the same time, the index addition process is performed, and by writing this to the magnetic disk, the amount of data accessed to the magnetic disk can be minimized. Registration processing can be realized.
[0029]
【The invention's effect】
According to the present invention, by executing a process of adding a plurality of indexes on the memory at a time, the number of accesses to the magnetic disk can be reduced, so that a high-speed index adding process can be performed, and a document database can be added. The registration process can be performed at high speed.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a document search system to which the present invention has been applied.
FIG. 2 is a diagram illustrating a configuration example of an index.
FIG. 3 is a diagram illustrating a configuration example of an index when a keyword included in a new document is added.
FIG. 4 is a diagram showing a time chart at the time of index addition processing.
FIG. 5 is a diagram showing a configuration example of an index divided into blocks.
FIG. 6 is a diagram illustrating a configuration example of an index divided into blocks when a keyword included in a new document is added.
FIG. 7 is a diagram showing a time chart at the time of index addition processing when eight consecutive reading processing and eight consecutive writing processing of a logical block are performed.
FIG. 8 is a diagram showing a processing procedure of a keyword assignment program 2100.
FIG. 9 is a diagram showing a processing procedure of a keyword assignment step 2150.
FIG. 10 is a diagram showing a processing procedure of an index creation registration program 2200.
FIG. 11 is a diagram showing a processing procedure of an index addition processing step 2270 in block units.
FIG. 12 is a diagram illustrating a configuration example of a keyword management table.
FIG. 13 is a diagram illustrating a configuration example of a work table.
[Explanation of symbols]
101 Display
102 keyboard
103 CPU
104 memory
105 magnetic disk
106 FDD
107 floppy disk
2000 Registration Control Program
3000 search control program
4000 work area
5000 System control program
6000 search interface program
8000 index file

Claims (4)

A document search method for extracting a keyword from a document, creating an index including document identification information of the document in which the extracted keyword appears, and performing a search by referring to an index corresponding to the keyword that matches the query word at the time of search.
The index storage area on the secondary storage device for storing the index is divided into a predetermined number of blocks, and a keyword is set so that the total sum of the capacities of the indexes stored in each block over each block is substantially equal. Is allocated to the block, and an index corresponding to the block is stored. If the number of indexes added or updated in the block is greater than or equal to a predetermined number at the time of registering a document, the block is secondarily stored. In addition to reading from the measure into the main memory, for the keyword corresponding to the index in the block, an index adding process is performed on the main memory, and the block subjected to the additional processing is stored in the secondary storage device. In such a case, the index is read from the secondary storage device into the main memory, and the The keywords that performs additional processing of the index on the primary storage, document retrieval method and storing the index that is additional processing to the secondary storage device. The document search method according to claim 1,
When assigning a block to a keyword, for the keyword assigned to the block, the number of appearing documents of the keyword among all the documents to be indexed is calculated, and the sum of the number of appearing documents calculated for each keyword is calculated for each block. A document search method characterized by calculating and allocating keyword blocks so that the total number of appearing documents in each block is substantially equal in each block. A document search apparatus that extracts a keyword from a document, creates an index including document identification information of the document in which the extracted keyword appears, and performs a search by referring to an index corresponding to the keyword that matches the query word at the time of search.
Means for dividing the index storage area on the secondary storage device for storing the index into a predetermined number of blocks, and the sum total of the capacities of the indexes stored in each block over each block is substantially equal. Allocating means for allocating a keyword to the block and storing an index corresponding to the block ;
Means for determining whether the number of indices added or updated in the block is greater than or equal to a predetermined number when registering a document, and reading the block from the secondary storage device to the main storage when the determination result is greater than or equal to the predetermined number Means for performing an index addition process on a main memory for a keyword corresponding to an index in the block, storing the added block in a secondary storage device, and storing the index in a secondary storage device when the number is less than a predetermined number. A document which is read from a device into a main storage, performs a process of adding an index to a main storage for a keyword corresponding to the index, and stores the added index in a secondary storage device. Search device. The document search device according to claim 3,
The assigning means calculates, for each keyword assigned to the block, the number of appearing documents of the keyword out of all documents to be indexed, and calculates the sum of the number of appearing documents calculated for each keyword for each block A document retrieval apparatus characterized in that keyword blocks are allocated so that the sum of the number of appearing documents in each block is substantially equal in each block.

Images