JP3328334B2 - Full-text database search device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[Industrial applications] Recent advances in computer technology
A large number of documents have been digitized, and the demand for searching them has been increasing. In a database search, there is a method of assigning a keyword in advance, but it is difficult to assign a keyword to a large amount of full-text data.

On the other hand, there is also a full-text search system in which a text is directly referred to and collated based on a keyword freely designated by a searcher without preparing secondary information in advance. However, this has a problem in execution time for searching the full text. In the middle of these two, there is an attempt to treat all characters in the whole sentence as a keyword, give an index to every character in the whole sentence, and perform a high-speed search without keywords. .

[0003] The present invention relates to a full-text database search device that holds address information of each character as described above and performs a full-text search using the address information as an index. In particular, the present invention relates to a technique for creating the index. It is.

[0004]

2. Description of the Related Art As described above, in a full-text database search apparatus that holds address information of each character and performs a full-text search based on the address information, conventionally, the address information of each character is stored in each character. On the other hand, the next character information appearing in the text and its position were held as a pair.

However, in the above-described conventional holding method, it is necessary to hold the following character information and position information for the characters in the entire text, and the storage capacity required for this index is 2 bytes in the text. More than twice as needed. Also, at the time of the search, it is necessary to sequentially look at the character appearing next to each character, so that it has been difficult to increase the search speed.

[0006]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and provides a high-speed access to a full-text database and a conventional index storage area. It is an object of the present invention to provide a full-text database search device that can be reduced in number as compared with the first embodiment.

[0007]

FIG. 1 is a block diagram showing the principle of the present invention. In the figure, 1 is a full-text storage unit that stores a full-text text 1a, 2 is an index area unit that stores the position of each character of the full-text text 1a in the full-text storage unit 1, and 3 is an index area for a given key. A search unit 4-1 to 4-n for searching the full-text storage unit 1 using the index information of the unit 2 directly records information on the character appearance position at a position corresponding to the code of the character next to each character code. Array (circles indicate areas where data is present), 2a is direct array 4-1 to 4-
This is a compressed array in which n is shifted to the right or left by an appropriate amount and superimposed.

As shown in FIG. 1, a full-text storage unit 1 holding a full-text text 1a and a full-text text 1a for retrieving the contents of a digitized document. An index area section 2 for holding the appearance position of one character, and a search section 3 for performing a full-text search with reference to the index of the index area section 2. In a full-text database search device that also holds the next character information in the full-text database, based on a list of pairs of the next character information and information on the character appearance position given to each character code, .., 4-n in which character appearance position information is recorded at a position corresponding to the code, and elements whose values are defined do not overlap. .., 4-n obtained from each character code are shifted to the right or to the left by an appropriate amount, and one compressed array 2a is obtained from a set of a plurality of direct arrays. Area part 2
And a requested character string is searched from the full-text database using the compressed array 2a stored in the index area 2.

According to a second aspect of the present invention, in the first aspect of the present invention, information regarding each character located before the next character information is not explicitly provided in the compression array 2a.
Full text 1a based on character appearance position information in a
Search for each of the above characters, and based on the search result,
In this case, it is confirmed that the information of the character appearance position extracted from the character a is correct information.

According to a third aspect of the present invention, in the first aspect, a two-character string of the same combination appears a plurality of times in the full-text text 1a, and the position information of a specific character code is included in the compressed array 2a. Are stored in a different area from the compressed array 2a. According to a fourth aspect of the present invention, in the first aspect, the same combination of two character strings appears a plurality of times in the full-text text 1a, and there is a plurality of pieces of position information of a specific character code in the compressed array 2a. At that time, the plurality of pieces of information are
This is held in the continuous area of a.

According to a fifth aspect of the present invention, in the fourth aspect of the present invention, a plurality of pieces of position information are held in the divided continuous area of the compressed array 2a by holding the pointer flag in the compressed array 2a. It is like that.

[0012]

According to the present invention, for each character of the full-text text 1a stored in the full-text storage unit 1, the character appearance position is set at the position corresponding to the code of the next character based on the pair information of the next character and the character appearance position. ,..., 4-n in which
Create Then, the compressed array 2a is obtained by directly shifting the array to the right and left by an appropriate amount so that the data is not overlapped to obtain the compressed array 2a and storing it in the index area 3.

At the time of retrieval, the retrieval unit 3 retrieves information corresponding to the requested key from the compressed array stored in the index area unit 2, accesses the full-text storage unit 1 using this information, and Retrieve the contents. In the invention of claim 1 of the present invention, as described above, the direct sequence 4-
Since the compressed sequence 2a is obtained from 1,..., 4-n and the full-text data 1a is searched using the compressed sequence 2a, high-speed search is possible and the direct sequence 4a is obtained.
By superimposing 1,..., 4-n, a sparse array is compressed.

According to a second aspect of the present invention, in the first aspect, each element in the compression array 2a is:
Since the character information is obtained by referring to the full text 1a, the index area 2 can be made compact. According to a third aspect of the present invention, in the first aspect of the invention, when there is a plurality of pieces of position information of a specific character code in the compressed array 2a, the plurality of pieces of information are stored in an area different from the compressed array 2a. Therefore, it is possible to cope with a case where two character strings of the same combination appear a plurality of times in the full text 1a.

In the invention of claims 4 and 5 of the present invention,
In the invention of claim 1, when a two-character string of the same combination appears a plurality of times in the full-text text 1a and there is a plurality of pieces of positional information of a specific character code in the compressed array 2a, the plurality of pieces of information are transferred to the compressed array 2a. Or the compressed area 2a is divided and held in a continuous area, so that the index area 2 can be made compact.

[0016]

FIG. 2 is a diagram showing the basic configuration of a full-text database search apparatus according to the present invention. In FIG. 2, reference numeral 11 denotes a full-text storage unit for storing the entire text to be searched;
Is an index area, which stores the position of each character in the full-text storage as described later. Reference numeral 13 denotes an index creation unit that is started when a part or all of the text in the full-text storage unit 11 is created or updated, and creates an index of the text. A search unit 14 searches the full-text storage unit 11 for a given key using the index information stored in the index area unit 12. Reference numeral 15 denotes a user interface unit that receives a search request from a user and outputs a search result.

In FIG. 1, when the text in the full-text storage unit 11 is created or updated, the index creation unit 1
3 is activated. The index creator 13 determines, for each character in the full-text text stored in the full-text storage 11,
An index is created as described later, and this index is stored in the index area unit 12. When the user issues a search request from the user interface unit 15,
The search unit 14 extracts information corresponding to the requested key from the index area unit 12 and accesses the full text storage unit 11 using this information to extract the requested content.

The user interface unit 1 informs the user of the retrieved result or, if the retrieval has failed, that fact.
5 to the user. Next, an embodiment of index creation and search processing in the full-text database search device shown in FIG. 2 will be described. (1) Creation of Index FIG. 3 is a diagram showing a flowchart of an index creation process in the index creation unit 13, and referring to FIG. The creation will be described.

In step S1 of FIG. 3, the full-text file stored in the full-text storage unit 11 is opened. In step S2, it is determined whether or not the file has been completed. In step S3, one character is read from the full-text storage unit 11, and step S4
go to. In step S4, a pair of the position of the read character in the full-text file and the next character is output to the work file. Then, the above processing is repeated until the end of the file, the above processing is performed for all the characters of the full text, and when the processing is completed, step S
Go to 5.

FIG. 4 is a diagram showing an example of a full-text text to be indexed and an example of the work file created in step S4. 4A shows an example of a full text, FIG. 4B shows a created work file, and FIG.
The numbers attached to the vertical and horizontal axes are addresses indicating the position of each character.

In (b), the leftmost column is a heading character, and the next column is an address indicating the next character of the heading character and its position in the full-text, and as shown in FIG. For characters that appear multiple times in
Only one character is recorded in the column of heading characters, and a plurality of characters following the heading character are recorded in the column in which the character next to the heading character is recorded. For example, the word "day" is used as "Japan", "daily", and "daily" in the full text, but "date" appears only once in the column of heading characters.
In the column indicating the next character, "book", "at", "around" and addresses indicating their positions in the full text are recorded.

When the work file is created for the full text as described above, the process goes to step S5 in FIG. 3, and an array is created directly for each character code. FIG.
Is a diagram showing an example of a direct arrangement. In FIG.
Indicates a heading character, and 52 indicates a direct arrangement corresponding to the heading character. In the figure, as an example, the character "day" is a heading character, and the characters following "day" are "book", "chi",
"Circle" is address 4,8,2 in full text
The array when it appears at position 4 is shown.

In the figure, an array 52 has areas corresponding to the number of character codes (character sets) that can be used in the full text, and each area is arranged in the order of the character code. Corresponds to the character code. That is, by specifying the position of the array 52, the character code can be specified. For example, when the character "book" appears at address 4 in the full text, "4" is recorded at the position of the character code "book" in the array 52 as shown in FIG. Similarly, if the character "" appears at address 8 in the full-text,
“8” is recorded at the position of the character code “2” of “2”.

Further, "nil" (meaning no data) is recorded in an area other than the area where the address is recorded. Therefore, if the heading characters are
In the case of “を” and “day”, an array as shown in FIG. 6 is created, and its address (for example, “9”, “2”,
Characters (eg, “7”) from the position where they were recorded
"No", "i", "day", etc.).

That is, for each character code i in the full-text text, an array Ai of elements of the number of character sets is prepared, and when the character of code j exists as the next character of the character of code i, Ai [ j] stores a pointer (address) to a list that stores information indicating the position of the character in the full-text database. When the character of code j does not exist as the next character of the character of code i, “n
il ”is stored.

In the above example, for convenience of explanation, an example is shown in which a pointer (address) to a list storing information indicating the position of the code j in the full-text database is stored. A pointer (address) to a list storing the information to be indicated may be stored. After an array is directly created for each character in the full text as described above, the process goes to step S6 in the flowchart of FIG. 3, and is shifted to the right or left appropriately so that non- "nil" information does not overlap the compressed array. Create a compressed array by directly overlaying the arrays.

FIG. 7 is a diagram showing the array overlapping process in step S6. In the figure, 71, 72,
73 denotes an array corresponding to the first to third character codes, 74 denotes a compressed array in which the arrays 71, 72, 73 are superimposed, and in the arrays 71, 72, 73 in FIG. "Nil" is indicated, and the position with a circle indicates a non-nil (the above-mentioned address is recorded).

In order to create the compressed array 74, FIG.
When arrays 71, 72, and 73 shown in FIG. 2 are given, as shown in FIG. 2B, for example, array 71 is shifted to the left and arrays 72 and 73 are shifted to the right so that the positions of the circles do not overlap. Shift one. Next, as shown in FIG.
The compression array 74 is created by superimposing 1, 72 and 73. In addition, in order to indicate which character code corresponds to each position of the compressed array, as shown in FIG. 10C, in the area corresponding to each position of the compressed array, which character code corresponds to the position is shown. Is recorded. For example, as shown in FIG. 7C, the leftmost column of the compressed array 74 corresponds to the first character code of the array 71 of FIG. "1" is recorded in the area corresponding to the column, and similarly, to indicate that the rightmost column of the compressed array 74 corresponds to the third character code of the array 73, the leftmost column is displayed in the compressed array 74. “3” is recorded in the corresponding area.

Further, the shift amounts of the arrays 71, 72.73 and the like are stored as a shift amount table. FIG. 8 is a diagram showing a schematic diagram of a compressed arrangement of a shift amount table and an index.
FIG. 7A shows an example of a shift amount table, and FIG. 7B shows an example of a compressed array of indexes.

The shift amount table records a heading character and a shift amount corresponding to each heading character.
For example, as shown in FIG.
The shift amounts of "2" and "wo" are recorded as "-29". In addition, the compressed array of indexes has a structure as shown in FIG.
The address 24 of the character "Chiro" in the character string "Chiro" is
In the compressed array, the shift amount of the character “day” is “−125” from the original position of the “period” in the character set.
125 "). (2) Search When the user issues a search request from the user interface unit 15, the search unit 14 extracts information corresponding to the requested key from the compressed array of the indexes stored in the index area unit 12, and The full text storage unit 11 is accessed using the information to extract the requested content.

In order to search for a character string in which the code i and the code j are continuous, first, a shift amount Ti corresponding to the character code i is obtained with reference to a "shift table" shown in FIG. Next, the address L in the full-text text in the full-text storage unit 11 is extracted from the Ti + j-th element in the index area shown in FIG. Also, Ti + j in the index area
Information indicating which character code the area corresponding to the element corresponds to is extracted (see FIG. 7C). If the character code of the extracted information is the character code i, the address L is correct. Let it be information. If the character code of the extracted information is not the character code i, it indicates that there is no character string in which the character codes i and j are continuous.

For example, in FIG. 8, when searching for a character string in which "date" and "book" are continuous, first, FIG.
Referring to the “shift table” shown in FIG.
25 ”is taken out, and“ j-125 ”is obtained from the character code j of“ book ”and the above“ shift amount ”. Next, FIG.
The address 4 recorded at the position of “j-125” is extracted from the index area of (b). Also, "j-12
Extract the character code corresponding to the area of "5",
When the character code is a character code corresponding to "day", it is assumed that a character string in which "day" and "book" are continuous is located at the address 4 in the full text.

In the above embodiment, in order to indicate which character code corresponds to each position of the compressed array, FIG.
As shown in (c), information indicating which character code the position corresponds to is recorded in an area corresponding to each position in the compressed array, and the above information is referred to at the time of search with reference to the above information. As described above, the example of confirming that there is a character string in which the character codes i and j are continuous has been described. However, as described above, the character string is continuous without having information indicating which character code corresponds to the position. You can also verify that a string exists.

That is, since each character and its address are stored in the full-text text in the full-text storage unit 11,
When the address L in the full text is extracted from the Ti + j-th element of the index area, the full text storage unit 11
Is searched for the position of address L from the full-text text, and if the character code before that position (if the address L indicates the position of the previous character, the character code at that position) is the character code i, the character code i It can be confirmed that the codes i and j are continuous. (3) Creation of an index when the same combination of two character strings appears multiple times in the full text.

In the above (1), the case where the two character strings appear only once in the full-text text has been described. However, there are cases where the same combination of two character strings appears multiple times in the full-text text. In this case, it is necessary to hold a plurality of pieces of position information in the index. FIG.
(A) is a diagram showing a work file in the case where the two character strings of the same combination appear in the full-text text a plurality of times, and shows a continuous character string composed of a heading character β and a character code E. Appear four times at addresses 8, 59, 64, and 96.

FIG. 9B is a diagram showing an embodiment in a case where a plurality of pieces of position information need to be held in the index as described above. In this embodiment, a plurality of pieces of position information are not provided in the compressed array of indexes, and the plurality of pieces of position information are stored in a location different from the compressed array. As a method of storing in a different location, for example, the head address of a list storing a plurality of pieces of position information is stored in the compressed array, and the plurality of pieces of position information are read out using the head address of the list.

Alternatively, an array H different from the compressed array is prepared, and the above-mentioned position information is stored in a continuous area of the array H, and the number of data or the end mark of the data is marked so that the number or end of the data can be known. Provide. Then, the index of the first data in the array H in which the data is stored is stored in the compressed array, and a plurality of pieces of position information are read out with reference to the index.

FIG. 9C is a diagram showing another embodiment in which a plurality of pieces of position information need to be held in the index. FIG. 9C shows a case where a plurality of pieces of position information are stored in an area different from the compressed array. Instead of doing so, an embodiment is shown in which it is stored directly in a compressed array. If there is at most one piece of position information for a given two-character continuation, the actual position information is simply stored in the compressed array. An area mark bit area is introduced, and as shown in FIG. 10C, while the data is continuous, the continuous area mark bit is turned on.
Position information is stored on the compressed array area.

FIG. 9D shows a method of storing a plurality of pieces of position information when a long continuous area cannot be secured when the plurality of pieces of position information are directly stored in the compressed array. When a long continuous area cannot be secured, the area is divided and stored. That is, as shown in FIG. 9D, a pointer bit is prepared in addition to the above-mentioned mark bit, and when the pointer bit is on,
The data is not actual data but indicates an index on the compression area where the next data is stored.

[0040]

As is apparent from the above description,
In the present invention, for each character of the full-text text, a direct array is created in which the character appearance position is recorded at the position corresponding to the code of the next character based on the pair information of the character that appears next and the character appearance position Then, the direct array is shifted to the right and left by an appropriate amount and superimposed so that the data does not overlap to obtain a compressed array, and the full-text text is accessed using the compressed array as an index. From the character code of the character, the position information in the full text of the character to be searched can be immediately obtained, and the access can be speeded up. Further, it becomes possible to reduce the storage area for the index as compared with the related art.

[Brief description of the drawings]

FIG. 1 is a principle block diagram of the present invention.

FIG. 2 is a diagram showing a basic configuration of a full-text database search device of the present invention.

FIG. 3 is a diagram showing a flowchart of index creation.

FIG. 4 is a diagram showing an example of a full text and a work file.

FIG. 5 is a diagram illustrating a configuration of a direct array.

FIG. 6 is a diagram showing a state of an index array before compression.

FIG. 7 is a diagram illustrating the superposition of arrays.

FIG. 8 is a diagram illustrating an example of a shift amount table and an index array after compression.

FIG. 9 is a diagram showing a storage state of position information in a compressed array.

[Explanation of symbols]

 1,11 full-text storage unit 1a full-text 2,12 index area 2a compressed array 3,14 search unit 4-1 ... 4-n direct array 13 index creation unit 15 user interface unit

────────────────────────────────────────────────── ─── Continuation of front page (56) References JP-A-4-242864 (JP, A) JP-A-4-215181 (JP, A) JP-A 4-205560 (JP, A) Yasuji Ogawa, short unit Text Database System Based on Keywords, Information Processing Society of Japan Research Report 92-DBS-90, Japan, Information Processing Society of Japan, September 11, 1992, Vol. 92. No. 71, pages 45 to 54 (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 17/30 JICST file (JOIS)

Claims (5)

(57) [Claims] 1. A full-text storage unit (1) holding a full-text (1a) for searching the contents of a digitized document.
And an index area (2) that holds the appearance position of each character in the full-text text (1a), and a search unit (3) that performs a full-text search by referring to the index of the index area (2). In a full-text database search device that also holds the next character information in the full-text database in addition to the appearance position of each character as an index, a pair of the next character information and character appearance position information given for each character code , A direct array (4-1, ... 4-n) in which character appearance position information is recorded at a position corresponding to the character code of the following character information, and the element whose value is defined is By overlapping the direct arrays (4-1,... 4-n) obtained from each character code to the right or left by an appropriate amount so as not to overlap, one compressed array (2a Get on Stored in the index area portion (2), the full text database using stored compressed sequence in the index area portion (2) and (2a), full-text database retrieval apparatus characterized by retrieving the requested string. The compressed array (2a) does not explicitly have information on each character positioned before the next character information.
Full text (1a) based on character appearance position information in (a)
Search for each of the above characters, and use the compressed sequence (2
2. The full-text database search device according to claim 1, wherein the character appearance position information extracted from a) is confirmed to be correct information. 3. When two character strings of the same combination appear multiple times in the full-text text (1a) and there is a plurality of pieces of position information of a specific character code in the compressed array (2a), the plurality of pieces of information are compressed. 2. The full-text database search device according to claim 1, wherein the full-text database search device is held in an area different from the array (2a). 4. When two character strings of the same combination appear multiple times in the full-text text (1a) and there are a plurality of pieces of position information of a specific character code in the compressed array (2a), the plurality of pieces of information are compressed. 2. The full-text database search device according to claim 1, wherein the full-text database search device is stored in a continuous area of the array (2a). 5. The full-text database according to claim 4, wherein a plurality of pieces of position information are stored in divided continuous areas of the compressed array (2a) by holding pointer flags in the compressed array (2a). Search device.