JP2001043237A - Data file and data retrieving method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten time required for reading from a disk device by composing a data file of a transposed file for which all respective divided blocks are linked and reprogrammed into groups and these groups are linked later. SOLUTION: A block 4 is constituted by dividing M records of a preset fixed number in the respective fields of a source file storing plural records composed of plural fields and these respective blocks 4 are linked and reprogrammed into group 5. Then, all the records of the source file are reprogrammed into groups and the transposed file is prepared by linking these groups 5 later. Further, a bit map index is generated by reading only the block of an index key field required for index generation from the transposed file. Thus, time required for reading from the disk device can be shorten.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data file and a data search method for efficiently extracting only necessary data from a large amount of data in a system for storing a large amount of data such as a database management system. is there.

[0002]

2. Description of the Related Art In a conventional database management system, as a data file for efficiently extracting only necessary data, for example, a hash file shown in FIG. And B + Tree, and a search method using a hash index, a B + Tree index, a bitmap index, or the like as a secondary index.

FIG. 24 shows a hash file. The hash file includes a hash file index unit 32 composed of a packet directory storing a hash function, each hash value and data area information storing data having the hash value, and a data area 33 storing a record corresponding to each hash value. And a hash file data section 34. When a record is added, the value of the key field 31 (FieldA) of the input record 3 is applied to a hash function to obtain a hash value, and a data area 33 for storing the record is obtained from the hash value using a packet directory and stored. In the created hash file, only records having the corresponding hash value are stored in each data area 33.

[0004] In FIG. 25, the product number in the sales table is used as a hash key, and the hash function is Modulo5 (the remainder after division by 5). The hash value is an integer from 0 to 4, and each data area 33 of the data section 34 stores data corresponding to each hash value.

FIG. 26 shows a B + Tree. For B + Tree,
This is a method of determining the data area 33 to be stored in the range of the value of the key field 31 instead of the hash. Also in this case, the area of the index section 32 and the area of the data section 34 are set in advance. When adding a record, the value of the key field 31 (FieldA) of the input record 3 is compared with the key value of the root node (56 in this example). If it is greater than or equal to the key value of the node, the process moves to the intermediate node indicated by the right pointer, and if it is smaller than the key value, the process moves to the intermediate node indicated by the left pointer. Thereafter, the comparison with the key value in the node is repeated sequentially, and finally the storage position of the leaf node (leaf page), that is, the data area
33 is decided. In the data area, records are further stored in ascending key order. In each case, the cluster index is used, and the data storage order and the index order are the same.

FIG. 27 shows an example of a bitmap index. This is a secondary index unlike the above-mentioned two cluster indexes, in which an index is added later to a data part created in advance by another method. The bitmap index secures bits of possible values of the index key field (FieldA) 31 by the number of data areas, and turns ON the bit of the data area including the record of the corresponding key in the data area (1). Bit of data area not included
Create F (0). In this example, there are five possible values of the index key field: 'a', 'b', 'c', 'd', and 'e', so that an area of 5 (bits) × L (the number of data areas) Secure,
If the first area contains a record where FieldA = 'a', the corresponding bit is set to 1; if not, 0 is set.
If a record where FieldA = 'b' is included, the corresponding bit is set to 1; here,
The first area includes records in which FieldA is' b ',' c ', but does not include records in which FieldA is'a','d','e.

In any of the above cases, data is stored in the data area in the data section 34 of the disk device in units of records in the order of records, and is also stored in the record in the order of definition of the fields.

As shown in FIG. 24, when a record having an index key field 31 (FieldA) having a specific value is read out by using a hash file, a hash function is used to perform a corresponding operation from the specific value. The hash value is obtained, the data area 33 storing the record is specified from the hash value and the packet directory, and only the necessary data area 33 may be read. Because the amount of data to read decreases,
The time required for reading can be reduced, and the processing can be made more efficient.

As shown in FIG. 26, even in the B + Tree index, when reading out a record having a specific value in the index key field 31 (Field A), the key value of the root node is compared with the value, and then the value is compared. By specifying the next intermediate node from the comparison result and repeating it up to the leaf node, the data area including the record can be specified.
Therefore, the amount of data to be read is reduced, so that the time required for reading can be reduced, and the processing can be made more efficient.

As shown in FIG. 27, in the bitmap index, when reading out a record having a specific value in the index key field 31 (FieldA), the bitmap of the value is examined and the data in which the bit is set is read. It is sufficient to read only the area 33. Therefore, when the ratio of the data area in which bits are set is small, the amount of data to be read is reduced, so that the time required for reading can be reduced and the processing can be made more efficient.

However, in the case of the above-mentioned hash file or B + Tree, even if it is desired to read only a specific field of a record having a specific value of the index key field 31, the data area 33 in the data area 33 specified from the key value is read. Since data is stored in record order in units of records,
It is necessary to read all fields in the data area. In the example of the hash file in FIG. 24, even when it is desired to read only Field 2 of a record in which the index key field 31 (Field A) takes a specific value (for example, the hash value is 1), the index key field 31 (FieldA) is included in the data area 33 corresponding to the hash value 1 All fields (from Field1 to FieldN) must be read. That is, there is a problem in that fields that are not necessary for processing must be read.

In the bitmap index, even when it is desired to read out only a specific field of a record having a specific value of the index key field 31, the bitmap is checked from the value of the index key field 31 and the data in which the bit is set. Although the area 33 can be specified, all fields in the data area where bits are set must be read. Also in this case, there is a problem that fields that are not necessary for processing must be read. Other secondary indexes have similar problems.

In the secondary index, an index can be generated in principle by reading only the field for which the index is to be generated. However, data is stored in the disk device in the order of records, and is also stored in the record in the order of field definitions. Therefore, it is not possible to read only the field for generating the index, and it is necessary to read all data in the data area. Therefore, there is a problem that data unnecessary for index generation must be read.

On the other hand, as shown in Japanese Patent Application No. 9-319527, there is also a file management method for creating an inverted file by changing the data storage order. Referring to FIG.
A file management method using an inverted file will be described. 1 is record 3 composed of multiple fields 2
Is stored in a plurality of files, and a predetermined number of records in each field, for example, L records are divided from the original file into blocks 4. In the transposition file 6, the field 1 is stored in order from the block, and the field
Stores N blocks. A group obtained by rearranging the blocks from the field 1 to the block of the field N on the transposition file 6 is called a group 5. All records 3 of the original file 1 are reorganized into the group 5 and connected to generate the transposed file 6.

As shown in FIG. 29, in the transposed file 6, only the field 2 necessary for processing can be read. That is, only the block 4 composed of the field 2 necessary for processing can be read from the group 5 and stored in the input / output buffer. In this example, Field1,
Only the blocks of Field4 and Field99 are stored in the input / output buffer. As a result, it is possible to save the size of the input / output buffer and increase the processing efficiency by reducing the I / O time.

As shown in FIG. 30, in the file management method using the transposed file 6, in order to further speed up the processing, the maximum value and the minimum value of all the fields are stored for each data area 33, and the record input is performed. It is stated that the order is added. In this case, in order to read a record in which a specific field 2 (for example, FieldA) has a specific value, only a data area including the value between the maximum value and the minimum value of the field in each data area can be read. I just need.
However, if data is evenly distributed in each data area 33, there is a problem that almost all of the data area 33 must be read.

[0017]

In the conventional data file and retrieval method, for example, in order to read only a specific field of a specific record having a key value, the data area 33 is specified from the key value. There was a problem that all fields of all records in the area had to be read. Also, in the generation of the secondary index, there is a problem that fields unnecessary for the index generation must be read. On the other hand, in the data management method shown in the file management method using an inverted file, when reading only a specific field of a specific record with a key value, the record of the key value having a specific value or In some cases, the data area including the record cannot be specified well, and as a result, there is a problem in that fields required for processing must be read from all records.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. A method for specifying a data area including a certain key value, and reading only fields necessary for processing after specifying the data area. The purpose of this is to reduce the time required for reading from the disk device and to increase the efficiency of processing. Also, in the secondary index generation, by reading only the fields necessary for the index generation, it is possible to reduce the time required for reading from the disk device and to generate the index at a high speed.

[0019]

A data file according to the present invention is classified based on the values of fields of a record input from an original file, and an index for setting a pointer indicating an area for storing the record for each classification. A predetermined number of records of the records classified by the index unit are divided for each field and stored as blocks in the area indicated by the pointer, and the divided blocks are all connected to be regrouped. It is composed of an inverted file in which all the records are organized, reorganized into the above groups for all the records, and the groups are connected.

Further, a predetermined number of the fields described above are divided from the original file into blocks by dividing the original file from the original file storing a plurality of records composed of a plurality of fields, and all the divided blocks are connected. An inverted file in which the records are reorganized into groups and all of the records are reorganized into the groups after the reorganization into the groups, and an address of an array indicating the storage area of the records for each of the classifications classified based on the values of the fields. And a second index unit composed of a plurality of arrays in which record IDs indicating the storage locations of the records classified for each address are stored.

In the data search method according to the present invention, a predetermined number of the above-mentioned fields is divided from the original file into blocks by dividing the original file storing a plurality of records composed of a plurality of fields into blocks. In a transposed file in which all the blocks are connected and reorganized into groups and all the records are reorganized into the groups and then connected to the groups, the group including the key values of the preset fields of the records in the transposed file Is generated, and when a record corresponding to a preset key value is searched, a record of the group in which the flag corresponding to the key value is set is searched.

The key value comprises a hash value of the key.

Further, in the table, the flag is set for the group obtained by dividing the group.

Further, the fixed number is set for each table.

Further, the table comprises a first table of preset fields and a second table of preset fields, and a group common to each of the first table and the second table. Are searched for records in the above group under the AND condition of each corresponding key value.

Also, a predetermined number of the above-mentioned fields are divided from the original file into blocks by dividing the original file storing a plurality of records composed of a plurality of fields into blocks, and all the divided blocks are connected. In a transposed file in which all the records are reorganized into groups and all the records are reorganized into the groups and the groups are connected, the table lists the field values that can be taken for each group of the preset fields of the records, and creates a table. It is generated and searches for records in the group corresponding to the field value.

Further, a hash value is obtained by applying a key value of a predetermined field of a record of a record input from the original file to a hash function, and a predetermined fixed number of records including the above field is divided for each field. Stored as a block in the data area corresponding to the hash value, connected all the divided blocks, reorganized them into groups, and reorganized all of the records into the groups after reorganization into the groups. In the file, obtain a pointer to the data area corresponding to the hash value from the packet directory,
The record of the group indicated by the pointer is searched.

The key value classified into a hash value obtained by applying a key value of a preset field of a record input from the original file to a first hash function is applied to a second hash function, and the hash value is applied to the hash value. Is what you get.

Still further, the records of the group indicated by the pointer are searched based on a table in which a flag is set for each record of the group including a key value of a preset field.

Also, the classification of the key value of a preset field of the record input from the original file is determined,
The predetermined number of records including the above-mentioned fields is divided into each field and stored as a block in the data area corresponding to the above-mentioned classification. In a transposed file in which all the records are reorganized into the above groups after linking the groups, a pointer to the data area corresponding to the above classification is obtained from the key classification directory, and a record of the above group indicated by the pointer is retrieved. Things.

Further, when all the data areas of the above group have been stored, a new data area of one group is continuously set.

Further, the key value of a preset field of the record input from the original file is compared with the key value of the root node, and the process moves to the intermediate node according to the configuration rules of the balance tree structure. The comparison with the key value and the transition to the next node based on the result are repeated, a predetermined number of records including the above-mentioned fields are divided for each field and stored in the data area of the leaf node as a block, An intermediate node located at the lowest level of the intermediate node in a transposed file in which all of the divided blocks are connected and reorganized into a group, and all the records are reorganized into the group and then connected to the group. Is searched for the record of the group indicated by the pointer of the above.

Further, when all the data areas of the group have been stored, the intermediate node or the root node is divided according to the configuration rule, a new pointer is set, and the data area for one group indicated by the pointer is set. To set.

Also, a predetermined number of the fields described above are divided from the original file into blocks by dividing a predetermined number of the fields from the original file storing a plurality of records composed of a plurality of fields, and all the divided blocks are connected. After reorganizing into groups and reorganizing all of the above records into the above groups and concatenating those groups, the hash value is obtained by applying the key value of the above field to the hash function, and the record corresponding to the hash value A pointer to a data area corresponding to the hash value is obtained from a record pointer array in which the record ID of the group is stored, and a record of the group indicated by the pointer is retrieved.

Further, from the original file storing a plurality of records composed of a plurality of fields, a predetermined number of the above-mentioned fields are divided from the original file into blocks, and all of the divided blocks are connected. In the transposed file in which the records are reorganized into groups and all the records are reorganized into the groups and the groups are connected, pointers corresponding to the respective key values of the leaf nodes of the index of the balance tree structure are obtained, and the pointers are obtained. Is searched for the record by the record ID corresponding to the key value in the record pointer array shown in FIG.

Also, a predetermined number of the above-mentioned fields are divided from the original file into blocks by dividing the original file from the original file storing a plurality of records each composed of a plurality of fields, and all the divided blocks are connected. In a transposed file in which all the records are reorganized into groups and the groups are linked after reorganization into the groups, a table in which flags are set for the records including the key values of the fields of the records is set for each of the groups. When searching for a record generated and corresponding to a key value set in advance, the record in which a flag corresponding to the key value is set is searched for each group.

Furthermore, a predetermined number of the fields described above are divided from the original file into blocks by dividing a predetermined number of the fields from the original file storing a plurality of records composed of a plurality of fields, and all the divided blocks are connected. After reorganizing all the records into the above group and reconnecting them to the above group, obtain the pointers corresponding to each key value of the leaf node of the balance tree structure in the transposed file in which those groups are connected. A search is made for the record in the group in which the flag corresponding to the indicated key value is set.

[0038]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG. 1 shows a bitmap indexed inverted file according to the first embodiment of the present invention. This bitmap index is a secondary index, and for a previously generated inverted file,
A bitmap index is added later. The transposed file is composed of a predetermined number of records in each field, for example, M records are divided into blocks 4 of the original file storing a plurality of records composed of a plurality of fields, and the blocks are connected to form a group 5. Reorganized into
After reorganizing all the records of the original file into the above groups, these groups are linked and created. This figure shows a case where fields and records have a fixed length.

The bitmap index is an index key field
(FieldA) Secures the bits of the possible value of 31 by the number of data areas, sequentially reads the blocks of the field for creating the bitmap of the data area, and reads the bits of the data area containing the value of the corresponding index key field. ON (1)
Then, the bit of the data area not included is set to OFF (0). However, the key of the index key field here does not necessarily mean a primary key, but is used in a broad sense as a field for generating an index. In this figure, the possible values of the index key field (FieldA) 31 are '
a ',' b ',' c ',' d ', and' e ', where data areas and groups correspond one-to-one, and 5 (bits) × L (number of groups of transposed files ). Next, if the first group contains 'a' in the FieldA block,
ON (1) for the corresponding bit, OFF if not included
(0). Similarly, if "b" is included, the corresponding bit is set to ON (1), and if not, OFF (0). In this figure, in the first group, FieldA is 'b', 'c'
This is the case where the records that include Field A are 'a', 'd', and 'e, but are not included. Similarly, bitmap indexes are generated for all groups.

FIG. 2 shows a read processing flow when processing a record in which the index key field 31 has a specific value using a transposed file with a bitmap index. The bit map is read (step S10), and it is checked whether the group bit is ON (step S11). If the bit is ON (1), a block of a field required for processing is read from the group (step S12), and it is checked whether the group is the last group on the bit map (step S13). finish. If it is not the last group, the process is repeated from the process of reading the bitmap of the next group (step S10). When step S11 is NO, that is, when the bit is OFF (0), step S12 is skipped, and the process is continued from step S13. This processing flow is for a case where a bitmap is created for each group.

When reading a specific field of a record in which the index key field 31 has a specific value, the bit map of the specific value is checked to obtain the group number of the group whose bit is ON (1). Since only the blocks of the fields necessary for the processing in the group need to be read, the amount of reading can be reduced, the reading time can be reduced, and the processing can be more efficient. Also, there is an effect that processing can be performed with a small input / output buffer.

Furthermore, when a bitmap index is generated, it can be generated by reading only the block of the index key field necessary for index generation because the file is an inverted file, so that other fields in the record need to be read. Without
This has the effect that it can be generated faster than in a normal storage format.

In FIG. 1, a bitmap is created for each group. However, it is also possible to create a bitmap index by grouping a plurality of groups, or to create a bitmap index by dividing one group into a plurality. It is. FIG.
In, a bitmap is created in units of dividing a group into two. FIG. 4 is a schematic diagram of reading in that case. Since the bitmap was created in half the unit of the group, the read unit is also half of the group. In this figure, the field F of the group 2-1 / 2 (the first half of the group 2) in which the bit of the key value 'd' is ON (1)
Indicates that ield1, FieldA, and FieldN are to be read.

In the bitmap index, the greater the proportion of bits that are OFF (0), the more efficient the processing. Therefore, it is necessary to adjust the unit of bitmap index generation. As a guideline of the bitmap index generation unit, the number of records included in the bitmap index generation unit may be the same as or smaller than the type (cardinality) of the possible value of the bitmap generation field. By doing so, at least one OFF (0) bit is created. FIG.
Then, by generating a bitmap in half a block, when reading a record where FieldA is 'd',
The latter half of the data area of group number 2 (2 /
There is no need to read 2), which has the effect of reducing the amount of data to be read.

Since a bitmap index can be created for a plurality of arbitrary index key fields, it can also be used when searching for a record in which a specific field combination is a specific value combination. In such a case, a bitmap index is generated for a plurality of fields that form a pair, a bitmap of each value of the bitmap index of each field is read, and a logical sum (AND) is obtained. A bitmap indicating the data area can be configured. Figure 5 shows FieldA and Fiel
Generate bitmap index for dB, FieldA, FieldB
Shows a case where a record is read in which the combination of the key values 'e' and 'x' is a combination of the key values 'e' and 'x'. Only the group number 3 where both bits are set needs to be read. For example, “e” represents a product name and “x” represents the color of the product. Embodiment 2 FIG. FIG. 6 shows a data enumeration type indexed transposed file according to the second embodiment of the present invention. This data enumeration type index is also a secondary index, and adds a data enumeration type index to the transposed file 6 created in advance. The data enumeration index is
Index key fields (F
ieldA) 31 to enumerate the possible values, secure the data enumeration area by the number of index target areas, sequentially read the blocks of the index target area, index key field (FieldA) 31
If the value taken by is a new value, that is, a value that first appears in the index target area, the value is added to the data enumeration area and created. When the data enumeration area for a certain index target area becomes full, a new area is reserved and used as a data enumeration area. In this figure, the index target area is grouped.
First, the block of FieldA of group number 1 is read, and when new data appears, it is added to the data enumeration area. It is shown that the group number 1 includes a record in which FieldA is 'b' and a record in which it is 'c'. 'Is included.

FIG. 7 shows a reading process flow when processing a record in which the index key field 31 has a specific value using a transposition file with a data enumeration type index. Read (Step S2
0), check whether the value of the index key field to be read is enumerated (step S21), and if enumerated, read the block of the field required for processing from the group
(Step S22), it is checked whether the group is the last group on the data enumeration index (Step S23). If it is the last group, the process is terminated. If it is not the last group, the process of reading the data enumeration type index of the next group (step S20)
Repeat from If step S21 is NO, that is, if no data is listed, step S22 is skipped and the process is continued from step S23. This processing flow is for the case where the index target areas are grouped.

When reading a specific field of a record in which the index key field has a specific value, the specific value is compared with the value listed in the data enumeration index, and the value is read from the group including the value. Only blocks of specific fields required for processing need to be read, the amount of reading can be reduced, and the reading time can be shortened, so that processing can be made more efficient. Also, there is an effect that processing can be performed with a small input / output buffer. Embodiments 1 and 2 both use an index indicating whether or not a record having a specific value in an index key field exists in a group of transposed files or an index target area obtained by combining or dividing groups. This is a method of searching for an inverted file. More generally, these modified examples include, for each value of the index key field, a value that exists in each index target area obtained by combining or dividing the group of the transposed file or the group, or a value that does not exist or exists. It is possible to generate an index that provides a means for determining a value that is likely to be present and a value that is not likely to exist, and implement the method as a search method of an inverted file using the index. As a means for the determination, for example, a method of holding a value obtained by converting each value of the index key field according to a certain conversion rule, instead of holding the value of the index key field itself, can be considered. In FIG. 8, a hash function (Module 5) is applied to each value of the index key field to obtain a hash value.
A method of holding a hash value for each index target area is shown. By using such an index, when reading a specific field of a record where the index key field has a specific value, the data area that needs to be read can be narrowed down, and the reading time can be shortened. Can be more efficient. Also, there is an effect that processing can be performed with a small input / output buffer. Embodiment 3 FIG. FIG. 9 shows an inverted hash file generation method according to the third embodiment of the present invention. The transposed hash file is a transposed hash file index unit 32 composed of a packet directory storing a hash function, each hash value and data area information for storing data having the hash value, and stores records corresponding to the respective hash values. And a transposed hash file data section 34 comprising a data area to be transferred. This figure shows a case where the data area and the group 5 correspond one-to-one. When adding a record, index key field 3 of input record 3
The value of 1 (FieldA) is multiplied by a hash function to obtain a hash value, a pointer to a data area 33 corresponding to the hash value is obtained from the packet directory, and a data area for storing the record, that is, a group 5 is obtained, and the record is obtained. Transpose and add sequentially and store. In the created transposed hash file, records having a hash value corresponding to each group 5 are transposed and stored in block 4 units.

FIG. 10 shows that the data area and the group 5 have a 1: N ratio.
It is a schematic diagram in the case corresponding to (N> = 1). When adding a new record to the transposed hash file, if the data area group to which the record is to be added is full and the record cannot be added, a new area for the group is secured and the data area information in the packet directory is added. The newly secured group information is added and transposed and stored at the head of the newly secured group. In this example, the area for storing the second group is secured because the area of hash value = 0 is full. When there are a plurality of groups corresponding to one hash value, only the information (pointer) of the first group is stored in the packet directory, a header area is provided in each group, and the information (pointer) of the next group is stored in the header area. May be.

FIG. 11 shows a reading processing flow when processing a record in which the index key field 31 has a specific value using the transposed hash file. A hash value is obtained from the value of the index key field to be read using a hash function (step S1), and packet directory information for obtaining a data area pointer is obtained from the hash value.
(Step S2), the group information is obtained from the packet directory (Step S3), and the block 4 of the field 2 necessary for processing is read from the group 5 (Step S2).
4), it is checked whether the group is the last group of the packet directory obtained in S2 (step S5), and if it is the last, the process is terminated. If it is not the last group, the next group information is obtained (step S3), and the process is repeated. This flow is for storing a plurality of group information in the packet directory.

FIG. 12 is a schematic diagram for reading data from an inverted hash file. In the transposed hash file,
When reading a specific field of a record in which an index key field has a specific value, a data area including the record is obtained from a hash value (for example, a hash value of 1) obtained by using a hash function from the specific value. Identify and specify specific fields (eg,
Since only the blocks of (Field1 and FieldN) can be read, the input / output buffer can be saved and the read processing can be made more efficient. Fourth Embodiment FIG. 13 shows an inverted range file according to a fourth embodiment of the present invention. The transposed range file is a transposed range file index unit 32 composed of a key range determining unit and a key range directory storing each key range and data area information storing data having the value of the key range. Transposed range file data section 34 consisting of a data area for storing the corresponding records
Consists of This figure shows a case where the data area and the group 5 correspond one-to-one. At the time of record addition, a key range including the value of the index key field 31 (FieldA) of the input record 3 is specified, a pointer to the corresponding data area 33 is obtained from the key range directory, and a data area for storing the record, That is, the records are transposed for the group 5 and sequentially added and stored. In the created transposition range file, records having the corresponding key ranges are transposed and stored in units of blocks 4 in each group 5.

If the data area corresponding to the key range overflows at the time of adding a record, a new area for the group is secured, the newly secured group information is added to the key range directory, and the top of the newly secured group is added. Transpose from and add. In this case, the data area and the group correspond to 1: N (N> = 1). When there are a plurality of groups corresponding to one key range, only the information (pointer) of the first group is stored in the key range directory,
A header area may be provided in each group, and information (pointer) of the next group may be stored in the header area.

FIG. 14 shows a reading processing flow when processing a record in which the index key field 31 has a specific value using the key range file. In order to read a specific field of the record where the index key field has a specific value, a key range is obtained from the specific value of the index key field using a range determination unit (step S41).
Key range directory information is obtained from the key range (step S42), group information is obtained from the key range directory (step S43), and block 4 of field 2 necessary for processing is read from group 5 (step S44). It is checked whether the group is the last group in the key range directory obtained in S42 (step S45). If it is the last group, the process is terminated. If it is not the last group, the next group information is obtained (step S43), and the process is repeated.

If a corresponding key range does not exist when a record is added, a new key range is added. To add a key range, secure a new data area (group), secure an area for a new key range in the key range directory, and add information on the key range to be added and the newly secured data area. And do it.

In the inverted range file, when reading a specific field of a record whose index key field has a specific value, the record is included from the key range value obtained by using the range determination unit from the specific value. Since the data area can be specified and only the block of the specific field in the data area can be read, the input / output buffer can be saved and the read processing can be made more efficient.

In both the third and fourth embodiments, records are classified by the value of the index key field, and the records are transposed and stored separately for each classification in a data area to be stored. . These variants include:
More generally, it is a data storage method including an index unit that provides a unit for classifying records by the value of an index key field, and a data unit including a data area in which data is transposed and stored for each classification. For example, as a means for classifying records, there is a method in which the values of the index key fields are classified in advance, and the records are classified and stored according to the classification. FIG. 15 shows that the index key fields are the names of prefectures, the index key fields are classified as in Hokkaido / Tohoku region, Kanto region, Chubu region, etc., a data storage area is prepared for each classification, and records are stored. This is a case where the information is stored for each region to which the prefecture belongs.

By adopting such a data storage method,
When reading a specific field of a record whose index key field has a specific value, use the index part to specify the data area that contains the record from the specific value, and only block the specific field in that data area. Since reading can be performed, input / output buffers can be saved and the reading process can be performed more efficiently. Embodiment 5 FIG. FIG. 16 shows a method of generating a transposed B + Tree according to the fifth embodiment of the present invention. Transpose B + Tr
ee is composed of an area of the index section 32 and an area of the data section 34. The index unit is composed of one root node and zero or more intermediate nodes. The data section is composed of one or more leaf nodes. Only the key value and the pointer to the next node are stored in the root node and the intermediate node, and all the fields including the key value are stored in the leaf node. For a positive integer d, the d-order transpose B + Tree is as follows: (a) The height from the root node to the leaf node is the same. (B) In the root node, i (1 <= i <= 2d) key values are stored in order. (C) In the intermediate node, i (d <= i <= 2d) key values are stored in order. Storing (d) In the leaf node, i records (d <= i <= 2d) are transposed and stored. (E) A non-leaf node having i key values Key (1) to Key (i) has pointers P (1) to P (i + 1) to i + 1 child nodes, and a pointer P (j) The key value PK (j) included in the child node pointed to by (1 <= j <= i + 1) satisfies the following. (1) PK (1) <Key (1) (2) Key (i) <= PK (i + 1) (3) Other than the above, Key (j) <PK (j) <= Key (j + 1) )
(1 <j <i + 1) However, if Key (j) = Key (j + 1), PK (j) = Ke
y (j) (f) If necessary, a header area is secured in the leaf node, and a one-way or two-way chain between the leaf nodes having the same value is stored. FIG. 16 shows a transposed B + Tree of order INT [L / 2] (L is the number of records forming a block).
Here, INT [X] represents the largest integer not exceeding X. When adding a record, index key field 3 of input record 3
Set the value of 1 (FieldA) to the key value of the root node (5 in this example)
Compare with 6). According to the construction rule (e), if the key value is greater than or equal to the key value of the node, the process moves to the intermediate node indicated by the right pointer, and if smaller, the process moves to the intermediate node indicated by the left pointer. Thereafter, the comparison with the key value in the node is repeated in order to obtain the child node based on the above configuration rule (e), and finally, the storage position of the leaf node (leaf page), that is, the data area 33 is determined. This figure shows the data area
This is the case where 33 and the group correspond one to one. If there is a vacancy in the group (S in this case) in the obtained data area, the record is transposed to the end of the group, added, and the processing ends. In an inverted file, data is read in units of blocks, so that it is not necessary to arrange data in ascending or descending order in the data area. When the group in the data area is full, node division occurs according to the above configuration rule (e). That is, an increasingly new group T is prepared. Transpose INT [(L + 1) / 2] records in ascending order of key value, out of L + 1 records obtained by combining the L records (units that form the block) and the inserted records of group S that were found. Transpose the rest into S and store in group T. Further, the minimum value of the index key field of the record in T is paired with the pointer to T and inserted into the intermediate node pointing to S.

FIG. 17 shows a read processing flow when processing a record in which the index key field 31 has a specific value using the transposed B + Tree. First, the value of the index key field to be read is determined. The key value is compared with the key value of the root node (step S31), and the next node is obtained according to the above configuration rule (e). It is checked whether the acquired next node is a leaf node (step S32). If it is a leaf node, a necessary block is read from a group of leaf nodes (step S3).
4) In order to check if there is any other group corresponding to the key value, the parent node is checked again (step S35), and if not, the process ends. If there is, the necessary block is read from the next group of leaf nodes, and the processing from step S35 is repeated. If the next node is not a leaf node in step S32, an intermediate node is read and the next node is acquired in accordance with the above configuration rule (e) (step S33), the next node is read, and the processing from step S32 is repeated.

Similarly to the transposed hash file, the transposed B + Tr
Even in ee, when reading a specific field of a record whose index key field has a specific value, the data area containing the record having that value is specified using the index part from the value of the index key field, and the data area Since only a block of a specific field in the area can be read, the input / output buffer can be saved and the read processing can be made more efficient.

As a modification of the present embodiment, more generally, an index unit having a hierarchical structure (tree structure) and providing means for classifying records by the value of an index key field, and data for each classification are provided. This is a data storage method including a transposed and stored data area. FIG. 18 shows two different hash functions of “Hash function 1” as a first hash function and “Hash function 2” as a second hash function as index parts having a hierarchical structure (tree structure). Is used. Also, the address is used as an index key field, and the index is divided into three layers. The highest level is classified into regions (Kanto, Chubu, etc.), the middle level is classified into prefectures, and the lowest level is classified as municipalities. There is also a method of classifying records according to municipalities and transposing and storing the records.

By adopting such a data storage method,
When reading a specific field of a record in which an index key field has a specific value, a data area including a record is specified by using the index part hierarchically from the specific value, and a specific field in the data area is specified. Can read only the blocks, so that I / O buffers can be saved and the read processing can be made more efficient. Embodiment 6 FIG. FIG. 19 shows an inverted file with a hash index according to the sixth embodiment of the present invention. The sixth embodiment is also a secondary index like the first embodiment, in which a hash index is added later to an inverted file generated by any method. The index unit includes a hash function, a packet directory storing a hash value and a pointer to a record pointer array, and a record pointer array of each hash value. The record pointer array stores the record ID of the record corresponding to each hash value. Each record ID is composed of a group (group number / intra-group offset). The hash index reads a block of fields for generating an index sequentially from each group, obtains a hash value by using the value and a hash function for each field value in the block, and obtains a packet directory for the obtained hash value. From there, obtain the address of the record pointer array for that hash value, add the record ID (group number / offset within the group) to that address, create it, and perform this operation for the corresponding blocks in all groups. create.

When a record having a specific value in the index key field is read by adding a hash index to the transposed file, a hash value is obtained from the specific value using a hash function, and a corresponding packet is read from the hash value. Gets the record pointer array of the directory and records (or corresponding groups) listed there
Only reading is necessary, and the reading time can be shortened to improve the processing efficiency.

In the sixth embodiment, the record ID is
(Group number / offset within group)
Since the read unit is a block, it is possible to improve the processing efficiency by using only the group number without describing the intra-group offset.

When a hash index is added to a normal file, all fields in all data areas must be read. In the case of an inverted file, a hash index can be obtained by reading only a block of a field to be indexed. Since the index can be created, there is an effect that the index generation time can be reduced. Embodiment 7 FIG. FIG. 20 shows a transposed file with a B + Tree index according to the seventh embodiment of the present invention. This Embodiment 7 is also a secondary index like Embodiments 1 and 6, and B + Tr is added to an inverted file generated by some method later.
An ee index is added. The index unit is composed of one root node, zero or more intermediate nodes, and one or more leaf nodes. The root node and the intermediate nodes store a key value and a pointer to the next node. Stores a pointer to a record pointer array that stores the value and the record pointer ID of the key value. The record pointer array stores the record ID of the record corresponding to each key value. The record ID is composed of a group (group number / intra-group offset).

With respect to the positive integer d, the construction rule of the d-th order B + Tree indexed transposed file is the same as that of the fifth embodiment except for the rule (d) of the transposed B + Tree leaf node. The configuration rules for leaf nodes are as follows. (D ′) In the leaf node, a record pointer array storing i (d <= i <= 2d) record pointers is stored. The B + Tree index reads sequentially the blocks of the field for which the index is generated from each group, compares the value of the field in the block with the key value of the root node, and compares the value with the key value of the root node. ) To get the next node. If the next node is not a leaf node, repeat until the node becomes a leaf node. When the leaf node corresponding to the key value is obtained, a record pointer array is obtained from the leaf node, and a record ID (group number / offset within the area) is added thereto. When the number of records stored in the record pointer array becomes full (2d), node division occurs according to the configuration rules.

By adding a B + Tree index to the transposed file, when reading out a record whose index key field has a specific value, the specific value is compared with the key value of the root node to acquire the next node. According to the configuration rule (e) of the fifth embodiment, the comparison is repeated until the next node becomes a leaf node, a corresponding record pointer array is obtained from the leaf node, and only the groups listed therein need to be read. Processing time can be shortened and processing efficiency can be improved.

In the seventh embodiment, the record ID is
(Group number / offset within group)
Since the read unit is a block, it is possible to shorten the read time by using only the group number without describing the intra-group offset.

In both the sixth embodiment and the seventh embodiment, an index that provides a means for specifying a location where a record is stored is generated for each of the values of the index key fields for the existing transposed file later. Then, data is searched using the index. FIG. 21 shows a method of creating a bitmap of the value of the index key field for each record ID as means for specifying the location where a record is stored as a modification of the above. Also, B + Tr
There is also a method using a tree-structured index other than ee.

By adding these indexes to the transposed file, when reading a record having a specific value in the index key field, the location where the record is stored is specified using the index from the specific value, and the index is used. Only the block of the specific field of the group including the record needs to be read, so that the reading time can be reduced and the processing can be made more efficient.

As a modification of all the embodiments described above, a combination thereof can be considered. FIG. 22 shows a combination of the transposed hash file and the bitmap index shown in the third embodiment. A data area in which a record whose index key field has a specific value exists is determined based on the hash file. Indicates that the value of the index key field is provided in a bitmap for each record of. As a result, it is possible to efficiently perform random access to a record whose index key field has a specific value.

FIG. 23 shows a combination of the bitmap index and the B + Tree index shown in the first embodiment, and is used to search for a record whose index key field has a specific value.
By using the B + Tree index for the part where the bitmap of the specific value is obtained, the bitmap can be obtained more efficiently and the processing can be made more efficient.

[0071]

Since the present invention is configured as described above, it has the following effects.

Since a predetermined fixed number of records classified by the index unit for setting a pointer indicating an area for storing records is configured as a transposed file, a data file with high input / output efficiency and high search efficiency at the time of retrieval can be obtained. Obtainable.

Further, a first index section in which the addresses of the arrays are stored based on the values of the fields of the transposed file, and a plurality of arrays storing the record IDs classified in each of the addresses are stored. And a second index unit made up of the above, it is possible to obtain a data file with a variable search method and high search efficiency.

Further, a table in which a flag is set in a group including a key value of a preset field of the record of the transposed file is generated, and when a record corresponding to the preset key value is searched, the key value is used. Since a search is made for records in the above-mentioned group in which the corresponding flag is set, it is possible to reduce the time required for reading by reading only the fields necessary for the search.

Further, since the key value is constituted by the hash value of the key, a search method with high search efficiency can be obtained.

Furthermore, since the table is configured to set a flag for the group after the group is divided, a search method with high search efficiency can be obtained.

Further, since a fixed number is set for each table, a search method with high search efficiency can be obtained.

Further, the table is composed of a first table of preset fields and a second table of preset fields.
Since the group records are searched under the ND condition, a search method with high search efficiency can be obtained.

Further, a table is generated by listing field values that can be taken for each group of preset fields of the record, and the record of the group corresponding to the above-mentioned field value is searched. A high search method can be obtained.

Further, a pointer to a data area corresponding to the hash value is obtained from the packet directory,
Since the record of the group indicated by the pointer is configured to be searched, a search method with high search efficiency can be obtained.

Further, since the key value classified into the hash value obtained by applying the key value of the field to the first hash function is applied to the second hash function to obtain the hash value, the retrieval efficiency is improved. You can get a search method.

Further, since the record of the group indicated by the pointer is configured to be searched based on a table in which a flag is set for each record of the group including a key value of a field set in advance, the record of the group is searched. A highly efficient search method can be obtained.

Further, since a pointer to the data area corresponding to the classification is obtained from the key classification directory of the transposed file and the record of the group indicated by the pointer is searched, a search method with high search efficiency can be obtained. Can be.

Furthermore, when all the data areas of the group have been stored, a new data area for one group is continuously set, so that a search method with high search efficiency can be obtained.

Also, since the record of the group indicated by the pointer of the intermediate node located at the lowest position of the intermediate node in the balance tree structure is searched, a search method with high search efficiency can be obtained.

Further, when all the data areas of the group have been stored, the intermediate node or the root node is divided according to the configuration rule, a new pointer is set, and the data area for one group indicated by the pointer is set. Therefore, a search method with high search efficiency can be obtained.

The hash value is obtained by applying the key value of the field of the transposed file to the hash function, and the record area corresponding to the hash value is transferred from the record pointer array storing the record ID of the record corresponding to the hash value to the data area corresponding to the hash value. Is obtained and the record of the group indicated by the pointer is searched, so that a search method with high search efficiency can be obtained.

Further, a pointer corresponding to each key value of the leaf node of the index of the balance tree structure in the transposed file is obtained, and a record is searched by the record ID corresponding to the key value in the record pointer array indicated by the pointer. With such a configuration, a search method with high search efficiency can be obtained.

In the transposed file, a table in which a flag is set for a record including a key value of a field of a record is generated for each group, and when a record corresponding to a preset key value is searched, the key value is used. Since the record in which the flag corresponding to is set is searched for each group, a search method with high search efficiency can be obtained.

Further, in the transposed file, a pointer corresponding to each key value of the leaf node of the balance tree structure is obtained, and a record of a group in which a flag corresponding to the key value indicated by the pointer is set is searched. With this configuration, a search method with high search efficiency can be obtained.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a transposed file with a bitmap index according to the first embodiment of the present invention.

FIG. 2 is a flowchart of a reading process of an inverted file with a bitmap index according to the first embodiment of the present invention;

FIG. 3 is a configuration diagram showing a bitmap index created by dividing a group according to the first embodiment of the present invention;

FIG. 4 is a schematic diagram of reading a bitmap index created by dividing a group according to the first embodiment of the present invention;

FIG. 5 is a schematic diagram showing a combination of a plurality of bitmap indexes according to the first embodiment of the present invention.

FIG. 6 is a configuration diagram of a data enumeration type indexed transposed file according to the second embodiment of the present invention.

FIG. 7 is a flowchart of a reading process of a data enumeration type indexed transposed file according to the second embodiment of the present invention;

FIG. 8 is a configuration diagram of a transposition file with a hash value bitmap index showing a modification of the first and second embodiments.

FIG. 9 is a configuration diagram of an inverted hash file according to the third embodiment of the present invention.

FIG. 10 is a configuration diagram of an inverted hash file in which a data area includes a plurality of groups according to the third embodiment of the present invention.

FIG. 11 is a flow of a process of reading an inverted hash file according to the third embodiment of the present invention.

FIG. 12 is a schematic diagram of reading an inverted hash file according to the third embodiment of the present invention.

FIG. 13 is a configuration diagram of an inverted range file according to the fourth embodiment of the present invention.

FIG. 14 is a flowchart of a process of reading an inverted range file according to the fourth embodiment of the present invention.

FIG. 15 is a configuration diagram of a transposition classification file according to a modification of the third and fourth embodiments.

FIG. 16 is a configuration diagram of a transposed B + Tree showing the fifth embodiment of the present invention.

FIG. 17 is a flowchart illustrating a transposed B + Tree reading process according to the fifth embodiment of the present invention;

FIG. 18 is a configuration diagram of a hash tree that is a modification of the fifth embodiment of the present invention.

FIG. 19 is a configuration diagram of an inverted file with a hash index according to the sixth embodiment of the present invention.

FIG. 20 is a configuration diagram of a B + Treer indexed transposed file according to the seventh embodiment of the present invention.

FIG. 21 is a configuration diagram showing a bitmap index for each record, which is a modification of the sixth and seventh embodiments of the present invention.

FIG. 22 is a schematic diagram showing a combination of a hash file and a bitmap index.

FIG. 23 is a schematic diagram showing a combination of a bitmap index and a B + Tree index.

FIG. 24 shows a configuration diagram of a conventional hash file.

FIG. 25 shows a conventional hash file.

FIG. 26 shows a conventional B + Tree.

FIG. 27 shows a conventional bitmap index.

FIG. 28 shows a conventional transposition file.

FIG. 29 is a schematic diagram showing conventional transposition file reading.

FIG. 30 is a configuration diagram using a maximum value / minimum value for a conventional transposition file.

[Explanation of symbols]

 1: Original file 2: Field 3: Record 4: Block 5: Group 6: Inverted file 31: Key value 33: Data area

Claims (19)

[Claims] An index unit for classifying based on field values of a record input from an original file and setting a pointer indicating an area for storing the record for each of the classifications;
A predetermined number of records of the records classified by the index part are divided for each field, stored as blocks in the area indicated by the pointer, and all the divided blocks are connected and reorganized into a group. And a data file comprising an inverted file in which all the records are reorganized into the groups and connected to the groups. 2. A method according to claim 1, wherein a predetermined number of fields in said field are divided from said original file into blocks, and said divided blocks are connected to each other. An inverted file in which the records are reorganized into groups and all of the records are reorganized into the groups after the reorganization into the groups, and an address of an array indicating the storage area of the records for each of the classifications classified based on the values of the fields. A data file comprising: a first index unit storing a plurality of arrays in which record IDs indicating storage locations of records classified by the addresses are stored; . 3. An original file storing a plurality of records composed of a plurality of fields, dividing a predetermined number of predetermined fields in the field from the original file into blocks, and connecting all of the divided blocks. In a transposed file in which the records are reorganized into groups and all the records are reorganized into the groups after the reorganization into the groups, a table in which flags are set for the groups including key values of preset fields of the records is stored. A data search method comprising: generating a record corresponding to a preset key value; and searching a record of the group in which a flag corresponding to the key value is set. 4. The data search method according to claim 3, wherein said key value comprises a hash value of the key. 5. The data search method according to claim 3, wherein the flag is set in the group after the group is divided in the table. 6. The data search method according to claim 3, wherein the fixed number is set for each of the tables. 7. The method according to claim 1, wherein the table is a first table of predetermined fields and a second table of predetermined fields.
The record of the said group is searched by the AND condition of each corresponding key value of the group common to each of said 1st table and said 2nd table. Item 7. The data search method according to any one of Items 6. 8. An original file storing a plurality of records composed of a plurality of fields, dividing a predetermined number of predetermined fields in the field from the original file into blocks, and connecting all of the divided blocks. In a transposed file in which all the records are reorganized into groups and all the records are reorganized into the groups and the groups are connected, the table lists the field values that can be taken for each group of the preset fields of the records, and creates a table. A data search method, comprising: generating and searching a record of a group corresponding to the field value. 9. A hash value is obtained by applying a key value of a predetermined field of a record of a record input from an original file to a hash function, and a predetermined fixed number of records including the field is divided for each field. Stored as a block in the data area corresponding to the hash value, connected all the divided blocks, reorganized them into groups, and reorganized all of the records into the groups after reorganization into the groups. A data search method comprising: obtaining a pointer to a data area corresponding to the hash value from a packet directory in a file; and searching a record of the group indicated by the pointer. 10. A hash value obtained by subjecting a key value of a preset field of a record input from the original file to a hash value obtained by subjecting the key value to a first hash function and subjecting the key value to a second hash function. 10. The data search method according to claim 9, further comprising: 11. The record of the group indicated by the pointer is searched based on a table in which a flag is set for each record of the group including a key value of a preset field. The data search method according to claim 9. 12. A classification of a key value of a preset field of a record input from a source file is determined, and a preset fixed number of records including the field are divided for each field and classified as blocks. Are stored in the data area corresponding to the above, all the divided blocks are connected and reorganized into a group, and all the records are reorganized into the group after the reorganization into the group, and the key classification is performed in the transposed file in which the groups are connected. A data retrieval method, comprising: acquiring a pointer to a data area corresponding to the classification from a directory; and retrieving a record of the group indicated by the pointer. 13. The data search method according to claim 11, wherein when all the data areas of the group have been stored, a new data area for one group is continuously set. . 14. A key value of a preset field of a record input from an original file is compared with a key value of a root node, and is shifted to an intermediate node according to a configuration rule of a balance tree structure, and is sequentially set in the intermediate node. The comparison with the key value and the transition to the next node based on the result are repeated, a predetermined number of records including the above-mentioned fields are divided for each field and stored in the data area of the leaf node as a block, An intermediate node located at the lowest level of the intermediate node in a transposed file in which all of the divided blocks are connected and reorganized into a group, and all the records are reorganized into the group and then connected to the group. A data search method for searching records in the group indicated by the pointer. 15. When all the data areas of the group have been stored, the intermediate node or the root node is divided according to the configuration rule, a new pointer is set, and a data area for one group indicated by the pointer is set. The data search method according to claim 14, wherein the data search is performed. 16. An original file storing a plurality of records composed of a plurality of fields, a predetermined number of the fields are divided from the original file into blocks, and all of the divided blocks are connected. After reorganizing into groups and reorganizing all of the records into the above groups and linking those groups, a hash function is applied to the key values of the above fields to obtain a hash value, and a record corresponding to the hash value is obtained. Acquiring a pointer to a data area corresponding to the hash value from a record pointer array in which the record ID is stored, and searching for a record of the group indicated by the pointer. 17. A pre-determined fixed number of the above fields are divided into blocks from an original file storing a plurality of records composed of a plurality of fields from the original file, and all the divided blocks are connected. In the transposed file in which the records are reorganized into groups and all the records are reorganized into the groups and the groups are connected, pointers corresponding to the respective key values of the leaf nodes of the index of the balance tree structure are obtained, and the pointers are obtained. A data search method for searching for the record by a record ID corresponding to the key value of the record pointer array indicated by the following. 18. An original file storing a plurality of records each consisting of a plurality of fields, a predetermined number of records in the field are divided from the original file into blocks, and all of the divided blocks are connected. In a transposed file in which all the records are reorganized into groups and the groups are linked after reorganization into the groups, a table in which flags are set for the records including the key values of the fields of the records is set for each of the groups. A data search method comprising: generating a record corresponding to a preset key value; and searching for the record in which a flag corresponding to the key value is set for each of the groups. 19. An original file storing a plurality of records composed of a plurality of fields, dividing a predetermined number of predetermined fields in the field from the original file into blocks, and connecting all of the divided blocks. In the transposed file in which the records are reorganized into groups and all the records are reorganized into the groups and the groups are connected, the pointers corresponding to the respective key values of the leaf nodes of the balance tree structure are obtained and indicated by the pointers. A data search method comprising: searching for the record in the group in which a flag corresponding to a key value is set.