US20010011321A1 - Information storage and retrieval system - Google Patents
Information storage and retrieval system Download PDFInfo
- Publication number
- US20010011321A1 US20010011321A1 US09/254,437 US25443799A US2001011321A1 US 20010011321 A1 US20010011321 A1 US 20010011321A1 US 25443799 A US25443799 A US 25443799A US 2001011321 A1 US2001011321 A1 US 2001011321A1
- Authority
- US
- United States
- Prior art keywords
- key
- alternate
- block
- blocks
- primary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9017—Indexing; Data structures therefor; Storage structures using directory or table look-up
Definitions
- the invention relates to the field of information storage and retrieval, and particularly to the storage and reading of data with computers and also particularly to data management in that it provides data storage and retrieval at high speeds, high ratios of data to storage capacity and greatly reduced maintenance time.
- a semiconductor may be used as the primary memory device or may be provided to the invention as an external memory device.
- FIG. 4 illustrates the structure of a block.
- the FROM and TO segments in FIG. 4 represent the minimum key value and maximum key value of the block, respectively, but neither is absolutely necessary and the invention may be implemented with either one alone.
- a single record may be matched to a single block, and a format in which a single record is stored over two or more blocks (spanned records) is also possible. All primary blocks in a single file and all overflow blocks in a single file are of equal lengths, facilitating region management.
- Blocks are classified into primary blocks and overflow blocks, and only the primary blocks are managed in the location table. Records are stored first of all in primary blocks. If storage in a block is rendered impossible due to insertion of a record, a single overflow block is allocated to that block. If that overflow block is not sufficient for storage, one more overflow block is allocated. Overflow blocks are managed as dependent blocks of their primary block; they are not managed in the location table but only by pointing from the primary block.
- a deadlock occurs when two different tasks on a single computer perform exclusion operations on two or more identical resources in different sequences but does not occur if the sequence of exclusion is identical, and narrower ranges of exclusion reduce the probability of deadlock.
- Conventional indices are made up of multiple levels. When a record is accessed and a record's key is modified when an index is updated, the index is also modified. When this affects the lowest-level index alone, the exclusion range is restricted, but in some cases this may affect higher-level indices. Since exclusion affects a range of multiple records and it takes a great deal of time to update the indices in such cases, exclusion takes up long periods of time and is a cause of multiple instances of deadlock.
- FIG. 1 illustrates the structures of a location table and a final pointer, either or both of which may be implemented with different lengths, the number of bytes given for each being for illustrative purposes only.
- FIG. 2 illustrates the relationship between a location table and its blocks.
- FIG. 3 illustrates the structures of an alternate-key table (and its entries) and an alternate-key block, either or both of which may be implemented with different lengths, the number of bytes given for each being for illustrative purposes only, in which the FROM and TO segments hold the minimum and maximum key values, respectively, of the block and the alternate-key block may hold both FROM and TO values or it may hold only one of these.
- FIG. 4 illustrates the structure of a block.
- FIG. 5 illustrates the structure of a pre-alternate-key block.
- FIG. 6 illustrates the relationship between primary blocks and overflow blocks in which the blocks occupy a contiguous region in order to facilitate understanding, but there being no requirement that they do so in practice.
- FIG. 7 illustrates an example of keys applied to text data.
- Records stored with this system are restricted to records having a single unique primary key and none or one or more non-unique (though these may be unique) keys (alternate keys).
- the system does not accommodate records that do not have a unique key.
- a serial number or other unique key may be forcibly applied to a record not having a unique key when such a record is added and reading performed only in the physical order of records or in the order of their alternate keys. Since insertion is not thus involved, an overflow block is required only when updating a record increases the length of the record.
- dition refers to storage of a record having a primary key larger than the largest primary key of records currently stored
- insertion refers to storage of a record having a primary key smaller than the largest primary key of records currently stored
- the size of a location table is calculated from the number of records planned to be stored, the size of their blocks, and the number of primary-key blocks per record in the location table, and a contiguous region sufficient for this size is secured in a storage medium.
- the size and number of alternate-key blocks is determined and a contiguous region sufficient for this size secured so as to enable storage of entries in the number of records also storing alternate-key blocks.
- an additional contiguous storage region is secured and an address-conversion table used to treat multiple contiguous regions as though they were one contiguous region, thus allowing the system to accommodate situations in which the number of records stored exceeds the number originally assumed.
- FIG. 2 illustrates the relationship between a location table and its blocks.
- a location table is created for each sub-range in a size suited to the number of records intended to be stored in the sub-range.
- Each location table must be in a contiguous region, but all location tables need not be in a single contiguous region.
- An alternate-key block is set up not split across regions, but in a contiguous region of a size corresponding to all records.
- a final pointer When storing the first record, first a final pointer is referenced in exclusion mode.
- the final pointer controls how far the block and location table are used and so has the format illustrated in FIG. 1. Since there are no stored records to begin with, the first block is registered in the final pointer as the final block. The block number and primary-key value are registered in the final pointer.
- the first primary block is secured in exclusion mode, and its physical address and block number ( 0 in this case since numbering starts from 0) are registered.
- a block is secured on disk, its entry includes the value of its primary key.
- records are registered in that block. Then all exclusion is released.
- the final pointer is referenced in exclusion mode and it is determined whether the primary key is greater than the value of the key in the final pointer.
- the block number is 0, block number 0 in the location table is referenced in exclusion mode, the physical location of block 0 obtained and block 0 at that physical location read. If there is sufficient space in that block, the record is stored, the value of the primary key registered in the final pointer and all exclusion released.
- a binary search is one example of a high-speed search method. Though a binary search method is used here as an example, other methods may also be used to find a target entry. The method used here is to find dichotomous points and compare the value of the primary key of the record stored in that block (abbreviated below as “stored primary-key value”), including the overflow block if the primary block that record points to has an overflow block, with the value of the primary key of the record inserted (abbreviated below as “inserted primary-key value”).
- the inserted primary-key value is greater than the smallest of the stored primary-key values of the block or blocks and smaller than the smallest of the stored primary-key values in the next block, it is stored in the first of those two blocks. Otherwise, the sizes of the stored primary-key values of that block and the insertion-block primary-key value are again compared, dichotomous points are obtained in the former if the inserted primary-key value is smaller and in the latter if the inserted primary-key value is not smaller, and like operations performed to identify the block to be used for storing the record.
- the location table is composed of multiple contiguous regions, a binary search cannot be performed as is, but using an address-conversion table to treat it as though it were a single contiguous region allows a binary search to be performed.
- the location table is split across sub-ranges, first a comparison is made with the value of either the initial or the final primary key of the location table in each sub-range to determine which sub-range holds the target record. A binary search is then performed on that sub-range in like manner as described above to identify the block that holds the target record.
- the location within the block where the record is to be inserted is identified. Since records are arranged in the order of their primary keys, the location of the insertion will be that location immediately prior to the record having a primary key larger than the record being inserted. First, a check is performed for duplication of the primary keys. If there is any duplication of primary keys, an error is output since the storage cannot be performed. If there is no duplication of primary keys, one or more records located behind the record being inserted are moved rearwards to create a space exactly equal to the size of the record being inserted.
- FIG. 6 illustrates the logical relationship between primary blocks and overflow blocks.
- an overflow block already exists, it is to be preferred for it to be possible to store the record combining a primary block and its overflow block. There is also a possibility that only a part of an overflow block is used and the region not used efficiently. In order to avoid such, a single overflow block may be provided for multiple primary blocks. It is also possible to provide all overflow blocks in an identical size smaller than that of their primary block. It is also possible use a single overflow block by pointing to it from multiple primary blocks.
- Another method that may be adopted is to store a record that overflows in an independent storage region and provide a pointer to it from its primary block.
- this method results in a large number of overflow records, it entails the disadvantage of long retrieval times compared with a method using overflow blocks.
- the storage method selected should be that suited to the number of records that are to be generated.
- Alternate-key tables are stored in an alternate-key block in the order of their alternate keys.
- Entries in an alternate-key table consist of an alternate key, the physical address of the block where the record of that key value is stored, and the primary key of the record of that key value.
- the number of entries in an alternate-key table changes when its records are added or updated, but there is a high possibility that an increase in the number of entries will result in entry insertion and a very low possibility that such increase will result in entry addition. Therefore, the management methods used for primary keys are not appropriate.
- insertions can be processed efficiently by providing a space of defined size when storing an entry in an alternate-key block, but if the initial number of records is less than the final number of stored records planned, key insertions will result in a multiple overflow alternate-key blocks.
- a pre-alternate-key block is used in such cases.
- a pre-alternate-key block has the same structure as an alternate-key block, and the number of pre-alternate-key blocks is the size that can accommodate entries in the number of alternate-key blocks divided by the size of the pre-alternate-key blocks.
- the number of entries in a pre-alternate-key block becomes equal to the number of alternate-key blocks, these entries are moved from the pre-alternate-key block to an alternate-key block.
- entries having an identical alternate key are stored in the same alternate-key block. If the number of entries having an identical alternate key is too large to store them in the corresponding alternate-key block, they are stored in an added alternate-key overflow block.
- pre-alternate-key blocks are provided. Taking the example described above, 10,000 entries will be stored in a pre-alternate-key block, but this pre-alternate-key block is provided two levels since 100 entries can be stored in a single block, 100 entries are managed in the first pre-alternate-key block and when the number of entries in that first pre-alternate-key block reaches 100, they are transferred to the second-level pre-alternate-key block.
- FIG. 5 illustrates an example of such a transfer in which a two-level pre-alternate-key block is provided.
- the target primary-key value is greater than the smallest of the stored primary-key values—and smaller than the smallest of the stored primary-key values in the next block, either the target record exists in that block or the record of that key value does not exist in the file. Since records in a block are arranged in the order of their primary keys, searching the block can detect the target record or confirm whether the record does not exist in the file. Otherwise, the sizes of the stored primary-key values of that block and the primary-key value of the target record are again compared, dichotomous points are obtained in the former if the target primary-key value is smaller and in the latter if the target primary-key value is not smaller, and like operations performed to identify the block storing the record.
- the physical block is accessed from the physical block number in that entry and the record in that block identified that matches the primary-key value in the entry. And since alternate keys may be non-unique, the next entry in the alternate-key block is examined. If it has an identical alternate-key value, the record corresponding to that entry is also retrieved, and this operation repeated until no entry has an identical key value.
- Generation may be useful in a variety of situations, among them when multiple records already exist in this system's file and regeneration is required for such reason as an increased number of overflow blocks, when restoring information from a back-up medium to the medium on which this system has been implemented, and when moving records stored by a method other than that described here to this system. Any of these may be effected by the same means.
- files are read in the order of their primary keys with a sequential access method and a sequential file is created.
- a sequential file is similarly created for generation.
- the location tables and alternate-key block are to be prepared.
- the number of location table entries is obtained by dividing the number of records planned to be stored by the number of records that can be stored in a single block, and that amount of space is secured in a contiguous region.
- Alternate-key blocks are secured for each type of alternate key. All alternate-key blocks for a given type of alternate key have the same size, and the number of such blocks is determined thus:
- the number of entries (A) that can be stored in a single alternate-key block is obtained, and the number of records planned to be stored is divided by A to obtain the number of alternate-key blocks.
- Pre-alternate-key blocks are also to be prepared.
- Records are stored in a block in the order of their primary keys.
- the frequency of insertions may be estimated in advance or calculated from statistical data, and a certain proportion of empty space provided within a given block. This also allows for instantaneous storage of records. This is determined by how often insertions are to be performed. The proportion of empty space provided within a block may vary from block to block.
- an entry is written to the location table. If a record has an alternate key, first an alternate-key entry is created from the record and stored in a sequential file. When alternate-key entries have been created for all records, the entries are sorted in the order of the alternate keys and the sorted entries stored in the alternate-key block. The number of records generated is divided by the maximum planned number of records, and storage is performed in the alternate-key block in the proportion corresponding to that quotient. This is because the alternate keys are inserted rather than added.
- An alternate key may also be put to special use. Since keys are currently assigned to fields stored in a specific location in a record, they have an identical length and format, such as a product code or a customer code. Since keys and records may be made relational in this system, keys may be created for text data. Keys may also be created for records of nonspecific format, as illustrated in FIG. 7. As in this example, alternate keys may be assigned to records not having fields of uniform location or fixed field lengths.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- 1. Field of the Invention
- The invention relates to the field of information storage and retrieval, and particularly to the storage and reading of data with computers and also particularly to data management in that it provides data storage and retrieval at high speeds, high ratios of data to storage capacity and greatly reduced maintenance time.
- 2. Description of Related Art
- Conventional information storage systems are subject to numerous practical constraints such as contiguity in the physical location of blocks and the requirement that storage blocks be created in advance. Moreover, in most systems random access requires the creation of indices, which take a long time to generate, and the structure of these systems makes them prone to deadlock since the indices are updated and the range of exclusion broadened when the referent information is modified. Although direct access systems do not use indices, these systems relate record keys with storage locations by means of special programs known as randomizing routines. In practice they do not allow for sequential access and exhibit low storage efficiency when compared with methods that use indices.
- Since conventional information storage and retrieval systems presumed their data would be stored on magnetic disk, in certain respects their shortcomings are unavoidable. On the other hand, the price of semiconductors has fallen remarkably and conditions for using semiconductors as memory devices are falling into place. Semiconductors do not necessitate physical movement or rotation and enable high-speed storage and reading even if addresses are not contiguous. The construction of a memory device taking advantage of these properties and comprising semiconductors alone or semiconductors and random access memory media would enable high-speed storage and retrieval processing. A semiconductor may be used as the primary memory device or may be provided to the invention as an external memory device. Storing information sequentially in blocks, using overflow blocks for storage when the insertion of information causes an overflow, using location tables or alternate-key blocks that manage storage blocks rather than using indices, and performing retrievals from these location tables or alternate-key blocks enables high-speed storage and reading, improves the efficiency of information storage and minimizes the occurrence of deadlock.
- Location tables and alternate-key tables are introduced in place of indices. Records are not stored independently; rather, multiple records are stored in a single block. FIG. 4 illustrates the structure of a block. The FROM and TO segments in FIG. 4 represent the minimum key value and maximum key value of the block, respectively, but neither is absolutely necessary and the invention may be implemented with either one alone. When using long records, a single record may be matched to a single block, and a format in which a single record is stored over two or more blocks (spanned records) is also possible. All primary blocks in a single file and all overflow blocks in a single file are of equal lengths, facilitating region management. Storage in blocks allows block detection by means of fixed-length operations even when variable-length records are stored because the block units are of a fixed length. The location table secures the required contiguous region in advance. A single record (entry) in a location table manages a single primary block in the data record storage region. Although it is also possible to implement methods of managing multiple primary blocks, since primary blocks may be of any given size it is simpler not to manage multiple blocks if the implementation allows modification of the size of blocks. Blocks are classified into primary blocks and overflow blocks, and only the primary blocks are managed in the location table. Records are stored first of all in primary blocks. If storage in a block is rendered impossible due to insertion of a record, a single overflow block is allocated to that block. If that overflow block is not sufficient for storage, one more overflow block is allocated. Overflow blocks are managed as dependent blocks of their primary block; they are not managed in the location table but only by pointing from the primary block.
- Since overflow blocks are not managed in the location table, records are not inserted into the location table, the time it takes to re-write the location table is minimized and re-writing of the location table is a single record. Therefore, the range of any exclusion that may occur is radically minimized and the possibility of a deadlock is greatly reduced.
- A deadlock occurs when two different tasks on a single computer perform exclusion operations on two or more identical resources in different sequences but does not occur if the sequence of exclusion is identical, and narrower ranges of exclusion reduce the probability of deadlock. Conventional indices are made up of multiple levels. When a record is accessed and a record's key is modified when an index is updated, the index is also modified. When this affects the lowest-level index alone, the exclusion range is restricted, but in some cases this may affect higher-level indices. Since exclusion affects a range of multiple records and it takes a great deal of time to update the indices in such cases, exclusion takes up long periods of time and is a cause of multiple instances of deadlock.
- Accordingly, several objects and advantages of the invention are:
- (a) To provide high-speed random access by using primary keys and alternate keys;
- (b) To provide high-speed sequential read and write operations;
- (c) To provide minimal modification caused by updating, modification and deletion of data, short processing time, and high-speed storage by using location tables and alternate-key blocks instead of indices;
- (d) To provide creation (generation) of location tables and alternate-key blocks in shorter times than are required for the creation of indices;
- (e) To eliminate the necessity of creating blocks in advance;
- (f) To provide the ability to add blocks until no physical space remains on a storage medium;
- (g) To provide record-compression capabilities deriving from the capability of handling variable-length records and record insertion (though in most cases compressed records cannot be updated);
- (h) To provide the capability of using multiple alternate keys; and
- (i) To provide greatly reduced database maintenance time.
- FIG. 1 illustrates the structures of a location table and a final pointer, either or both of which may be implemented with different lengths, the number of bytes given for each being for illustrative purposes only.
- FIG. 2 illustrates the relationship between a location table and its blocks.
- FIG. 3 illustrates the structures of an alternate-key table (and its entries) and an alternate-key block, either or both of which may be implemented with different lengths, the number of bytes given for each being for illustrative purposes only, in which the FROM and TO segments hold the minimum and maximum key values, respectively, of the block and the alternate-key block may hold both FROM and TO values or it may hold only one of these.
- FIG. 4 illustrates the structure of a block.
- FIG. 5 illustrates the structure of a pre-alternate-key block.
- FIG. 6 illustrates the relationship between primary blocks and overflow blocks in which the blocks occupy a contiguous region in order to facilitate understanding, but there being no requirement that they do so in practice.
- FIG. 7 illustrates an example of keys applied to text data.
- Records stored with this system are restricted to records having a single unique primary key and none or one or more non-unique (though these may be unique) keys (alternate keys). The system does not accommodate records that do not have a unique key.
- However, a serial number or other unique key may be forcibly applied to a record not having a unique key when such a record is added and reading performed only in the physical order of records or in the order of their alternate keys. Since insertion is not thus involved, an overflow block is required only when updating a record increases the length of the record.
- In the description following, “addition” refers to storage of a record having a primary key larger than the largest primary key of records currently stored, and “insertion” refers to storage of a record having a primary key smaller than the largest primary key of records currently stored.
- First, we will describe the storage system. The size of a location table is calculated from the number of records planned to be stored, the size of their blocks, and the number of primary-key blocks per record in the location table, and a contiguous region sufficient for this size is secured in a storage medium. Similarly, the size and number of alternate-key blocks is determined and a contiguous region sufficient for this size secured so as to enable storage of entries in the number of records also storing alternate-key blocks. However, in the event storage exceeds the number originally assumed, there is a possibility that the contiguous region may fill up, making further storage impossible. In such cases, an additional contiguous storage region is secured and an address-conversion table used to treat multiple contiguous regions as though they were one contiguous region, thus allowing the system to accommodate situations in which the number of records stored exceeds the number originally assumed.
- When there are multiple alternate keys, a region is secured for each one. There is no need for the alternate-key blocks corresponding to different alternate keys to be contiguous. FIG. 2 illustrates the relationship between a location table and its blocks.
- When using a storage system split into sub-ranges, a location table is created for each sub-range in a size suited to the number of records intended to be stored in the sub-range. Each location table must be in a contiguous region, but all location tables need not be in a single contiguous region. An alternate-key block is set up not split across regions, but in a contiguous region of a size corresponding to all records.
- When storing the first record, first a final pointer is referenced in exclusion mode. The final pointer controls how far the block and location table are used and so has the format illustrated in FIG. 1. Since there are no stored records to begin with, the first block is registered in the final pointer as the final block. The block number and primary-key value are registered in the final pointer. Next, the first primary block is secured in exclusion mode, and its physical address and block number (0 in this case since numbering starts from 0) are registered. When a block is secured on disk, its entry includes the value of its primary key. Next, records are registered in that block. Then all exclusion is released.
- To register the second record, first the final pointer is referenced in exclusion mode and it is determined whether the primary key is greater than the value of the key in the final pointer. First we will describe how a record is added. Since the block number is 0,
block number 0 in the location table is referenced in exclusion mode, the physical location ofblock 0 obtained andblock 0 at that physical location read. If there is sufficient space in that block, the record is stored, the value of the primary key registered in the final pointer and all exclusion released. - Subsequent additions are performed in like fashion. Sometimes the space in
block 0 will not be sufficient to store an additional record. Ifblock 0 lacks sufficient space when the above operation is performed andblock 0 read on the addition of record number m, a single primary block (block number 1) is secured in exclusion mode. Record number m is then stored inblock 1. The second record in the location table is then referenced in exclusion mode, and the physical location ofprimary block 1 registered in this record. All exclusion is then released. In this way additions are stored in the location that physically follows the last record. When split across sub-ranges, the same operations are performed for each sub-range. - Next we describe how a record is inserted. Assume that multiple location-table records, primary blocks and data records already exist.
- It is first of all necessary to determine which block the record inserted should be stored in.
- This is done by searching the location table. A binary search is one example of a high-speed search method. Though a binary search method is used here as an example, other methods may also be used to find a target entry. The method used here is to find dichotomous points and compare the value of the primary key of the record stored in that block (abbreviated below as “stored primary-key value”), including the overflow block if the primary block that record points to has an overflow block, with the value of the primary key of the record inserted (abbreviated below as “inserted primary-key value”). If the inserted primary-key value is greater than the smallest of the stored primary-key values of the block or blocks and smaller than the smallest of the stored primary-key values in the next block, it is stored in the first of those two blocks. Otherwise, the sizes of the stored primary-key values of that block and the insertion-block primary-key value are again compared, dichotomous points are obtained in the former if the inserted primary-key value is smaller and in the latter if the inserted primary-key value is not smaller, and like operations performed to identify the block to be used for storing the record.
- If the location table is composed of multiple contiguous regions, a binary search cannot be performed as is, but using an address-conversion table to treat it as though it were a single contiguous region allows a binary search to be performed.
- If the location table is split across sub-ranges, first a comparison is made with the value of either the initial or the final primary key of the location table in each sub-range to determine which sub-range holds the target record. A binary search is then performed on that sub-range in like manner as described above to identify the block that holds the target record.
- If the record is to be stored in block number n, the following shall apply. The location within the block where the record is to be inserted is identified. Since records are arranged in the order of their primary keys, the location of the insertion will be that location immediately prior to the record having a primary key larger than the record being inserted. First, a check is performed for duplication of the primary keys. If there is any duplication of primary keys, an error is output since the storage cannot be performed. If there is no duplication of primary keys, one or more records located behind the record being inserted are moved rearwards to create a space exactly equal to the size of the record being inserted. If the records thus moved fit within the block, insertion is completed, but if those records do not fit within the block, a single overflow block is provided, a pointer to it provided from the primary block, and as many records stored in the overflow block as necessary. Then the record to be inserted and any that follow it are stored in the primary block. FIG. 6 illustrates the logical relationship between primary blocks and overflow blocks.
- If an overflow block already exists, it is to be preferred for it to be possible to store the record combining a primary block and its overflow block. There is also a possibility that only a part of an overflow block is used and the region not used efficiently. In order to avoid such, a single overflow block may be provided for multiple primary blocks. It is also possible to provide all overflow blocks in an identical size smaller than that of their primary block. It is also possible use a single overflow block by pointing to it from multiple primary blocks.
- Another method that may be adopted is to store a record that overflows in an independent storage region and provide a pointer to it from its primary block. However, if this method results in a large number of overflow records, it entails the disadvantage of long retrieval times compared with a method using overflow blocks. In practice, the storage method selected should be that suited to the number of records that are to be generated.
- Next, we describe storage and updating with alternate keys. Alternate-key tables are stored in an alternate-key block in the order of their alternate keys. Entries in an alternate-key table consist of an alternate key, the physical address of the block where the record of that key value is stored, and the primary key of the record of that key value. The number of entries in an alternate-key table changes when its records are added or updated, but there is a high possibility that an increase in the number of entries will result in entry insertion and a very low possibility that such increase will result in entry addition. Therefore, the management methods used for primary keys are not appropriate. If there are already more stored records than the final number planned, insertions can be processed efficiently by providing a space of defined size when storing an entry in an alternate-key block, but if the initial number of records is less than the final number of stored records planned, key insertions will result in a multiple overflow alternate-key blocks. A pre-alternate-key block is used in such cases.
- A pre-alternate-key block has the same structure as an alternate-key block, and the number of pre-alternate-key blocks is the size that can accommodate entries in the number of alternate-key blocks divided by the size of the pre-alternate-key blocks. When the number of entries in a pre-alternate-key block becomes equal to the number of alternate-key blocks, these entries are moved from the pre-alternate-key block to an alternate-key block. When such entries are moved, in principle a single entry is stored in any given alternate-key block, but entries having an identical alternate key are stored in the same alternate-key block. If the number of entries having an identical alternate key is too large to store them in the corresponding alternate-key block, they are stored in an added alternate-key overflow block.
- For example, assume that the final number of stored records planned is one million records. If 100 entries can be stored in a single alternate-key block, 10,000 alternate-key blocks will be required. The entries are stored in a pre-alternate-key block until there are 10,000 of them, and when the number of entries reaches 10,000, the entries are transferred to an alternate-key block.
- If there is a large number of alternate-key blocks and a single level of pre-alternate-key blocks is provided, there is a possibility that this will result in a large number of pre-alternate-key blocks, a high frequency of insertions and inefficient updating. In such cases, multiple levels of pre-alternate-key blocks are provided. Taking the example described above, 10,000 entries will be stored in a pre-alternate-key block, but this pre-alternate-key block is provided two levels since 100 entries can be stored in a single block, 100 entries are managed in the first pre-alternate-key block and when the number of entries in that first pre-alternate-key block reaches 100, they are transferred to the second-level pre-alternate-key block.
- FIG. 5 illustrates an example of such a transfer in which a two-level pre-alternate-key block is provided.
- Next, we will describe retrieval of records using primary keys. This operation is performed in the same fashion used to determine the insertion location when a record is inserted. The example of retrieval described here uses the same binary-search method used in the example described for insertion. First, dichotomous points are found in the location table, and the value of the primary key of the record stored in that block (abbreviated below as “stored primary-key value”), including the overflow block if the primary block that record points to has an overflow block, is compared with the value of the primary key of the target record (abbreviated below as “target primary-key value”). If the target primary-key value is greater than the smallest of the stored primary-key values—and smaller than the smallest of the stored primary-key values in the next block, either the target record exists in that block or the record of that key value does not exist in the file. Since records in a block are arranged in the order of their primary keys, searching the block can detect the target record or confirm whether the record does not exist in the file. Otherwise, the sizes of the stored primary-key values of that block and the primary-key value of the target record are again compared, dichotomous points are obtained in the former if the target primary-key value is smaller and in the latter if the target primary-key value is not smaller, and like operations performed to identify the block storing the record.
- Next, we will describe retrieval using alternate keys. Retrieval with alternate keys is performed by searching alternate-key blocks. The binary search method is typically used, but the discussion is omitted here since it is described above with respect to retrieval using primary keys. The alternate-key block including the target alternate key is identified. Then the target alternate-key table in the alternate-key block is identified. The result is that, as when using primary keys, either the entry exists in that block or a record with that alternate-key value does not exist in the file and the entry does not exist. If the alternate-key block has any alternate-key overflow blocks, all such blocks are searched.
- If the alternate-key table (or entry) having the target key value is identified, the physical block is accessed from the physical block number in that entry and the record in that block identified that matches the primary-key value in the entry. And since alternate keys may be non-unique, the next entry in the alternate-key block is examined. If it has an identical alternate-key value, the record corresponding to that entry is also retrieved, and this operation repeated until no entry has an identical key value.
- Next, we will describe generation. Generation may be useful in a variety of situations, among them when multiple records already exist in this system's file and regeneration is required for such reason as an increased number of overflow blocks, when restoring information from a back-up medium to the medium on which this system has been implemented, and when moving records stored by a method other than that described here to this system. Any of these may be effected by the same means.
- For regeneration, files are read in the order of their primary keys with a sequential access method and a sequential file is created. A sequential file is similarly created for generation.
- Next, the location tables and alternate-key block are to be prepared. The number of location table entries is obtained by dividing the number of records planned to be stored by the number of records that can be stored in a single block, and that amount of space is secured in a contiguous region. Alternate-key blocks are secured for each type of alternate key. All alternate-key blocks for a given type of alternate key have the same size, and the number of such blocks is determined thus: The number of entries (A) that can be stored in a single alternate-key block is obtained, and the number of records planned to be stored is divided by A to obtain the number of alternate-key blocks. Pre-alternate-key blocks are also to be prepared.
- Records are stored in a block in the order of their primary keys. The frequency of insertions may be estimated in advance or calculated from statistical data, and a certain proportion of empty space provided within a given block. This also allows for instantaneous storage of records. This is determined by how often insertions are to be performed. The proportion of empty space provided within a block may vary from block to block. When a record is stored, an entry is written to the location table. If a record has an alternate key, first an alternate-key entry is created from the record and stored in a sequential file. When alternate-key entries have been created for all records, the entries are sorted in the order of the alternate keys and the sorted entries stored in the alternate-key block. The number of records generated is divided by the maximum planned number of records, and storage is performed in the alternate-key block in the proportion corresponding to that quotient. This is because the alternate keys are inserted rather than added.
- An alternate key may also be put to special use. Since keys are currently assigned to fields stored in a specific location in a record, they have an identical length and format, such as a product code or a customer code. Since keys and records may be made relational in this system, keys may be created for text data. Keys may also be created for records of nonspecific format, as illustrated in FIG. 7. As in this example, alternate keys may be assigned to records not having fields of uniform location or fixed field lengths.
Claims (2)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/119,145 US6584555B2 (en) | 1997-07-11 | 2002-04-10 | Information storage and retrieval system |
US10/390,815 US6654868B2 (en) | 1997-07-11 | 2003-03-17 | Information storage and retrieval system |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP9/219700 | 1997-07-11 | ||
JP9-219700 | 1997-07-11 | ||
JP21970097A JP3345628B2 (en) | 1997-07-11 | 1997-07-11 | Data storage and retrieval methods |
PCT/JP1998/003067 WO1999003039A1 (en) | 1997-07-11 | 1998-07-08 | Data storage/retrieval system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP1998/003067 A-371-Of-International WO1999003039A1 (en) | 1997-07-11 | 1998-07-08 | Data storage/retrieval system |
Related Child Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/119,145 Continuation US6584555B2 (en) | 1997-07-11 | 2002-04-10 | Information storage and retrieval system |
US10/119,145 Division US6584555B2 (en) | 1997-07-11 | 2002-04-10 | Information storage and retrieval system |
US10/390,815 Division US6654868B2 (en) | 1997-07-11 | 2003-03-17 | Information storage and retrieval system |
Publications (2)
Publication Number | Publication Date |
---|---|
US20010011321A1 true US20010011321A1 (en) | 2001-08-02 |
US6415375B2 US6415375B2 (en) | 2002-07-02 |
Family
ID=16739604
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/254,437 Expired - Fee Related US6415375B2 (en) | 1997-07-11 | 1998-07-08 | Information storage and retrieval system |
US10/119,145 Expired - Fee Related US6584555B2 (en) | 1997-07-11 | 2002-04-10 | Information storage and retrieval system |
US10/390,815 Expired - Fee Related US6654868B2 (en) | 1997-07-11 | 2003-03-17 | Information storage and retrieval system |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/119,145 Expired - Fee Related US6584555B2 (en) | 1997-07-11 | 2002-04-10 | Information storage and retrieval system |
US10/390,815 Expired - Fee Related US6654868B2 (en) | 1997-07-11 | 2003-03-17 | Information storage and retrieval system |
Country Status (4)
Country | Link |
---|---|
US (3) | US6415375B2 (en) |
EP (1) | EP0923030A4 (en) |
JP (1) | JP3345628B2 (en) |
WO (1) | WO1999003039A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060129565A1 (en) * | 2002-10-21 | 2006-06-15 | Annex Systems Incorporated | Database accelerator |
US20070078909A1 (en) * | 2004-03-08 | 2007-04-05 | Masaharu Tamatsu | Database System |
US20110320451A1 (en) * | 2010-06-23 | 2011-12-29 | International Business Machines Corporation | Apparatus and method for sorting data |
US20210326271A1 (en) * | 2020-04-18 | 2021-10-21 | International Business Machines Corporation | Stale data recovery using virtual storage metadata |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6444072B1 (en) * | 1999-08-11 | 2002-09-03 | Southpac Trust International | Process for producing holographic material |
AU2001243459A1 (en) | 2000-03-09 | 2001-09-17 | The Web Access, Inc. | Method and apparatus for performing a research task by interchangeably utilizinga multitude of search methodologies |
JP2001356945A (en) * | 2000-04-12 | 2001-12-26 | Anetsukusu Syst Kk | Data backup recovery system |
US6697797B1 (en) * | 2000-04-28 | 2004-02-24 | Micro Data Base Systems, Inc. | Method and apparatus for tracking data in a database, employing last-known location registers |
US8335779B2 (en) * | 2002-08-16 | 2012-12-18 | Gamroe Applications, Llc | Method and apparatus for gathering, categorizing and parameterizing data |
JPWO2004025475A1 (en) * | 2002-09-10 | 2006-01-12 | 玉津 雅晴 | Database reorganization system and database |
JPWO2005103901A1 (en) * | 2004-04-19 | 2008-03-13 | アネックスシステムズ株式会社 | Computer system |
US20060010122A1 (en) * | 2004-07-07 | 2006-01-12 | International Business Machines Corporation | System and method for improved database table record insertion and reporting |
US7965297B2 (en) * | 2006-04-17 | 2011-06-21 | Microsoft Corporation | Perfect hashing of variably-sized data |
US7523288B2 (en) * | 2006-09-06 | 2009-04-21 | Microsoft Corporation | Dynamic fragment mapping |
US7872648B2 (en) * | 2007-06-14 | 2011-01-18 | Microsoft Corporation | Random-access vector graphics |
WO2009144942A1 (en) * | 2008-05-30 | 2009-12-03 | 日本電気株式会社 | Database system, database management method, database structure, and computer program |
TWI428023B (en) * | 2008-11-18 | 2014-02-21 | Ind Tech Res Inst | Decoding method and apparatus |
CN102495836B (en) * | 2011-10-28 | 2013-07-10 | 宁波三星电气股份有限公司 | Searching method for last record of accumulative power consumption |
US9870416B2 (en) | 2014-04-24 | 2018-01-16 | International Business Machines Corporation | Method for rebalancing data partitions |
US10089342B2 (en) * | 2014-07-10 | 2018-10-02 | Sap Se | Main memory database management using page index vectors |
CN105426408B (en) * | 2015-11-02 | 2019-03-08 | 北京锐安科技有限公司 | A kind of data processing method and device of more indexes |
US20170255565A1 (en) * | 2016-03-02 | 2017-09-07 | Intel Corporation | Method and apparatus for providing a contiguously addressable memory region by remapping an address space |
US11782875B2 (en) * | 2017-11-13 | 2023-10-10 | Weka.IO Ltd. | Directory structure for a distributed storage system |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4571700A (en) * | 1983-06-16 | 1986-02-18 | International Business Machines Corporation | Page indexing system for accessing sequentially stored data representing a multi-page document |
US5842224A (en) * | 1989-06-16 | 1998-11-24 | Fenner; Peter R. | Method and apparatus for source filtering data packets between networks of differing media |
US5490258A (en) * | 1991-07-29 | 1996-02-06 | Fenner; Peter R. | Associative memory for very large key spaces |
US5860136A (en) * | 1989-06-16 | 1999-01-12 | Fenner; Peter R. | Method and apparatus for use of associated memory with large key spaces |
EP0410210A3 (en) * | 1989-07-24 | 1993-03-17 | International Business Machines Corporation | Method for dynamically expanding and rapidly accessing file directories |
US5555388A (en) * | 1992-08-20 | 1996-09-10 | Borland International, Inc. | Multi-user system and methods providing improved file management by reading |
US5826076A (en) * | 1994-05-13 | 1998-10-20 | Liant Software Corporation | Computer-based information access method and apparatus to permit SQL-based manipulation of programming language-specific data files |
JP3371044B2 (en) * | 1994-12-28 | 2003-01-27 | 株式会社日立製作所 | Area allocation method and disk array access method for disk array |
US5613110A (en) * | 1995-01-05 | 1997-03-18 | International Business Machines Corporation | Indexing method and apparatus facilitating a binary search of digital data |
US6226728B1 (en) * | 1998-04-21 | 2001-05-01 | Intel Corporation | Dynamic allocation for efficient management of variable sized data within a nonvolatile memory |
US6212647B1 (en) * | 1998-06-02 | 2001-04-03 | Hewlett-Packard Company | Systems and methods to perform defect management to block addressable storage media |
US6401160B1 (en) * | 1999-03-31 | 2002-06-04 | Intel Corporation | Method and apparatus to permit adjustable code/data boundary in a nonvolatile memory |
-
1997
- 1997-07-11 JP JP21970097A patent/JP3345628B2/en not_active Expired - Lifetime
-
1998
- 1998-07-08 US US09/254,437 patent/US6415375B2/en not_active Expired - Fee Related
- 1998-07-08 EP EP98931014A patent/EP0923030A4/en not_active Ceased
- 1998-07-08 WO PCT/JP1998/003067 patent/WO1999003039A1/en not_active Application Discontinuation
-
2002
- 2002-04-10 US US10/119,145 patent/US6584555B2/en not_active Expired - Fee Related
-
2003
- 2003-03-17 US US10/390,815 patent/US6654868B2/en not_active Expired - Fee Related
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060129565A1 (en) * | 2002-10-21 | 2006-06-15 | Annex Systems Incorporated | Database accelerator |
US7505979B2 (en) * | 2002-10-21 | 2009-03-17 | Annex Systems Corporation | Database accelerator |
US20070078909A1 (en) * | 2004-03-08 | 2007-04-05 | Masaharu Tamatsu | Database System |
US20110320451A1 (en) * | 2010-06-23 | 2011-12-29 | International Business Machines Corporation | Apparatus and method for sorting data |
US8725734B2 (en) * | 2010-06-23 | 2014-05-13 | International Business Machines Corporation | Sorting multiple records of data using ranges of key values |
US20210326271A1 (en) * | 2020-04-18 | 2021-10-21 | International Business Machines Corporation | Stale data recovery using virtual storage metadata |
Also Published As
Publication number | Publication date |
---|---|
JPH1131096A (en) | 1999-02-02 |
WO1999003039A1 (en) | 1999-01-21 |
US20020133686A1 (en) | 2002-09-19 |
US20030159015A1 (en) | 2003-08-21 |
EP0923030A1 (en) | 1999-06-16 |
US6415375B2 (en) | 2002-07-02 |
US6654868B2 (en) | 2003-11-25 |
JP3345628B2 (en) | 2002-11-18 |
EP0923030A4 (en) | 2002-05-02 |
US6584555B2 (en) | 2003-06-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6415375B2 (en) | Information storage and retrieval system | |
US5497485A (en) | Method and apparatus for implementing Q-trees | |
US5813000A (en) | B tree structure and method | |
Litwin | Linear Hashing: a new tool for file and table addressing. | |
US4677550A (en) | Method of compacting and searching a data index | |
US5204958A (en) | System and method for efficiently indexing and storing a large database with high data insertion frequency | |
Buchholz | File organization and addressing | |
US5619693A (en) | Method for sorting and storing data employing dynamic sort tree reconfiguration in volatile memory | |
US7921086B1 (en) | Deterministic space management in deduplication applications | |
US7418544B2 (en) | Method and system for log structured relational database objects | |
US7831624B2 (en) | Skip list with address related table structure | |
US20110093664A1 (en) | Data de-duplication by predicting the locations of sub-blocks within the repository | |
US6427143B1 (en) | Method for loading rows into a database table while enforcing constraints | |
US7080206B2 (en) | System and method for adaptively loading input data into a multi-dimensional clustering table | |
US20100010996A1 (en) | Method for the allocation of data on physical media by a file system that eliminates duplicate data | |
US20110231404A1 (en) | File storage and retrieval method | |
JP2000181768A (en) | Data storage/retrieval system | |
WO2001025962A1 (en) | Database organization for increasing performance by splitting tables | |
EA005269B1 (en) | Organising data in a database | |
US20240193142A1 (en) | Method of processing data in a database | |
Ramamohanarao et al. | Partial match retrieval using recursive linear hashing | |
EP0111689A2 (en) | Method of storing a B-tree type index file on rotating media devices | |
Manolopoulos et al. | Performance of linear hashing schemes for primary key retrieval | |
Orsborn | DATABASE TECHNOLOGY-1DL124 | |
JPH043234A (en) | Filing control method for fixed length type record |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ANNEX SYSTEMS INCORPORATED, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TAMATSU, MASAHARU;REEL/FRAME:009936/0074 Effective date: 19990417 |
|
AS | Assignment |
Owner name: ANNEX SYSTEMS INCORPORATED, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ANNEX SYSTEMS INCORPORATED;REEL/FRAME:013386/0500 Effective date: 20021005 Owner name: TAMATSU, MASAHARU, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ANNEX SYSTEMS INCORPORATED;REEL/FRAME:013386/0500 Effective date: 20021005 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
REMI | Maintenance fee reminder mailed | ||
LAPS | Lapse for failure to pay maintenance fees | ||
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20100702 |