JP5626561B2 - Information processing system and data management method thereof - Google Patents

Description

  The present invention relates to an information processing system and a data management method thereof, and more particularly to a data management method of a database.

  Conventionally, in a system in which database data is divided into a plurality of groups and arranged on a main storage device capable of high-speed reading or writing, all the data is arranged on the main storage device even if it is arranged on the main storage device. Since the possible capacity is limited, there is a method of storing data that cannot be arranged in a file on a external storage device or a relational database. At this time, when accessing data in an external storage device that is subjected to low-speed reading or writing, the high-speed search processing is impaired.

  Therefore, for example, in Patent Document 1, the elapsed time since the last data access is used as an index indicating the access frequency, and when the elapsed time passes a certain threshold, it is determined that the access frequency is low, and the usage frequency is high. It is described that the high speed of data access is maintained by arranging (storing) the divided data in the main storage device and arranging the data with low use frequency in the external storage device.

  Other database-related technologies include those described in Patent Documents 2 to 5. Patent Document 2 describes an invention related to a cache control method for a cache storage device. In Patent Document 3, the address area of the main storage device is divided into a main storage area and an extended storage area. The extended storage area is used for page storage, and when there is no page on the main storage area, high-speed transfer is performed from the extended storage area. It describes what to do.

  Patent Document 4 describes that, for example, a memory having the oldest access time is output to a secondary medium when a memory overload is detected. Patent Document 5 describes that compressed data is stored in a plurality of memories, redundant information is deleted without loss of information amount, and the memory capacity is increased.

JP 2009-217688 A JP 2000-285005 A JP 62-037754 A Japanese Patent Laid-Open No. 02-072448 Japanese Patent Application Laid-Open No. 07-105092

  In the technique of Patent Document 1, when there is a large amount of frequently accessed data and the capacity of the main storage device cannot be placed on the main storage device, even the frequently accessed data is unconditionally placed on the external storage device. Therefore, the high-speed search process that is a feature of the memory database is impaired.

  An object of the present invention is to provide an information processing system and a data management method therefor that can maintain high speed search processing even when the capacity of the main storage device is often exceeded and cannot be arranged on the main storage device. There is to do.

  The present invention stores in the storage means management information indicating the free space of the main part and the sub part of the main storage device and the data with the oldest access date and time for the data respectively stored in the main part and the sub part. The input data is appended with additional information including the date and time information when the data was last accessed, the capacity and compression capacity of the data, and the next data information indicating the data with the next oldest access date and time. . Then, when the data is input, the control means sends the new data of the access date to the main unit based on the management information of the main part and the sub part of the main storage device and the additional information added to the input data. Is compressed and moved to the sub-unit, and the data whose access date and time is older than the sub-unit data is moved to the external storage device.

  According to the present invention, when the input data exceeds the capacity of the main unit when stored in the main unit of the main storage device, the data with the oldest access date is compressed and moved to the sub unit of the main storage device. As a result, the free space in the main part of the main storage device can be secured, and the high speed of data access can be maintained even when there is a lot of data with high access frequency. Even if the data moved to the sub section is accessed again, since the data is on the main memory that can be read or written at high speed, the high speed of data access can be similarly maintained. .

It is a block diagram which shows the structure of one Embodiment of the information processing system which concerns on this invention. It is a figure which shows an example of the database which concerns on this invention. It is a figure which shows an example of the additional information of the column data which concerns on this invention. It is a figure which shows an example of the information for managing the whole column data of the main part and sub part of the main memory which concerns on this invention. It is a flowchart explaining operation | movement of the information processing system which concerns on this invention. It is a figure which shows the example of the additional information of the column data in operation | movement of the information processing system which concerns on this invention, the information which manages the whole column data of the main part of a main memory, and the information which manages the whole column data of a sub part. It is a figure which shows the example of the additional information of the column data in operation | movement of the information processing system which concerns on this invention, the information which manages the whole column data of the main part of a main memory, and the information which manages the whole column data of a sub part.

  Next, embodiments for carrying out the invention will be described in detail with reference to the drawings. In particular, the present invention often divides database data into a plurality of groups on a main storage device that can be read or written at high speed, and cannot be placed on the main storage device beyond the capacity of the main storage device. In a database system, a main storage device having a main part and a sub part is used. Data with high access frequency is compressed into the main part, data with low access frequency is compressed and data with low access frequency is further compressed into the sub part. Is moved to an external storage device. By doing so, data with high access frequency is arranged in the main storage device as much as possible, and the high-speed search processing of the database is ensured. Low access frequency means new and old access date.

  FIG. 1 is a block diagram showing a configuration of an embodiment of an information processing system according to the present invention. In the figure, 110 is a computer that performs information processing, and 120 is an external storage device connected to the computer 110. In the computer 110, a control device 111, a main storage device 112, and the like are provided.

  The main storage device 112 includes a main unit 113 that arranges (stores) data with high access frequency and a sub-unit 114 that compresses and arranges (stores) data with low access frequency. The main storage device 112 is, for example, a semiconductor memory, and is a storage device capable of high-speed reading or writing. For example, a magnetic disk device or the like is used as the external storage device 120. In the present embodiment, an example in which the external storage device 120 is provided separately from the computer 110 will be described. However, the external storage device 120 may be a hard disk device or the like provided in the computer 110.

  The control device 111 includes a CPU (Central Processing Unit) (not shown) that executes predetermined processing in accordance with a program, a memory (not shown) for storing the program, and the like. The data placement and movement of the unit 114 and the external storage device 120 are controlled.

  In addition, when the control device 111 newly arranges data divided from the program in the main storage device 112, if the main unit 113 of the main storage device 112 has free space, the control device 111 transfers the corresponding divided data to the main storage device. 112 is arranged in the main part 113. If there is no free space, among the divided data arranged in the main unit 113 of the main storage device 112, the divided data with the oldest access date is compressed and moved to the sub unit 114 of the main storage device 112 to be moved to the main storage device. The free space is secured in the main unit 113 of 112, and the corresponding divided data is arranged in the main unit 113 of the main storage device 112.

  Further, when the control unit 111 moves the divided data arranged in the main unit 113 of the main storage device 112 to the sub unit 114 and the sub unit 114 of the main storage device 112 has free space, the corresponding division is performed. The data is arranged in the sub unit 114 of the main storage device 112. If there is no free space, among the divided data arranged in the sub unit 114 of the main storage device 112, the divided data with the oldest access date is moved to the external storage device 120 to the sub unit 114 of the main storage device 112. Control such as securing free space and moving the corresponding divided data to the sub-unit 114 of the main storage device 112 is performed.

  The control device 111 is arranged in the main unit 113 when moving the column data of the main unit 113 whose elapsed time from the date and time when the column data is accessed from the program exceeds a certain threshold to the sub unit 114. Using the oldest column data address of the information for managing the entire column and the next data information of the additional information of the column data, the column data whose access date and time exceeds the threshold is searched. If the corresponding column data exists, the above-described processing is performed to move the corresponding column data to the sub unit 114 of the main storage device 112.

  Furthermore, the control device 111 is arranged in the sub unit 114 when moving the column data of the sub unit 114 whose elapsed time from the date and time when the column data was accessed from the program exceeds a certain threshold to the external storage device 120. Column data whose access date and time exceeds the threshold is searched using the oldest column data address of the information for managing the entire column and the next data information of the additional information of the column data. If the corresponding column data exists, the above-described processing is performed to move the corresponding column data to the external storage device 120.

  Here, the database of the information processing system is registered in a table format. FIG. 2 shows an example of a database. As shown in FIG. 2, the product, the manufacturer of the product, and the price information of the product are described in a table format. In addition, “product name” is described in the first column, “manufacturing company” is described in the second column, and “product price” is described in the third column. The product name A shown in the first line is manufactured by company X and indicates that the price is N yen.

  The database table is divided for each column, and the divided data is a column. Additional information for managing the column, that is, the date and time when the column data was last accessed, the next data information for establishing a link in order of the date and time when the column data (+ additional information) was accessed, the capacity of the column data, the column A compression capacity when the data is compressed and a flag indicating whether or not the column data is compressed are added to the column data.

  The next data information is a pointer (data address where the column data is arranged), and next data information (column data with the next oldest access date / time) indicating the column data with the next oldest access date / time with respect to the corresponding column data. Data address). When the column data is moved, the additional information is moved along with the movement of the column data.

  If the date and time when the column data was last accessed exceeds the capacity that can be stored in the main unit 113 when the column data is moved to the main unit 113 of the main storage device 112, the column data is compressed and the sub-data of the main storage device 112 is compressed. Whether there is column data exceeding a certain elapsed time (threshold value), and whether or not the column data is compressed and moved to the sub-unit 114 of the main storage device 112 is determined. Information. The date / time information is acquired from a date / time control device (not shown) of the computer 110.

  The next data information for establishing a link in order of the date and time when the column data was accessed is used when referring to the access date and time of the column data. When column data is newly input, the corresponding column data is the most recently accessed column data, so nothing is set in the next data information. If the next accessed column data exists, the data address of the column data corresponding to the next data information is set.

  The flag indicating whether or not the column data is compressed is set to “0” when not compressed and “1” when compressed. When the column data is newly input, a flag indicating whether or not the column data is compressed is “0” indicating that the column data is not compressed. The column data is compressed by a commonly used lossless compression (lossless compression, LZH, ZIP, etc.). It is assumed that a compression unit (not shown) that compresses this data is arranged in the computer 110.

  FIG. 3 is a diagram showing an example of additional information of column data. In FIG. 3, the corresponding column data is last accessed at 09:24:22 on December 25, 2009 (access date), the capacity is 4 GB, the compression capacity is 2 GB, and the compression flag is “0”. Since the compression flag is “0”, it indicates that the column data is not compressed. In the example of FIG. 3, nothing is set for the next data information.

  The free capacity, threshold value, and data address of the oldest column data in the main unit 113 of the main storage device 112, which are information for managing the entire database column arranged in the main unit 113 of the main storage device 112, are stored in advance. Arranged in the main section 113 of the main storage device 112. Similarly, the free capacity, threshold value, and data address to the oldest column data in the sub unit 114 of the main storage device 112, which are information for managing the entire database column arranged in the sub unit 114 of the main storage device 112 Are arranged in advance in the sub-unit 114.

  The threshold value of the main unit 113 of the main storage device 112 is a criterion for determining whether to compress the column data and move it to the sub unit 114 of the main storage device 112. The threshold value of the sub unit 114 of the main storage device 112 is: This is a criterion for determining whether or not to move the compressed column data to the external storage device 120.

  FIG. 4 shows information 401 for managing the entire database column arranged in the main unit 113 of the main storage device 112, and manages the entire database column arranged in the sub unit 114 of the main storage device 112. An example of the information 402 for this is shown. As indicated by 401 in FIG. 4, the free capacity of the main unit 113 of the main storage device 112 is 5 GB, the threshold used for movement determination is 30 minutes, and the address to the oldest column data is 50006. Similarly, as indicated by 402 in FIG. 4, the free capacity of the sub unit 114 of the main storage device 112 is 2 GB, the threshold used for movement determination is 60 minutes, and the address to the oldest column data is 17022.

  Next, the overall operation of the present embodiment will be described in detail with reference to FIGS. FIG. 5 is a flowchart for explaining the operation of this embodiment. In the present embodiment, the divided data is a database column. Further, in the present embodiment, since the data management method has characteristics, the contents related to the characteristics will be described in detail, and the database access method and the like are well known, and detailed description thereof will be omitted.

  First, as shown in FIG. 5, when the column data to which the additional information shown in FIG. 3 is added is newly input, the control device 111 uses the data capacity 4 GB of column data and 401 in FIG. It is determined whether or not column data can be arranged in the main unit 113 of the main storage device 112 with reference to the free space 5 GB of information for managing the entire column arranged in the main unit 113 of the main storage device 112 shown in FIG. (S501 in FIG. 5). The main unit 113 and the sub unit 114 of the main storage device 112 are in the state shown in FIG.

  The control device 111 determines that the column data can be arranged in the main unit 113 from the data capacity 4 GB of the column data and the free capacity 5 GB of the main unit 113 (Yes in S501), and arranges the corresponding column data in the main unit 113 ( S506), the free capacity of the main part 113 of the management information for managing the entire column is changed to 1 GB, and the process is terminated.

  In addition, when the column data to which the additional information indicated by 601 in FIG. 6 is newly input, the control device 111 (however, the management information of the main unit 113 and the sub unit 114 is in the state shown in FIG. 4). 6), it is determined that the column data cannot be arranged in the main unit 113 from the data capacity 7GB of the column data indicated by 601 in FIG. 6 and the free capacity 5GB of the main unit 113 indicated by 401 in FIG. 4 (No in S501). In that case, the oldest column data address of information for managing the entire column arranged in the main unit 113 is referred to search for the column data with the oldest access date and time. For example, if the column data indicated by 602 in FIG. 6 is the oldest column data, the corresponding column data is compressed (S502).

  Next, the control device 111 determines that the compressed column data can be arranged in the sub unit 114 from the compression capacity 1 GB of the column data indicated by 602 in FIG. 6 and the free capacity 2 GB of the sub unit 114 indicated by 402 in FIG. 4 (S503) Yes), the corresponding compressed column data is moved to the sub unit 114 (S504). Further, the free capacity of the main unit 113 is changed to 10 GB (603 in FIG. 6), the free capacity of the sub unit 114 is changed to 1 GB (604 in FIG. 6), and the oldest column data address of the main unit 113 is transferred to the sub unit 114. The data address of the column data having the next oldest access date is changed from the next data information of the additional information of the moved column data.

  That is, since the oldest column data of the main unit 113 is moved to the sub unit 114, it is necessary to change the oldest column data of the main unit 113, and the next column linked from the next data information of the oldest column data The data is changed to the oldest column data of the main unit 113. For example, the data address is “50010” as indicated by reference numeral 603 in FIG.

  The control device 111 moves the column data corresponding to 602 in FIG. 6 to the sub unit 114, and then the data capacity 7 GB of the column data indicated by 601 in FIG. 6 and the free space in the current main unit 113 indicated by 603 in FIG. It is determined that the column data can be arranged in the main unit 113 from the capacity 10 GB (Yes in S501). Thereafter, the corresponding column data is arranged in the main unit 113, the free capacity of the main unit 113 is changed to 3 GB (605 in FIG. 6), and the process is terminated (S506).

  Next, an operation when the column data indicated by 601 input in S501 cannot be arranged in the main unit 113 will be described (that is, S501 is NO and S503 is NO). The management information of the main unit 113 and the sub unit 114 is information 401 and 402 as shown in FIG. First, the control device 111 refers to the oldest column data address of information for managing the entire column arranged in the main unit 113 and searches for the column data with the oldest access date and time. For example, if the column data indicated by 701 in FIG. 7 is the oldest column data, the corresponding column data is compressed by the compression unit (S502).

  The control unit 111 determines that the column data compressed from the compressed capacity 4 GB of column data indicated by 701 in FIG. 7 and the free capacity 2 GB of the sub unit 114 indicated by 402 in FIG. 4 cannot be placed in the sub unit 114 (No in S503). ), The column data with the oldest access date and time is searched using the oldest column data address of the information for managing the entire column of the sub unit 114.

  For example, if the column data indicated by 702 in FIG. 7 is the oldest column data, the corresponding column data is moved to the external storage device 120 (S505), and the free capacity of the sub unit 114 is set to 5 GB (FIG. 7). 704). Further, using the next data information of the additional information of the column data moved to the external storage device 120, the oldest column data address of the sub unit 114 is changed to the data address of the column data with the oldest access date (FIG. 7). In 704, the data address is “19001”).

  Next, the control device 111 determines that the column data can be arranged from the compression capacity 4 GB of the additional information of the column data indicated by 701 in FIG. 7 and the current free capacity 5 GB of the sub unit 114 indicated by 704 in FIG. 7 (S503). Yes), the corresponding compressed column data is moved to the sub unit 114 (S504). Also, the free capacity of the main unit 113 is changed to 10 GB (705 in FIG. 7), the free capacity of the sub unit 114 is changed to 1 GB (706 in FIG. 7), and the oldest column data address of the main unit 113 is moved to the sub unit 114. The next data information of the column data additional information is changed to the data address of the column data with the next oldest access date. In 705 of FIG. 7, the data address is “70017”.

  Subsequently, the control device 111 determines that the column data can be arranged in the main unit 113 from the column data capacity 7 GB indicated by 601 in FIG. 6 and the current free capacity 10 GB of the main unit 113 indicated by 705 in FIG. 7 (S501). Is yes). Next, the corresponding column data is arranged in the main unit 113 (S506), the free capacity of the main unit 113 is changed to 3 GB (707 in FIG. 7), and the process is terminated.

  The control device 111 repeatedly performs the above-described processing until newly input column data can be arranged in the main unit 113. That is, when the data is input, the control device 111 sends the new data with the access date to the main unit 113 based on the management information of the main unit 113 and the sub unit 114 and the additional information added to the input data. Control is performed so that the compressed data is moved to the sub-unit 114 and the data whose access date is older than the data of the sub-unit 114 is moved to the external storage device 120.

  When data access is made to the column data of the sub unit 114 or the column data of the external storage device 120, the processing shown in FIG. 5 is performed as column data to which the corresponding column data is newly input. At that time, in the case of placing in the main unit 113 in S506 of FIG. 5, the compressed column data is decompressed and placed.

  Further, as shown in 401, 402, etc. in FIG. 4, a threshold is set for the information for managing the entire row of the main unit 113 and the sub unit 114, and the control device 111 determines the threshold from the access date and time in the main unit 113. Control is performed to move the data exceeding (elapsed time) to the sub unit 114. Further, control is performed to move data that exceeds a threshold (elapsed time) from the access date and time in the sub unit 114 to the external storage device 120.

  At that time, when the column data of the main unit 113 is moved to the sub unit 114 and the column data of the sub unit 114 is moved to the external storage device 120 when the elapsed time from the date and time when the column data is accessed by the program exceeds a certain threshold. The control device 111 stores the oldest column data address of information for managing the entire column arranged in the main unit 113, the next data information of the additional information of the column data, the date information of the date control device in the computer 110. Use to search for column data whose access date and time exceeded the threshold. If there is column data exceeding the threshold value, the processing of FIG. 5 is performed as column data to which the corresponding column data is newly input.

  For example, the elapsed time from the date and time when the column data was accessed by the program on October 1, 2009 at 17:00: 00 exceeded a certain threshold (401 in FIG. 4, 30 minutes, 60 in 402 in FIG. 4). It is assumed that the process of moving the column data of the main unit 113 to the sub unit 114 and the column data of the sub unit 114 to the external storage device 120 is performed. In this case, if the threshold value is 30 minutes of 401 in FIG. 4, column data that does not exceed 17:00 on 1st October 2009 by adding 30 minutes to the access date and time of the column data additional information Is the target column data to be moved to the sub unit 114.

  For example, since the column date and time indicated by reference numeral 602 in FIG. 6 is accessed on October 1, 2009, 16: 22: 5, adding 30 minutes results in October 1, 2009, 16: 52: 5, 2009 This is column data that does not exceed 17:00 on October 1 and is subject column data to be moved to the sub unit 114. The same applies when the column data of the sub unit 114 is moved to the external storage device 120.

  In this embodiment, when input data is stored in the main part of the main storage device, if the capacity of the main part is exceeded, the data with the oldest access date is compressed and moved to the sub part of the main storage device. Thus, it is possible to secure a free space in the main part of the main storage device. Accordingly, since a large amount of data can be stored in the main part, high-speed reading or writing of data becomes possible, and even if the data moved to the sub part is accessed again, the main storage device capable of high-speed reading or writing Therefore, it is possible to maintain high speed data access.

  Also, among the data arranged in the main part of the main storage device, data whose elapsed time from the access date and time exceeds a certain threshold is compressed and moved to the sub part of the main storage device. Can be freed up in the main part of the main storage device. Therefore, high-speed reading or writing of data can be performed in the same manner, and high-speed data access can be maintained even when the data moved to the sub part is accessed again.

  Therefore, even when there is a large amount of data that is frequently accessed and often cannot be placed on the main storage device beyond the capacity of the main storage device, it is possible to maintain the high speed search processing that is a feature of the memory database.

  The present invention arranges database data divided into a plurality of groups on a main storage device capable of high-speed reading or writing, and has a high access frequency on the main storage device, exceeding the capacity of the main storage device. The present invention can be suitably used for an information processing system that cannot be arranged.

110 Computer 111 Control Device 112 Main Storage Device 113 Main Unit 114 Sub Unit 120 External Storage Device

Claims (6)

A main storage device having a main unit for storing input data and a sub-unit for storing compressed input data;
An external storage device;
Storage means for storing management information indicating the data with the oldest access date and time with respect to the data stored in the main part and the sub part, respectively, the free capacity of the main part and the sub part of the main storage device,
Control means for controlling storage of input data to the main storage device and external storage device and data movement between the main storage device and external storage device,
The input data is appended with additional information including the date and time information when the data was last accessed, the capacity and compression capacity of the data, and the next data information indicating the data having the next oldest access date and time with respect to the data. And
When the data is input, the control means causes the new access date / time data to be sent to the main unit based on the management information of the main and sub units of the main storage device and the additional information added to the input data. The information processing system is characterized in that control is performed so that the data is compressed and moved to the sub unit, and the data whose access date and time is older than the data of the sub unit is moved to the external storage device.   If the input data cannot be stored in the main unit at the time of the data input, the control means may store the data of the main unit based on the management information of the main unit and the sub unit and additional information added to the input data. The data having the oldest access date and time is compressed and moved to the sub-part to store the input data in the main part, and the data of the main part is moved to the sub-part When the input data cannot be stored in the main unit, the control unit stores the input data in the main unit by moving the data of the sub unit having the oldest access date to the external storage device. The information processing system according to claim 1.   The management information of the main part and the sub part includes a threshold value indicating a predetermined elapsed time from the date and time when the data of the main part and the sub part are accessed, and the control means includes the main part and the sub part. Control for searching data that passes the threshold value with reference to the additional information added to the data, and compressing the corresponding data and moving it to the sub-part when there is data that passes the threshold value in the main part 3. The information processing system according to claim 1, wherein when there is data whose threshold value has elapsed in the sub unit, control is performed to move the corresponding data to the external storage device. A main storage device having a main unit for storing input data and a sub-unit for storing compressed input data;
An external storage device;
Storage means for storing management information indicating the data with the oldest access date and time with respect to the data stored in the main part and the sub part, respectively, the free capacity of the main part and the sub part of the main storage device,
Control means for controlling the storage of input data to the main storage device and the external storage device and the movement of data between the main storage device and the external storage device;
A data management method for a system comprising:
The input data is appended with additional information including the date and time information when the data was last accessed, the capacity and compression capacity of the data, and the next data information indicating the data having the next oldest access date and time with respect to the data. And
Based on the management information of the main part and the sub part of the main storage device and the additional information added to the input data, the control means sends new data with the access date to the main part when the data is input. The data management method is characterized in that control is performed such that the data is compressed and moved to the sub-part, and the data whose access date and time is older than the data of the sub-part is moved to the external storage device.   If the input data cannot be stored in the main unit at the time of the data input, the control means may store the data of the main unit based on the management information of the main unit and the sub unit and additional information added to the input data. The data having the oldest access date and time is compressed and moved to the sub-part to store the input data in the main part, and the data of the main part is moved to the sub-part When the input data cannot be stored in the main unit, the control unit stores the input data in the main unit by moving the data of the sub unit having the oldest access date to the external storage device. The data management method according to claim 4, wherein the data management method is a data management method.   The management information of the main part and the sub part includes a threshold value indicating a predetermined elapsed time from the date and time when the data of the main part and the sub part are accessed, and the control means includes the main part and the sub part. Control for searching data that passes the threshold value with reference to the additional information added to the data, and compressing the corresponding data and moving it to the sub-part when there is data that passes the threshold value in the main part 6. The data management method according to claim 4, wherein when there is data whose threshold value has passed in the sub unit, control is performed to move the corresponding data to the external storage device.