JP2003006007A - System for automatically managing time-series data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automate management of time-series data based on parameter values set by every kind of the time-series data as realizing efficient use of disk resources to store various kinds of data such to be periodically generated such as by unit of time and by unit of month, reduction of access performance degradation in use of data and reduction of maintenance costs. SOLUTION: Data 11 generated this time of data A is inserted into a table to store the data when the table exist after acquiring information retarding the data A from time-series data management information as a processing to store the data 11. Judgment about whether or not the table to store the data 11 exists is performed by comparison between a reference processing date 12 and a name of the table existing on the current disk device. When no table exists, creation of a table (security of an area) to newly store the data generated this time is performed and the data are inserted into it. A creating place of the table in this case is selected so that distribution of the respective tables to constitute the data A becomes even to all the disk devices to the utmost.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a time series data automatic management system, and more particularly to a time series data automatic management system for managing data accumulated in time series.

[0002]

2. Description of the Related Art Conventionally, this type of automatic time-series data management system first accumulates data that occurs periodically in a database system over a long period of time. The capacity for storing the data is estimated in advance, an area for the capacity is secured as one table, and the table is stored each time the data is generated.

[0003]

The conventional time-series data automatic management system of this type has the following problems.

The first problem is that it is difficult to calculate the capacity of the table to be initially secured. The reason is that the data to be accumulated is for a long period of time, and there is no guarantee that the current data generation amount will be the same in the future. If the amount of data generated increases from the initial estimate in the future, table overflow will occur before the planned cumulative period is reached. Further, if the amount of generated data is smaller than initially estimated, even if the cumulative period is reached, the data occupancy rate in the table becomes low, resulting in wasted use of the disk device.

The second problem is that the means for deleting old data whose cumulative period has elapsed is record-based deletion, resulting in a long update time. The reason is that all the data is stored in one table, and there is no means for deleting the data other than the deletion in record units. When deleting records in the database table, pre-update information (journal) always occurs, and when deleting a large number of records, it takes some processing time. In addition, an area for storing that much pre-update information is also required.

The third problem is that it becomes necessary to reconstruct the table periodically. The reason is the same as the reason for the second problem. When records are repeatedly deleted and inserted into a table, fragmentation of data blocks constituting the table and wasteful areas are inevitably caused, resulting in deterioration of access performance and deterioration of block usage rate.

A fourth problem is that the storage destination of each data becomes local and I / O is concentrated in a specific disk device when performing an all-data search process. The reason is the same as the reason for the second problem, because all the data is stored in one table, and it is difficult to evenly arrange the data even if there are a plurality of disk devices. is there.

As a method for solving these first to fourth problems, as in the present invention, one data is stored in a plurality of tables in a time series unit such as month or year, and one virtual data is stored. The means of defining and using as a table has existed in the past. However, in this case, it is necessary for the administrator to periodically create a table to store new data and delete unnecessary tables, and there is no effective method to automate it. Was the current situation.

An object of the present invention is to solve all of the above problems and to provide a method of automating all management by using parameter information prepared for each type of data.

[0010]

According to a first embodiment of the present invention, in a database system having a storage device for accumulating regularly generated data in time series over a long period of time, A time-series data automatic management method is provided, which has means for registering parameters for each type in advance, and manages time-series data according to the parameter values.

In the second embodiment of the present invention, a means for constructing a physical unit for storing time-series data by a table of time-series units such as years and months, and a parameter on management information of the time-series data. 2. The time-series data automatic system according to claim 1, further comprising means for changing a value and means for always allocating an area with an optimum size to enable efficient and efficient use of the storage device. Provides a management method.

A third embodiment of the present invention is characterized in that it has means for physically deleting data after the retention period has passed by a table unit, and enables deletion of unnecessary data. Provide the time series data automatic management method described in 2.

In a fourth embodiment of the present invention, a means for deciding a table creation destination so that the tables making up each time series data are evenly arranged in a plurality of storage devices, and a means for using the data. 3. The time-series data automatic system according to claim 2, further comprising: a means for distributing I / O to the disk device of FIG. 1 to minimize deterioration of access performance even if the amount of retained data increases. Provides a management method.

The fifth embodiment of the present invention has means for dividing each time-series data into a plurality of storage devices and storing the data in a time-series unit, and to reduce the influence of a disk failure. The time-series data automatic management system according to claim 4, wherein the data can be locally suppressed in time series.

In the sixth embodiment of the present invention, the generation data of the time-series data to be managed and the time-series data to be processed are acquired from the machine date or the like to determine when they are. The processing reference date in which the unified date information is stored, the time series data management information of information regarding each time series data to be managed, the occurrence data, the processing reference date, and the time series data management information. The automatic time-series data management system according to claim 1, further comprising means for pre-registering in a parameter format.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Next, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a first embodiment of the present invention. The first embodiment of the present invention stores data in a database system that accumulates and manages various types of data that occur on a daily or monthly basis in a time series on a disk device over a long period of time. A means for automating the management of disk resources based on the parameter values set for each type of time-series data while realizing efficient use of disk resources, reduction of access performance deterioration when using data, and reduction of maintenance costs. including.

Referring to FIG. 1, the data 1 of the current occurrence of A
1 is the data generated this time of the data A that is the time-series management target. The processing reference date 12 stores uniform date information acquired from the machine date or the like in order to determine when the processing target data is. In the time-series data management information 13, information about each time-series data to be managed is registered in advance in a parameter format.
The disk devices 14-1 to 14-4 are storage devices for storing time series data. A_1996 to A_2000 are database tables that store the time-series data A for each data generation year, and are named according to the naming convention of FIG.

As for the process of storing the presently generated data 11 of A for the data A which is the time-series management target, first the information about A is acquired from the time-series data management information, and then the current disk device status inquiry is performed. If there is a table that should store this data, data is inserted into that table. Whether or not there is a table to be stored is determined by comparing the processing reference date 12 with the name of the table currently existing on the disk device. If it does not exist, a table for newly storing the data generated this time is created (area is reserved), and data is inserted into the table. The location of the table at this time is determined from the inquiry about the status of the disk device, and the distribution of each table forming the data A is selected to be as uniform as possible for all the disk devices. The size of the table secured at this time uses the value registered in the time-series data management information 13.

Next, the cumulative period relating to the data A is acquired from the time-series data management information 13, and it is judged whether or not there is old data to be deleted after the cumulative period has passed. This deletion is not done in record units, but physically in table units.

Finally, a virtual table definition using SQL set operation (UNION) is performed so that the data A divided and stored in a plurality of tables can be used as one table.

FIG. 2 shows the registration contents of the time series data management information regarding the time series data A. Referring to FIG. 3, the “division unit” 22 has information on whether to divide the data of the time-series data name 21 into a plurality of tables and store the data in units of years. The “division size” 23 is a physical size assigned to one table, and represents the size of the table required for each “division unit” 22 of the corresponding data. This value is calculated or measured for each time series data and registered. The “cumulative period” 24 represents the minimum holding period of the time series data, and represents how many accumulations are necessary in the unit set in the “division unit” 22.

FIG. 3 is a logic block diagram showing the management system of the present invention, and the processing units of this management system are divided into blocks.

With reference to FIG. 3, the operation in the case of processing the data A generated this time will be described in detail.

When the storage request 31 for the time-series data A is made, the determination processing 3 of the storage destination table for storing the data 3
2 is performed.

In the storage destination table determination processing 32, the "division unit" of the data A is acquired from the time-series data management information 13 (FIG. 1), and the division unit 22 of the data A is the month (FIG. 2). I understand. Next, it is assumed that the processing reference date 12 (FIG. 1) is acquired, and as a result, the data to be processed this time is the data for January 2001. In light of the naming rules in Fig. 4, the name of the table that stores the data generated this time is "A_200
1 ”, and this table shows the disk devices 14-1 to 14-4.
(Fig. 1) It is searched whether or not it exists. As a result,
If the target table “A_2001” exists, the data storage process 34 is performed on the table. However, in the case of this embodiment, since the table does not exist, the storage destination table creation process 33 is performed. Become.

In the storage destination table creation processing 33, first, the table creation destination disk device is determined according to the flowchart of FIG. In step S1 of FIG. 5, the division size of the data A is acquired from the time-series data management information (3 in FIG. 1). From FIG. 2, this value is 1000 MByte. Step S
In 2, the number of partition tables forming the data A is present on each disk device 14-1 to 14-4 (Fig.) At present, and this is defined as the multiplicity for each disk device. Referring to FIG. 1, the multiplicity of each disk device for data A is as follows.

Disk device 14-1: Multiplicity 2 Disk device 14-2: Multiplicity 2 Disk device 14-3: Multiplicity 1 Disk device 14-4: Multiplicity 1 Therefore, in step S3 of FIG. 14-
The process proceeds to the following steps with 3 and 14-4 as targets (groups). Now, the free space of the disk device 14-3
3000MByte, free space of disk unit 14-4 is 4000Mby
If it is te, the disk A becomes the disk device 14-4 in step S4. In step S5, it is determined that the free capacity (4000 MBytes) of the disk device 4-4 is equal to or larger than the division size (1000 MBytes) of the time-series data A, the table creation destination disk is determined to be the disk device 14-4, and "A_2
A table is created (area is reserved) on the disk device 14-4 with the name "001". The size at this time is 1000 MBytes, which is the division size of the data A first.

In the data storage process 34, the table name "A_2001" obtained in the storage destination table determination process 32 is set to A
The presently generated data 11 (FIG. 1) is stored.

In the judgment processing 35 of the accumulated period elapsed data,
Processing reference date 12 (Fig. 1) and time series management information 13 (Fig. 1)
After acquiring the "cumulative period" registered in the table, search for all the tables forming the data A from the disk devices 14-1 to 14-4 (Fig. 1), and check whether there is a table that can be deleted. Make a decision. Since a new table has been newly created in the storage destination table creation process 33, there are seven tables that currently compose the data A, "A_1995" to "A_2001".
Since the “cumulative period” 25 (FIG. 2) of data A is 5,
It is necessary to keep data for at least the past 5 years.
Since the processing reference date 12 is January 2001, "A_199
If the table up to 6 ”exists, it means that at least the data for the past 5 years is held. Therefore, in this case, the deletion target is "A_1995", and the table "A_1995" is logically deleted from the database system and physically deleted from the disk device in the deletion process 36 of the retention period elapsed data. .

In the virtual table creation process 37, as shown in FIG. 6, all the tables in which the data A of the disk device 60 is divided and stored are used as one virtual table 80 of the data A. A virtual table is defined in a virtual table definition process 70 by a set operation (UNION). It is necessary to redefine the virtual table 80 each time there is a change in the members of the tables that make up the virtual table 80. In this method, the tables that make up the time-series data are newly created or deleted. Redefine.

[0032]

As described above, the present invention has the following features. -By registering parameters for each type of time-series data in advance, all management can be automated. -Since data after the retention period has passed is not deleted in record units but physically deleted in table units, pre-update information (journal) does not occur, enabling high-speed deletion of unnecessary data. -By this deletion method, physical fragmentation in the database block does not occur, so maintenance processing such as regular table reconfiguration is not necessary, and a large reduction in operating costs is possible. -Since the physical unit for storing time-series data is configured by a table of time-series units such as year and month, even if the amount of generated data of the target data increases or decreases due to changes in the amount of information, the time By changing the parameter value on the series data management information, the area is always secured with the optimum size, and the storage device can be efficiently used without waste. -Since the table creation destinations are determined so that the tables that make up each time-series data are evenly allocated to multiple storage devices, the I / O to the disk devices during data use is distributed and held. Even if the amount of data increases, it is possible to minimize access performance deterioration. -Since each time-series data is stored in multiple storage devices divided in time-series units, it is possible to suppress the effect of a disk failure locally in time-series for each data. .

[Brief description of drawings]

FIG. 1 is a diagram showing currently generated data of data A according to an embodiment of the present invention.

FIG. 2 is a diagram showing the contents of time-series data management information regarding data A in one embodiment of the present invention.

FIG. 3 is a block diagram showing a management system according to an embodiment of the present invention.

FIG. 4 is a diagram showing a naming convention in an embodiment of the present invention.

FIG. 5 is a diagram showing a flowchart in an embodiment of the present invention.

FIG. 6 is a diagram showing all tables in which data A of one embodiment of the present invention is divided and stored.

[Explanation of symbols]

This time's occurrence data of 11 A 12 Processing reference date 13 Time-series data management information 14-1 to 14-4, 60 Disk device 21 Time series data name 22, 41 division unit 23 division size 24 cumulative period 31 Storage request for time series data A 32 Storage Destination Table Judgment Processing 33 Storage table creation processing 34 Data storage processing 35 Judgment processing of accumulated period elapsed data 36 Deletion processing of retention period elapsed data 37 Virtual table creation processing 42 Contingency table name 70 Virtual table definition processing 80 virtual table

Claims (6)

[Claims] 1. A database system having a storage device for accumulating regularly generated data in time series over a long period of time, comprising means for previously registering parameters for each type of time series data, An automatic time-series data management method that manages time-series data according to parameter values. 2. A means for constructing a physical unit for storing time-series data as a table of time-series units such as a year and a month, and a means for changing a parameter value in management information of the time-series data are always optimal. The time-series data automatic management system according to claim 1, further comprising means for allocating an area of a different size, enabling efficient and efficient use of the storage device. 3. The time-series data automatic management method according to claim 2, further comprising means for physically deleting the data after the retention period has passed by a table unit so that unnecessary data can be deleted. . 4. A means for deciding a table creation destination so that a table constituting each time series data is evenly arranged for a plurality of storage devices, and an I for a disk device when using the data.
3. The time series data automatic management method according to claim 2, further comprising means for distributing / O, which makes it possible to suppress deterioration of access performance to a minimum even if the amount of retained data increases. 5. A means for storing each time-series data in a plurality of storage devices by dividing it in time-series units, and suppressing the influence at the time of a disk failure locally on each data in time series. The time-series data automatic management system according to claim 4, which enables the above. 6. The generation data of the time-series data to be managed and the unified date information acquired from the machine date or the like to determine when the time-series data to be processed are stored are stored. Processing reference date, time-series data management information of information relating to each time-series data to be managed, and means for previously registering the generated data, the processing reference date, and the time-series data management information in a parameter format. The automatic time-series data management system according to claim 1, characterized in that it has.