JP5222171B2 - Database management method and database management system - Google Patents

Description

The present invention relates to a technology of the database management method and database management systems.

  With the development of the information society, many companies provide various online services using systems including databases. For example, bank ATMs (Automated Teller Machines) and air ticket sales systems are examples of online services. In recent years, competition between companies has been intensifying, and each company has been trying to differentiate itself from other companies by expanding online service hours and expanding online service content.

  When constructing an online service system, various hardware and middleware are often combined. In this case, maintenance for each software and hardware and maintenance for the entire system are necessary, and the system management cost is very high. Since system management costs are always incurred during system operation, it is important to reduce system management costs and secure profits in order to win competition between companies. For this reason, it is expected that the system can be effectively used by effectively utilizing the time zone outside the online service, and an outcome corresponding to the system management cost can be obtained (see Non-Patent Document 1, etc.).

  In a database management system, statistical information or the like may be generated by batch processing data accumulated through online operations as a batch process, and the generated statistical information may be used for management decisions. In batch processing that generates statistical information, it is necessary to access a large amount of data, and the system load tends to increase during the execution of batch processing.

  In particular, when data is stored in an external storage medium such as a disk drive, the load due to data input / output increases. For this reason, batch processing is often executed at night when the load is relatively small and outside the online service time. Conventionally, all data at the time of batch processing is backed up regardless of whether or not there is an operation update.

Jim Gray, Andreas Reuters, “Transaction Processing <Concept> —Concepts and Techniques”, Nikkei Business Publications, October 2001, p. 743

  The batch processing time tends to be longer due to the increase in data processing amount accompanying the expansion of online services and the increase in online service users. However, each company has expanded online service time, and the time allocated for batch processing has become shorter. Therefore, shortening of the batch processing time is desired.

Next, the operation of batch processing in the database management system will be described with reference to explanatory diagrams (FIGS. 12 and 13) in a Gantt chart format with the horizontal axis as the time axis.
FIG. 12 is an explanatory diagram showing the batch process b1, the batch process b2, and the backup process (c1, c2) for each batch process disclosed in Non-Patent Document 1. When executing a plurality of batch processes, the data contents of the database at the end of each batch process (t1, t3) are backed up (c1, c2) in preparation for a failure.

  Here, the system load due to input / output with the external storage device when executing each batch process is very large. In order to reduce this system load, a process (hereinafter referred to as in-memory data processing) is considered in which the data stored in the external storage medium is stored in the memory, and the processing speed is increased by reading / writing the data on the memory. . In-memory data processing does not require access to an external storage device, so that a great effect can be obtained in reducing the system load.

FIG. 13 is an explanatory diagram showing an example of in-memory data processing.
At time t0 to t1, load processing (also referred to as in-memory) for reading data from the external storage medium and storing it in the memory is executed.
At time t1 to t2, the batch process b1 is executed by reading and writing data on the memory on which the load process has been performed.
At times t2 to t3, backup processing (c1), which is processing for writing data on the memory, which is the processing result of batch processing (b1), to an external storage medium is executed.
Similarly, at times t3 to t5, batch processing (b2) and backup processing (c2) are executed.
At times t5 to t6, as DB synchronization processing, data is read from the memory and stored in the external storage medium.

  Here, it is necessary to perform control so that the external storage medium cannot be updated until the DB synchronization processing at times t5 to t6 is completed. This waiting time increases the overall time (time t0 to t6) related to batch processing.

  SUMMARY OF THE INVENTION Accordingly, it is a main object of the present invention to solve the above-described problems and to shorten processing time related to data synchronization in a database management system.

The problem to solve, the present invention provides a database management method in a database management system having a database management system having a memory manages the data stored in the storage device a database,
The database management apparatus,
Executing a load process for reading the data of the database stored in the storage device into the memory;
Wherein for the load processed data, analyzes the access request received, it performs batch processing according to the result of the analysis of the access request,
The data that is updated in the memory that has been processed by said batch processing, performs the backup process to write to the backup device as backup data after the execution of the batch process,
The backup device is
In response to an instruction issued from the database management device every time the backup data is stored a plurality of times, the stored backup data is reflected in the data of the database stored in the storage device. A synchronization process for making the above data and the data stored in the storage device identical,
When the database management device newly performs the batch processing , the backup is performed on the backup data that has been backed up with respect to the batch processing and another batch processing executed before the batch processing . and wherein the synchronization process executed by the device, characterized in Rukoto is parallel processing.
Other means will be described later.

  According to the present invention, it is possible to shorten the processing time related to data synchronization in a database management system.

It is a block diagram which shows the structure of the database management system regarding one Embodiment of this invention. It is explanatory drawing which shows the main data flows about the process of the database management system regarding one Embodiment of this invention. It is explanatory drawing which shows the execution order of each process demonstrated in FIG. 2 regarding one Embodiment of this invention. It is explanatory drawing which shows an example of the data content in each process of FIG. 3 regarding one Embodiment of this invention. It is explanatory drawing which shows an example of the data content in each process of FIG. 3 regarding one Embodiment of this invention. It is explanatory drawing which shows an example of the data content in each process of FIG. 3 regarding one Embodiment of this invention. It is explanatory drawing which shows an example of the data content in each process of FIG. 3 regarding one Embodiment of this invention. It is explanatory drawing which shows the outline | summary of the data recovery process regarding one Embodiment of this invention. It is explanatory drawing which shows an example of the data content in the data recovery process regarding one Embodiment of this invention. It is a flowchart which shows the backup process of each batch process result regarding one Embodiment of this invention. It is a flowchart which shows the recovery process at the time of the failure generation regarding one Embodiment of this invention. It is explanatory drawing which shows the batch process and the backup process with respect to each batch process. It is explanatory drawing which shows an example of an in-memory data process.

  Hereinafter, an embodiment of a database management system to which the present invention is applied will be described with reference to the drawings.

  FIG. 1 is a configuration diagram showing the configuration of the database management system. The database management system is a system in which a database management device 1, a storage device 5, a terminal 7, and a backup device 8 are connected to each other through a network 6.

The database management device 1 is configured as a computer including a processor 1a, a memory 1b, and an interface 1c for communicating with other devices. The processor 1a constitutes a processing unit for executing various processes by processing a program stored in the memory 1b.
The memory 1b stores a program for configuring the DB business processing unit 10 and the DB operation processing unit 20. The DB task processing unit 10 includes an online request analysis unit 11, an online processing unit 12, a batch request analysis unit 13, and a batch processing unit 14. The DB operation processing unit 20 includes a DB load processing unit 21, a DB backup processing unit 22, a DB synchronization instruction unit 23, and a DB recovery processing unit 24. Details of these processing units will be described later.
The memory 1b stores an in-memory data buffer 15 and an update management table 16 as data. The in-memory data buffer 15 is a data storage destination when the data content of the database 52 is loaded 92 (see FIG. 2). The update management table 16 is a table for managing update contents from the data contents of the database 52 subjected to the load process 92.

  The storage device 5 stores the data managed by the database management device 1 in the database 52 of the storage device 51 in its own device. The storage device 5 may be configured as a storage device 51 in which a storage medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive) is incorporated as a storage medium for storing the database 52, or read / write. A storage device 51 to which a storage medium such as a possible DVD (Digital Versatile Disc) drive is detachable may be configured.

  The backup device 8 is, for example, a storage device, but may be an external storage medium directly connected to the database management device 1 or a tape device. The backup device 8 stores the full backup data 82 a and the differential backup data 82 b of the data in the in-memory data buffer 15 managed by the database management device 1 in the storage device 81. The full backup data 82a is data for storing all data in the in-memory data buffer 15, and the differential backup data 82b is data for storing a part of data (updated data) in the in-memory data buffer 15. .

Further, the backup device 8 has a DB synchronization processing unit 83. The DB synchronization processing unit 83 differentially updates the differential backup data 82b to the database 52 (application of a so-called patch), thereby changing the data identification state (synchronization state) between the in-memory data buffer 15 and the database 52. Realize. The trigger for the differential update process may be executed every time one update occurs, or may be executed collectively after a plurality of updates are stored.
Thus, by executing the differential update process of the storage apparatus 5 from the backup data of the backup apparatus 8, the load on the database management apparatus 1 can be reduced.

The terminal 7 receives a processing request for executing a business process via an input unit that receives an input from the user. The business process is classified into an online process 91 (see FIG. 2) executed mainly during business hours and a batch process 93 (see FIG. 2) executed mainly outside business hours.
The main difference between the online process 91 and the batch process 93 is the data access destination. In the online processing 91, data access is executed with respect to the database 52 of the storage apparatus 5. In the batch process 93, data access to the in-memory data buffer 15 of the database management apparatus 1 is performed at high speed.

The database management device 1 manages data stored in each database (data in the database 52, the in-memory data buffer 15, and the storage device 81), receives a data access request from the terminal 7, and receives data to be managed. Control reading and writing.
First, in order to execute the online processing 91, the terminal 7 provides a transaction with the database management apparatus 1. When data access to the database 52 of the storage apparatus 5 becomes necessary in the course of executing online processing, a data access request is transmitted to the database management apparatus 1 via the transaction. The access request is described according to, for example, the syntax of SQL (Structured Query Language).

  The online request analysis unit 11 of the database management apparatus 1 that has received the access request parses the access request and notifies the online processing unit 12 of the result. The online processing unit 12 processes the data access content indicated by the access request with respect to the database 52 of the storage apparatus 5.

  On the other hand, in order to execute the batch process 93, the terminal 7 transmits a data access request or the like to the database management apparatus 1. The batch request analysis unit 13 of the database management apparatus 1 that has received the access request parses the access request and notifies the batch processing unit 14 of the result. The batch processing unit 14 processes the contents of the data access indicated by the access request with respect to the in-memory data buffer 15 of the own device. Since the data access to the in-memory data buffer 15 is completed within the own device and uses a high-speed access medium called a memory device, the data access is generally faster than the data access to the database 52 of the storage device 5. Access can be expected.

The DB operation processing unit 20 accepts various requests related to database operation management transmitted by the terminal 7 or the like. Examples of the various requests include a load processing request, a backup processing request, a synchronization processing request, and a recovery processing request.
The DB load processing unit 21 processes a “load processing request” for reading the data contents of the database 52 into the in-memory data buffer 15.
The DB backup processing unit 22 processes a “backup processing request” for writing the updated contents of the in-memory data buffer 15 to the differential backup data 82b.
The DB synchronization instruction unit 23 processes a “synchronization processing request” for giving the DB synchronization processing unit 83 an instruction to update the differential backup data 82 b to the database 52.
The DB recovery processing unit 24 processes a “recovery processing request” for recovering from at least one of the database 52, the full backup data 82a, and the differential backup data 82b for the data contents of the in-memory data buffer 15. To do.

FIG. 2 is an explanatory diagram showing the main data flow in the processing of the database management system.
The online processing 91 for business processing is processing for reflecting an access request from the terminal 7 in the database 52, and is executed by the online processing unit 12.
The load process 92 (in-memory process) is a process for reading the data contents of the database 52 subjected to the online process 91 into the in-memory data buffer 15 as it is, and is executed by the DB load processing unit 21.
The business process batch process 93 is a process for reflecting an access request from the terminal 7 in the in-memory data buffer 15 subjected to the load process 92, and is executed by the batch processing unit 14.

The backup process 94 (difference data transmission process) is a process of writing the difference data updated by the batch process 93 out of the data contents of the in-memory data buffer 15 as the difference backup data 82b, and is executed by the DB backup processing unit 22. Is done.
The synchronization process 95 (difference data reflection process) is a process for reflecting the differential backup data 82b subjected to the backup process 94 in the database 52, and is executed by the DB synchronization processing unit 83 that has received an instruction from the DB synchronization instruction unit 23. Is done.
The recovery process 96 is a process for recovering the data contents of the in-memory data buffer 15 based on the differential backup data 82b subjected to the backup process 94 when the data contents of the in-memory data buffer 15 are lost due to a failure or the like. Yes, and executed by the DB recovery processing unit 24.

  FIG. 3 is an explanatory diagram showing the execution order of the processes described in FIG. This explanatory diagram is described in a Gantt chart format with the horizontal axis as the time axis, and in the vertical direction, each process is indicated by a solid line arrow or a dotted line arrow. The execution order of each process will be described with reference to FIGS. 4 to 7 as appropriate.

  The online processing 91 (O1) executed up to time t0 is processing executed by the online processing unit 12 during business hours, and the processing result is written in the database 52. Note that the notation such as “(O1)” described in each data structure in the figure indicates the processing result of the data structure.

The backup process (O1) executed at time t0 to time t1 is a process in which the DB backup processing unit 22 writes the data content as a result of the online process 91 (O1) from the database 52 to the full backup data 82a. . For example, in FIG. 4, all data with page IDs = 1 to 8 are written.
The in-memory data buffer 15, the full backup data 82a, and the differential backup data 82b (see FIG. 5) are storage areas for storing data contents of the database, respectively. The “ID” and the “page content” are managed in association with each other. Here, in FIG. 4, one integer value (for example, “50”) is set as one page content as a page content. However, this is merely an example for explanation, and in fact, one page has a fixed length or a variable length. Various data (binary data, text data, etc.) handled by the database are stored. The update management table 16 stores a “flag” indicating whether or not each page of the in-memory data buffer 15 is updated.

  The load process 92 (O1) executed from time t1 to time t2 is a process in which the DB load processing unit 21 reads the data content as a result of the online process 91 (O1) from the database 52 to the in-memory data buffer 15. It is. For example, in FIG. 4, all data with page IDs = 1 to 8 are read. Along with the load processing 92 to the in-memory data buffer 15, the DB load processing unit 21 initializes all the flags of the update management table 16 to “unupdated” (see FIG. 4).

  In the batch process 93 (B1) executed from time t2 to time t3, the batch processing unit 14 receives the analysis result by the batch request analysis unit 13 of the access request from the terminal 7, and the data content of the in-memory data buffer 15 Is a process of updating. For example, in FIG. 5, four pages of page ID = P2, P4, P5, and P7 are updated by the batch process 93. As the page is updated, the batch processing unit 14 changes the flag corresponding to the updated page of the update management table 16 to “update”.

  The backup process 94 (B1) executed from time t3 to time t4 is a process in which the DB backup processing unit 22 writes the updated contents of the in-memory data buffer 15 by the batch process 93 (B1) to the differential backup data 82b. . For example, in FIG. 5, the page contents of the updated four page IDs = P2, P4, P5, and P7 are written to the differential backup data 82b.

  In the synchronization process 95 (B1) executed from time t4 to time t5, the DB synchronization instruction unit 23 and the DB synchronization processing unit 83 use the online backup 91 (B1) to store the differential backup data 82b of the batch process 93 (B1). This is a process of reflecting the data contents of the database 52 as a result of O1). For example, in FIG. 6, the page contents of four pages of page ID = P2, P4, P5, and P7 are reflected in the database 52.

  Batch processing 93 (B2) executed from time t4 to time t6 is processing in which the batch processing unit 14 updates the data content of the in-memory data buffer 15. For example, in FIG. 5, three pages with page IDs = P5, P7, and P8 are updated by the batch process 93 (B2). As the page is updated, the batch processing unit 14 changes the flag corresponding to the updated page of the update management table 16 to “update”.

  Here, referring to FIG. 6, the synchronization processing 95 (B1) and the batch processing 93 (B2) are different in data to be accessed from each other, so that data collision does not occur and parallel processing is possible. It is. In other words, the synchronization process 95 (B1) and the batch process 93 (B2) are processed in parallel with each other starting from time t4, so that both processes are processed in series without parallel processing. The total processing time can be shortened.

  The backup process 94 (B2) executed from time t6 to time t7 is a process in which the DB backup processing unit 22 writes the updated contents of the in-memory data buffer 15 by the batch process 93 (B2) to the differential backup data 82b. . For example, in FIG. 6, the page contents of the three updated page IDs = P5, P7, and P8 are written to the differential backup data 82b.

  In the synchronization process 95 (B2) executed from time t7 to time t8, the DB synchronization instructing unit 23 and the DB synchronization processing unit 83 store the differential backup data 82b of the batch process 93 (B2) in the batch process 93 ( This is a process of reflecting the data content of the database 52 as a result of B1). For example, in FIG. 7, the page contents of three pages with page IDs = P5, P7, and P8 are reflected in the database 52.

  Online processing 91 (O2) executed from time t8 is processing executed by the online processing unit 12 during business hours, and the processing result is written in the database 52.

  6 and 7, the differential backup data 82b is created for each batch process 93 (B1, B2). Therefore, the DB synchronization processing unit 83 compares the differential backup data 82b corresponding to a plurality of batch processes 93, and when it finds a page in which the updates overlap (for example, page ID = P5, P7), it overwrites the data. In order to suppress the occurrence of this, updating of the page corresponding to the previous time zone (for example, P5 = 40, P7 = 10) may be omitted.

  FIG. 8 is an explanatory diagram showing an outline of the data recovery processing 96. This explanatory diagram is described in a Gantt chart format with the horizontal axis as the time axis, as in FIG. The processing in the time period from time t0 to time t5 in FIG. 8 is the same as that in FIG.

  At time t12, when the batch process 93 (B2) is being executed, a failure occurs in the batch process 93 (B2). The DB backup processing unit 22 receives the notification of the occurrence of the failure from the online processing unit 12, and performs the recovery processing 96 on the contents of the in-memory data buffer 15 until the time point of the result of the batch processing 93 (B1).

In FIG. 9, for example, examples of the recovery processes 96 (1) to (3) are shown.
In the recovery process 96 (1), the full backup data 82a is copied to the in-memory data buffer 15 as it is, thereby recovering to the time point of the result of the online process 91 (O1).
In the recovery process 96 (2), after the recovery process 96 (1), the differential backup data 82b, which is the execution result of the batch process 93 (B1), is reflected in the in-memory data buffer 15, so that the batch process 93 (B1 ) Recovers to the point of the result.
In the recovery process 96 (3), after the recovery process 96 (2), the differential backup data 82b, which is the execution result of the batch process 93 (B2), is reflected in the in-memory data buffer 15, so that the batch process 93 (B2 ) Recovers to the point of the result.

In this way, the DB backup processing unit 22 recovers the data contents of the in-memory data buffer 15 using the full backup data 82a and the differential backup data 82b at each time point.
For example, from time t12 to time t13 in FIG. 8, the DB backup processing unit 22 performs the recovery process 96 (2) after the recovery process 96 (1), thereby processing the contents of the in-memory data buffer 15 in a batch process. The recovery process 96 is performed up to the time point of the result of 93 (B1).
Returning to FIG. 8, after the recovery process, at times t <b> 13 to t <b> 14, the batch processing unit 14 again performs the batch process 93 (B <b> 2) on the recovered in-memory data buffer 15. The subsequent operation of FIG. 8 is the same as that of FIG.

FIG. 10 is a flowchart showing backup processing of each batch processing result executed by the DB backup processing unit 22.
In S101, a loop for sequentially selecting page IDs in the update management table 16 is started. Hereinafter, the page selected in this loop is referred to as a selected page.
In S102, it is determined whether or not the “flag” of the selected page in the update management table 16 is “update”. If Yes in S102, the process proceeds to S103, and if No, the process proceeds to S104.
In S103, “page content” in the selected page of the in-memory data buffer 15 is written out as differential backup data 82b, and the “flag” of the selected page in the update management table 16 is returned to “unupdated”.
In S104, the loop from S101 is terminated.
In the processing of FIG. 10 described above, the amount of disk writing can be reduced compared to the method of writing all pages by the differential format backup processing 94 that limits the pages to be written to updated pages.

FIG. 11 is a flowchart showing the recovery process 96 when a failure occurs, which is executed by the DB recovery processing unit 24.
In S201, it is determined whether or not the synchronization process 95 that is being processed in parallel with the batch process 93 at the time of failure has been completed. If Yes in S201, the process proceeds to S211. If No in S201, the process proceeds to S221.
In S211, the data content in the in-memory data buffer 15 is recovered by loading the data content in the synchronized database 52 into the in-memory data buffer 15 as it is. Then, the process proceeds to S231.
In S221, the data content of the full backup data 82a is loaded 92 into the in-memory data buffer 15 as it is, so that the data content in the in-memory data buffer 15 is recovered to a point before the batch processing.
In S222, the data contents of the differential backup data 82b are subjected to differential reflection processing to the in-memory data buffer 15 in the order of the oldest time, so that the data contents of the in-memory data buffer 15 are executed at the time of the failure. Recovery is made until the end of the previous batch processing 93. Then, the process proceeds to S231.
In S231, the batch processing unit 14 is instructed to restart the batch processing 93 when a failure occurs from the beginning.

According to the present embodiment described above, the predetermined batch process 93 (for example, B2) and the previous batch process result (for example, B1) synchronization process 95 are processed in parallel, thereby reducing the processing time. The sum can be shortened.
Furthermore, the processing time of the synchronization process 95 can be shortened by managing the batch processing results in a differential format called differential backup data 82b.

DESCRIPTION OF SYMBOLS 1 Database management apparatus 1a Processor 1b Memory 1c Interface 5 Storage apparatus 6 Network 7 Terminal 8 Backup apparatus 10 DB operation processing part 11 Online request analysis part 12 Online processing part 13 Batch request analysis part 14 Batch processing part 15 In-memory data buffer 16 Update Management table 20 DB operation processing unit 21 DB load processing unit 22 DB backup processing unit 23 DB synchronization instructing unit 24 DB recovery processing unit 51 storage device 52 database 81 storage device 82a full backup data 82b differential backup data 83 DB synchronization processing unit 91 Online processing 92 Load processing 93 Batch processing 94 Backup processing 95 Synchronization processing 96 Recovery processing

Claims (5)

A database management method in a database management system comprising a database management device for managing data of a database stored in a storage device and having a memory ,
The database management device includes :
Executing a load process for reading the data of the database stored in the storage device into the memory;
Wherein for the load processed data, analyzes the access request received, it performs batch processing according to the result of the analysis of the access request,
The data that is updated in the memory that has been processed by said batch processing, performs the backup process to write to the backup device as backup data after the execution of the batch process,
The backup device is
In response to an instruction issued from the database management device every time the backup data is stored a plurality of times, the stored backup data is reflected in the data of the database stored in the storage device. A synchronization process for making the above data and the data stored in the storage device identical,
When the database management device newly performs the batch processing , the backup is performed on the backup data that has been backed up with respect to the batch processing and another batch processing executed before the batch processing . wherein the synchronization process and are database management method comprising Rukoto is a parallel processing executed by the device. The backup device , when there is an update content at the same location for a plurality of the backup data generated by the backup processing multiple times, among the update content at the same location, the backup whose time comes first The database management method according to claim 1, wherein the data synchronization process is omitted. The database management equipment, said a newly performs batch processing, in parallel processing of the synchronization process, before Fang Tsu failure occurs to blood processing, and the pre-synchronization in Kiba pitch processing 2. The load process for reading data contents stored in the storage device identified by the completed synchronization process into the memory when the process is completed is executed. Or the database management method of Claim 2. The database management equipment, said a newly performs batch processing, in parallel processing of the synchronization process, before Fang Tsu failure occurs to blood processing, and the pre-synchronization in Kiba pitch processing 3. The database management method according to claim 1, wherein when the conversion process is not completed, the backup data already written in the backup process is reflected in the memory. A database management system comprising a database management device for managing data of a database stored in a storage device and having a memory ,
The database management device includes :
Executing a load process for reading the data of the database stored in the storage device into the memory;
Wherein for the load processed data, analyzes the access request received, it performs batch processing according to the result of the analysis of the access request,
The data that is updated in the memory that has been processed by said batch processing, performs the backup process to write to the backup device as backup data after the execution of the batch process,
The backup device is
In response to an instruction issued from the database management device every time the backup data is stored a plurality of times, the stored backup data is reflected in the data of the database stored in the storage device. A synchronization process for making the above data and the data stored in the storage device identical,
When the database management device newly performs the batch processing , the backup is performed on the backup data that has been backed up with respect to the batch processing and another batch processing executed before the batch processing . database management system and the synchronization process to be performed, characterized by Rukoto is parallel processing by the device.

Images