WO2017060941A1

WO2017060941A1 - Data management system and data management method which maintain consistency between plurality of data

Info

Publication number: WO2017060941A1
Application number: PCT/JP2015/078133
Authority: WO
Inventors: 有哉礒田
Original assignee: 株式会社日立製作所
Priority date: 2015-10-05
Filing date: 2015-10-05
Publication date: 2017-04-13
Also published as: JPWO2017060941A1

Abstract

Each time the data management system according to the present invention receives a transaction (Tx) processing instruction, the data management system: (A) registers a nonpersistent state in a storage space as a state associated with transaction processing specified by the received instruction; and then (B) changes the state associated with the transaction processing to a persistent state when the data management system completes the transaction. This allows all data for the Tx processing to be made persistent at once.

Description

Data management system and data management method for maintaining consistency of plural data

The present invention generally relates to data management that maintains the consistency of a plurality of data.

As an environment (system) that needs to maintain the consistency of multiple data, for example, there is a DBMS (Database Management System).

DBMS controls the consistency and persistence of records that are input and output data. As a result, the user can always input / output the intended record. Such DBMS consistency control is called Tx (transaction) processing. The Tx processing is used when a plurality of instructions (reference instruction and update instruction) are realized atomically. In Tx processing, a plurality of instructions in a specified section are processed atomically by declaring the beginning and end (Tx processing section) of the instruction section to be executed atomically. When Tx processing commits, the result is persisted to an external storage medium such as storage.

A ACID characteristics must be satisfied to achieve Tx processing. The ACID characteristic indicates atomicity, consistency, isolation, and durability. Atomicity means that a plurality of instructions in the Tx processing section are all executed or not executed at all. Consistency means not making a transition to a state other than the rules given to the database. Independence means that the result of Tx processing is always the same as when Tx processing is executed sequentially. With persistence, if the Tx process is committed, the consistency of the Tx process is maintained by satisfying the ACID characteristic.

Also, the DBMS is required to have performance and availability.

The performance is expected to be improved by the technique disclosed in

Patent Document

1 or 2, for example. According to Patent Document 1, performance improvement can be expected by an in-memory DBMS that expands a database or an index in a server memory and performs Tx processing. However, log processing also occurs in the in-memory DBMS, and log processing becomes a performance issue. According to the technique of Patent Document 2, it is expected to solve the problem of log processing performance. According to Patent Document 2, it is possible to refer to an uncommitted result by defining an uncommitted state before committing. As a result, for example, Tx を 2 can refer to the commit result of Tx 1 without waiting for the completion of the log processing of Tx 1 for the processing that was executing Tx 2 after Tx 、 1, It can be expected to hide the log processing.

Availability is expected to be improved by the technology disclosed in Patent Document 3, for example. According to Patent Document 3, an improvement in availability is expected by a log-linked HA (HighabilityAvailability) configuration. The HA configuration is system multiplexing, and includes an execution system that performs Tx processing and a standby system that synchronizes the execution system. Such an HA configuration is expected to improve availability by switching to the standby system immediately in the event of a failure of the active system.

US2,011 / 0252000 US8407195 JP 2013-84293 A

Improvements in performance and availability are expected by the techniques disclosed in

Patent Documents

1, 2, and 3. However, it is impossible to achieve both performance and availability in a single system. For example, the in-memory DBMS realizes high performance by expanding a database (for example, an index and a table) in a volatile memory. However, at the time of failure recovery, in general, the process of expanding (reading) a database from storage (typically a disk) to volatile memory, and the process of subsequently reading the log file from storage and reflecting the read log file to the database on the memory Is required and availability is low. On the other hand, it is conceivable that the database is not read at the time of failure recovery. However, in this case, it is necessary to frequently access the storage in order to expand the necessary data to the volatile memory by the Tx process, and the performance deteriorates.

It is desirable to achieve both performance and availability in a single system. For example, it is desired to perform failure recovery of the in-memory DBMS at high speed. In the in-memory DBMS, processing for expanding data from the storage into the memory and processing for reading the log file from the storage and adapting to the data on the memory are problems.

Such a problem is not limited to the DBMS, but may be present in other types of environments (systems) that need to maintain the consistency of a plurality of data, such as a file system or KVS (Key Value Store).

Each time the data management system receives a Tx (transaction) processing instruction,
(A) A state associated with the transaction processing according to the received instruction, which is a non-persistent state (a state meaning non-permanent) is registered in the storage space,
(B) When the transaction is terminated, the transaction processing state is changed to a permanent state (a state meaning permanent).

∙ Can achieve both performance and availability in a single system.

1 shows a configuration of an entire system according to an embodiment. A storage space and information stored in the storage space are shown. It is a functional block diagram of a management program. It is a flowchart of TxMS. It is a flowchart of TxMS. It is a flowchart of a NVMS_STATE management system. It is a flowchart of NVMS_DATA management system. It is a sequence diagram of the process which perpetuates several data atomically. The detailed example of NVMS_STATE, NVMS_DATA, and NVMS_TABLE shown in FIG. 2 is shown. Of the flowchart of the NVMS_DATA management system, this is a part including a difference from the flowchart shown in FIG. It is a part including the difference with the sequence diagram shown in FIG. 8 among the sequence diagrams of the process which perpetuates several data atomically. A detailed example of NVMS_STATE, NVMS_DATA, and NVMS_TABLE shown in FIG. 9 and SYSTEM_PARAM, which is further information stored in the storage space, are shown. Indicates further functions that NVMS has. It is a flowchart of a NVMS_START management system. Of the flowchart of the NVMS_STATE management system, this is a part including a difference from the flowchart shown in FIG. Of the flowchart of the NVMS_DATA management system, this is a part including a difference from the flowchart shown in FIG. 7 (or the flowchart including the part shown in FIG. 10). It is a sequence diagram of starting and ending.

Hereinafter, embodiments will be described with reference to the drawings, taking a DBMS (in particular, an in-memory DBMS) as an example of a data management system.

In the following description, processing may be described using a program as a subject. However, when the program is executed by a processor (for example, a CPU (Central Processing Unit)), a predetermined processing is appropriately performed in a memory and / or an interface. Since the processing is performed using a device (for example, a communication port), the subject of processing may be processing performed by a processor (or an apparatus or system having the processor). The processor may include a hardware circuit that performs a part or all of the processing. The program may be installed in a computer-like device from a program source. The program source may be, for example, a storage medium that can be read by a program distribution server or a computer. When the program source is a program distribution server, the program distribution server may distribute the program to be distributed to other computers.

In the following description, “transaction” is referred to as “Tx”. Processing performed by executing Tx is referred to as “Tx processing”. In addition, the term “SQL processing” can be used as a word meaning any one of generation processing (CREATE), insertion processing (INSERT), update processing (UPDATE), and delete processing (DELETE). The Tx process includes a begin process, one or more SQL processes, and a commit process.

Hereinafter, first, the basic part of the embodiment will be described with reference to FIGS. Next, with reference to FIGS. 9 to 11, a description will be given of processing for simultaneously updating, inserting or deleting a plurality of data. Finally, with reference to FIG. 12 to FIG. 17, a description will be given of processing for normal activation and termination.

FIG. 1 shows the configuration of the entire system according to the embodiment.

The computer system 110B can receive commands from the user 100 through the computer system 110A and the network 150. The computer system 110B may receive a command without going through the computer system 110A and the network 150. The computer system 110B can be connected to a plurality of computer systems 110A.

The computer system 110B has one or more nodes, for example, two nodes 130 and 140. These nodes 130 and 140 may be connected via a system bus 151. The external storage device 120 may be connected to the computer system 110B via the network 150.

The configuration of each of the one or more nodes may be the same. For example, the node 130 includes a memory 132, an internal storage device 133, an input / output device 134, and a processor 131 connected to them. Similarly, the node 140 includes a memory 142, an internal storage device 143, an input / output device 144, and a processor 141 connected thereto. In the following description, the node 130 is mainly taken as an example, but other nodes such as the node 140 may have the same function and configuration. The computer system 110B (or any one node) may be an example of a data management system.

The input / output device 134 may be a communication interface device that communicates with an external device such as the external storage device 120, a human machine interface device (eg, an input device such as a keyboard and a pointing device, and an output such as a display device). Device). The memory 132 may include at least one of a volatile memory such as DRAM (Dynamic Random Access Memory) and a non-volatile memory (hereinafter referred to as NVM) such as a flash memory. A program such as a DBMS is stored in the memory 132, and the program is executed by the processor 131.

FIG. 2 shows a storage space and information stored in the storage space.

The storage space 200 is based on at least the memory 132 among the memory 132, the internal storage device 133, and the external storage device 120. The storage space 200 includes a volatile space 201 and a nonvolatile space 202.

The volatile space 201 is a space based on a volatile storage area (for example, at least a part of the DRAM) in the memory 132. The non-volatile space 202 is a space based only on NVM (for example, at least a part of NVM) in the memory 132, but instead of or in addition, the internal storage device 133 (for example, SSD (Solid State Drive) or HDD ( Hard Disk Drive)) and an external storage device 120 may include a space based on at least a part thereof. However, it is possible to intentionally arrange the volatile space 201 in the NVM by setting the program. Hereinafter, regarding the operation of the storage space 200, it is assumed that the non-volatile space 202 is the operation target unless otherwise mentioned. The volatile space 201 may be used as a cache for the nonvolatile space 202.

The storage space 200 stores information such as NVMS_STATE 210, NVMS_DATA 220, and NVMS_TABLE 230. NVMS_STATE 210 is an example of state management information. NVMS_DATA 220 is an example of data management information. NVMS_TABLE 230 is an example of transaction management information.

NVMS_STATE 210 is information indicating the state of the processing target data (for example, a record in the database) in the Tx process for each Tx process. For example, NVMS_STATE 210 has an entry for each Tx process. ID 211 and STATE 212 are registered in each entry. The ID 211 is an ID (for example, an identification number) for Tx processing. STATE 212 is information indicating the state of data to be processed in the Tx process. Examples of the value of STATE 212 include “PROCESSING”, “COMMIT”, “ABORT”, and the like. “PROCESSING” means that since the Tx process is uncommitted, all data to be processed in the Tx process may be deleted from the nonvolatile space 202. On the other hand, “COMMIT” means that since the Tx process has been committed, all data to be processed in the Tx process must not be deleted from the nonvolatile space 202. According to the entry 213, the state of the processing target data in Tx 1 (Tx processing with ID “1”) is “PROCESSING”.

NVMS_DATA 220 is information indicating, for each data, a correspondence relationship between data in Tx processing (processing target data) and STATE 212 associated with the data. For example, NVMS_DATA 220 has an entry for each data in the Tx process. DATA 221 and STATE_pt 222 are registered in each entry. DATA 221 is data (or a duplicate thereof) in Tx processing. STATE_pt 222 is a pointer to STATE 212 associated with the data (information indicating the location of STATE 212 in NVMS_STATE 210). According to

entries

223, 224, and 225, each of data A, data B, and data C is data to be processed in Tx 1 (Tx processing with ID = 1), and is associated with STATE 212 “PROCESSING” in Tx 1. You can see that

NVMS_TABLE 230 is information indicating the correspondence between the Tx process and the data in the Tx process for each Tx process. For example, the NVMS_TABLE 230 has an entry for each Tx process. An ID 231 and one or more DATA_pts 232 are registered in each entry. The ID 231 is an ID for Tx processing. Each of DATA_pt 232 is a pointer to DATA 221 of data in Tx processing (information indicating the location of DATA 221 in NVMS_DATA 220). According to entry 235, the data at Tx 1 is data A, data B, and data C, which are associated with entries 223 to 225 in NVMS_DATA 220, respectively.

In the above description, the NVMS_TABLE 230 may be used when there is no function (hardware function or software function) for determining whether the NVMS_DATA 220 is written in the nonvolatile space 202. In other words, if there is a function for determining whether the NVMS_DATA 220 is written in the nonvolatile space 202, the NVMS_TABLE 230 may be omitted. STATE_pt 222 and DATA_pt 232 can be expressed using at least one of a unique number, an address address, an array number, and the like. Also, any of NVMS_STATE 210, NVMS_DATA 220, and NVMS_TABLE 230 may be stored in the nonvolatile space 202.

FIG. 3 is a functional block diagram of the management program.

The management program 240 is one of programs executed by the processor 131, and is a DBMS, for example. The management program 240 includes a Tx processing system 250 that controls the start and end of Tx processing, a TxMS (Transaction Management System) 260 that manages Tx processing, and an NVMS (Non-Volatile Management System) 270 that manages the nonvolatile space 202. And have. Each of these

elements

250, 260, and 270 may be a function (eg, a program module) of the management program 240. In addition, at

least elements

250 and 260 of these

elements

250, 260, and 270 may exist for each Tx process.

The NVMS 270 has an NVMS_STATE management system 271 and an NVMS_DATA management system 272. Each of these

elements

271 and 272 may be a function of NVMS 270 (eg, a program module).

TxMS 260 operates in conjunction with Tx processing system 250 and NVMS 270. The NVMS_STATE management system 271 manages the NVMS_STATE 210 and operates in conjunction with the NVMS_DATA management system 272. The NVMS_DATA management system 272 manages NVMS_DATA 220 and NVMS_TABLE 230 and operates in conjunction with the NVMS_STATE management system 271.

Such a management program 240 may exist in the storage space 200 or may be implemented as hardware.

Hereinafter, an example of data processing performed in the present embodiment will be described with reference to FIGS. In the description, one Tx process is taken as an example.

FIG. 4 is a flowchart of the Tx processing system 250.

The Tx processing system 250 starts (BEGIN) Tx processing in response to an instruction from the user 100 (for example, in response to an instruction from the computer system 110A) (S100). At this time, the Tx processing system 250 sends a start command (BEGIN) to the TxMS 260.

Next, the Tx processing system 250 generates a process according to an instruction from the user 100 (S110), and performs a process determination (S120). In S120, the Tx processing system 250 determines whether the processing instruction is COMMIT or ABORT. When it is determined that the processing instruction is neither COMMIT nor ABORT, the Tx processing system 250 performs processing execution A (S130). In S130, the Tx processing system 250 sends data to the TxMS 260 in response to the processing instruction. These processes are repeated until it is determined in S120 that the processing instruction is COMMIT or ABORT.

When it is determined that the processing instruction is COMMIT or ABORT, the Tx processing system 250 performs processing execution B (S140). In S140, the Tx processing system 250 sends data corresponding to the processing command (COMMIT or ABORT) to the TxMS 260.

Thereafter, the Tx processing system 250 ends (S150).

FIG. 5 is a flowchart of TxMS260.

The TxMS 260 starts by receiving a start command (BEGIN) from the Tx processing system 250 (S200). When receiving the processing instruction from the Tx processing system 250, the TxMS 260 determines the received processing instruction (S210).

If the processing instruction is BEGIN, the TxMS 260 sends a processing instruction (CREATE) to the NVMS_STATE management system 271 (S220). CREATE is an instruction to generate an NVMS_STATE entry (entry in NVMS_STATE 210) including the ID 211 and STATE 212 of the Tx process. After S220, TxMS 260 ends (S260).

When the processing command is INSERT, UPDATE, or DELETE, the TxMS 260 sends data corresponding to the processing command to the NVMS_DATA management system 272 (S230). Thereafter, the TxMS 260 ends (S260).

When the processing instruction is COMMIT, the TxMS 260 sends the processing instruction (CHECK) and the ID of the Tx process to the NVMS_DATA management system 272 (S240). CHECK is a command for confirming whether or not all data to be processed in the Tx process has been written in the nonvolatile space 202. After S240, the TxMS 260 sends a processing command (COMMIT) to the NVMS_STATE management system 271 (S250). After S250, the TxMS 260 sends a processing command (DELETE TABLE) to the NVMS_DATA management system 272 (S251). DELETE TABLE is an instruction to delete the NVMS_STATE entry corresponding to the Tx processing for which COMMIT (or ABORT) has been completed from the NVMS_STATE 210. After S251, the TxMS 260 ends (S260).

If the processing instruction is ABORT, the TxMS 260 sends the processing instruction (ABORT) and the ID of the Tx process to the NVMS_STATE management system 271 (S250), and then the TxMS 260 sends the processing instruction (DELETE TABLE) to the NVMS_DATA management system 272. (S251). Thereafter, the TxMS 260 ends (S260).

FIG. 6 is a flowchart of the NVMS_STATE management system 271.

The NVMS_STATE management system 271 starts when a processing command is received from the TxMS 260 (S300). The NVMS_STATE management system 271 determines the processing command received from the TxMS 260 (S310).

When the processing instruction is CREATE, the NVMS_STATE management system 271 secures an area for the NVMS_STATE entry (NVMS_STATE 210 entry) from the nonvolatile space 202 in accordance with the Tx processing (S320). Next, the NVMS_STATE management system 271 writes “PROCESSING” as the STATE 212 in the area (NVMS_STATE entry) secured in S320 (S321). Further, the NVMS_STATE management system 271 generates an ID for Tx processing (S322), and writes the generated ID as the ID 211 in the NVMS_STATE entry stored in S320 (S323). Next, the NVMS_STATE management system 271 confirms that the ID 211 and STATE 212 corresponding to the Tx process have been written in the NVMS_STATE entry (nonvolatile space 202) stored in S320 (S324). Thereafter, the NVMS_STATE management system 271 responds to the TxMS 260 with the written ID 211 and ends (S340).

If the processing instruction is COMMIT, the NVMS_STATE management system 271 searches the NVMS_STATE entry in which the same ID 211 as the received ID is registered from the NVMS_STATE 210 (S330), and writes “COMMIT” as the STATE 212 of the found NVMS_STATE entry (S331). ). That is, the NVMS_STATE management system 271 changes the STATE 212 of the Tx process to “COMMIT”. Thereafter, the NVMS_STATE management system 271 ends (S340).

When the processing instruction is ABORT, the NVMS_STATE management system 271 searches the NVMS_STATE entry in which the same ID 211 as the received ID is registered from the NVMS_STATE 210 (S330), and writes “ABORT” as the STATE 212 of the found NVMS_STATE entry (S331). ). Thereafter, the NVMS_STATE management system 271 ends (S340).

FIG. 7 is a flowchart of the NVMS_DATA management system 272.

The NVMS_DATA management system 272 starts when a processing command is received from the TxMS 260 (S400). The NVMS_DATA management system 272 determines the processing command received from the TxMS 260 (S410).

When the processing instruction is INSERT, the NVMS_DATA management system 272 secures an area for the entry part (DATA_pt 232) of the NVMS_TABLE entry including the received ID as the ID 231 from the nonvolatile space 202 (S420). Next, the NVMS_DATA management system 272 secures an area for the NVMS_DATA entry in which the INSERT target data is written as DATA 220 from the nonvolatile space 202 (S421). The NVMS_DATA management system 272 writes the location of the area (NVMS_DATA entry) secured in S421 as DATA_pt 232 in the entry part secured in S420 (S422). Then, the NVMS_DATA management system 272 writes the location of the NVMS_STATE entry including the received ID as the ID 231 as the STATE_pt 222 in the area (NVMS_DATA entry) reserved in S421, and the received data (INSERT target data) as the DATA 221. Write (S423). Thereafter, the NVMS_DATA management system 272 ends (S450).

When the processing instruction is CHECK, the NVMS_DATA management system 272 indicates that all data to be processed in Tx processing (that is, all data associated with all DATA_pt 232 in the NVMS_TABLE entry including the received ID as ID 231). It is confirmed (determined) whether data is written in the nonvolatile space 202 (S430). If all processing target data has been written, the NVMS_DATA management system 272 ends (S450). If at least one process target data is not yet written, the NVMS_DATA management system 272 waits for all the process target data to be written in the nonvolatile space 202.

When the processing instruction is DELETE TABLE, the NVMS_DATA management system 272 deletes the NVMS_TABLE entry including the received ID as the ID 211 from the NVMS_TABLE 230 (S440). Thereafter, the NVMS_DATA management system 272 ends (S450).

FIG. 8 is a sequence diagram of processing for atomically persisting multiple data. In the sequence diagram, the instruction is expressed as “Com.”.

The Tx processing system 250 sends a start command (BEGIN) to the TxMS 260 (300). The TxMS 260 starts in response to the received start command (BEGIN), and sends a processing command (CREATE) to the NVMS_STATE management system 271 (301). The NVMS_STATE management system 271 receives a processing command (CREATE). The NVMS_STATE management system 271 secures an NVMS_STATE entry in the nonvolatile space 202 in accordance with the received processing instruction (CREATE), generates an ID for Tx processing, and sets ID 211 (generated ID) and STATE 212 “PROCESSING” in the NVMS_STATE entry. Write (that is, write the ID 211 and STATE 212 to the nonvolatile space 202). Thereafter, the NVMS_STATE management system 271 returns the ID written in the nonvolatile space 202 to the TxMS 260. The TxMS 260 receives the ID from the NVMS_STATE management system 271 and returns the ID to the Tx processing system 250.

The Tx processing system 250 receives the started Tx processing ID from the TxMS 260 after transmitting the start command (BEGIN). The Tx processing system 250 sends the SQL command (INSERT), the INSERT target data, and the ID received from the TxMS 260 to the TxMS 260 (310). The TxMS 260 receives the SQL command (INSERT), data, and ID. In accordance with the received SQL command (INSERT), the TxMS 260 sends the processing command (INSERT) and the received data and ID to the NVMS_DATA management system 272 (311). The NVMS_DATA management system 272 receives a processing command (INSERT), data, and ID. In accordance with the received processing command (INSERT), the NVMS_DATA management system 272 stores DATA221 (received data) and STATE_pt222 (pointer to the NVMS_STATE entry corresponding to the received ID) in the NVMS_DATA entry (nonvolatile space 202) of the received data. And write. Note that the Tx processing system 250 and the TxMS 260 do not have to wait for the NVMS_STATE management system 271 to end the processing.

The Tx processing system 250 sends the end command (COMMIT) and the ID of the Tx process targeted for COMMIT to the TxMS 260 (320). The TxMS 260 receives the end command (COMMIT) and the ID. In accordance with the received end command (COMMIT), the TxMS 260 sends the processing command (CHECK) and the received ID to the NVMS_DATA management system 272 (321). The NVMS_DATA management system 272 receives the end command (COMMIT) and ID. The NVMS_DATA management system 272 confirms that all the processing target data (all DATA_pt 232) related to the received ID is written in the nonvolatile space 202 in accordance with the received processing command (CHECK). If the confirmation is obtained, the NVMS_DATA management system 272 sends a completion notification to the TxMS 260. The TxMS 260 receives the completion notification and sends the processing command (COMMIT) and the ID of the Tx processing to the NVMS_STATE management system 271 (322). The NVMS_STATE management system 271 receives the processing command (COMMIT) and ID. The NVMS_STATE management system 271 changes the STATE 212 corresponding to the received ID to “COMMIT” in accordance with the received processing command (COMMIT). With this change, the perpetuation of all the processing target data (DATA 221) in the Tx processing is completed.

The NVMS_STATE management system 271 sends a completion notification to the TxMS 260. The TxMS 260 receives the completion notification and sends the processing command (DELETE TABLE) and the Tx processing ID to the NVMS_DATA management system 272 (323). The NVMS_STATE management system 271 that has received the notification of completion of the processing command (COMMIT) sends the processing command (DELETE TABLE) after the commit because the NVMS_TABLE entry corresponding to the committed Tx processing (to each data in the committed Tx processing) (NVMS_TABLE entry including DATA_pt 232) is unnecessary (the same can be said for abort, so a processing instruction (DELETE TABLE) may be sent after abort). The NVMS_DATA management system 272 receives a processing command (DELETE TABLE) and ID. The NVMS_DATA management system 272 deletes the NVMS_TABLE entry including the same ID 211 as the received ID according to the received processing command (DELETE TABLE), and sends a completion notification to the TxMS 260. The TxMS 260 receives the completion notification and sends the completion notification to the Tx processing system 250.

These processes realize atomic persistence of multiple data.

When the Tx processing is canceled halfway, the Tx processing system 250 sends the processing command (ABORT) and the Tx processing ID to the TxMS 260 (330). The TxMS 260 receives the processing instruction (ABORT) and ID, and sends the processing instruction (ABORT) and ID to the NVMS_STATE management system 271 based on the processing instruction (ABORT) (331). The NVMS_STATE management system 271 receives the processing instruction (ABORT) and the ID, and changes the STATE 212 corresponding to the ID to “ABORT” according to the processing instruction (ABORT). This change cancels the persistence of all data to be processed in Tx processing. The NVMS_STATE management system 271 sends a completion notification to the TxMS 260. The TxMS 260 receives the completion notification and sends a processing command (DELETE TABLE) and ID to the NVMS_DATA management system 272 (332). The NVMS_DATA management system 272 receives the processing command (DELETE TABLE) and ID, deletes the NVMS_TABLE entry including the received ID as the ID 221 according to the processing command (DELETE TABLE), and sends a completion notification to the TxMS 260. The TxMS 260 receives the completion notification and sends the completion notification to the Tx processing system 250.

In a series of Tx processes, the management program 240 uses NVMS_DATA 220 (for example, first process target data (DATA 221) in the Tx process) corresponding to the Tx process in order to clearly indicate the state of the process target data (DATA 221) of the Tx process. Before writing to the nonvolatile space 202, the ID 211 and STATE 220 corresponding to the Tx process are written to the nonvolatile space 202.

FIG. 9 shows a detailed example of the NVMS_STATE 210, NVMS_DATA 220 and NVMS_TABLE 230 shown in FIG.

Each configuration (information item) of NVMS_STATE 210 and NVMS_TABLE 230 is as shown in FIG.

In each entry of NVMS_DATA 220, FLAG 224 and NEXT_pt 225 are written in addition to DATA 221 and STATE_pt 222. The FLAG 224 indicates the type of specific processing target data (DATA 221). For example, if the processing target data is deletion target data, FLAG 224 is “DELETE”, and if the processing target data is unnecessary data, FLAG 224 is “GARBAGE”. NEXT_pt 225 represents the location of the data next to the processing target data corresponding to the NEXT_pt 225 (data after the next SQL processing (for example, after updating)). According to NVMS_DATA 220 shown in FIG. 9, a plurality of data in one Tx process are associated with one-way pointers (NEXT_pt), but may be associated with two-way pointers (NEXT_pt and PREV_pt).

FIG. 10 shows a part of the flowchart of the NVMS_DATA management system 272 that includes a difference from the flowchart shown in FIG. Hereinafter, differences from FIG. 7 will be mainly described.

When the received processing instruction is UPDATE, the NVMS_DATA management system 272 secures an area for the entry part (DATA_pt 232) of the NVMS_TABLE entry including the received ID as the ID 231 from the nonvolatile space 202 (S450). Next, the NVMS_DATA management system 272 secures an area for the NVMS_DATA entry in which the UPDATE target data (updated data) is written as DATA 220 from the nonvolatile space 202 (S451). The NVMS_DATA management system 272 writes the location of the area (NVMS_DATA entry) secured in S451 as DATA_pt 232 in the entry part secured in S450 (S452). Then, the NVMS_DATA management system 272 writes the location of the NVMS_STATE entry including the received ID as the ID 231 as the STATE_pt 222 in the area (NVMS_DATA entry) reserved in S451, and the received data (UPDATE target data) as the DATA 221. Write (S453). The NVMS_DATA management system 272 writes the location of the UPDATE target data (DATA 221) as NEXT_pt 225 of the pre-update data corresponding to the UPDATE target data (S454). Thereafter, the NVMS_DATA management system 272 ends (S440).

When the processing command is DELETE, the same processing as S450 to S454 is performed. The difference is, for example, that the UPDATE target data is DELETE target data and that “DELETE” is written as FLAG 224 corresponding to the DELETE target data in S453.

In order to refer to the latest DATA 221 in the Tx process, the NVMS_DATA management system 272 follows the NEXT_pt 225. If the FLAG 224 of the latest DATA 221 is not “DELETE” and the STATE 212 corresponding to the Tx processing is “COMMIT”, it means that reference with consistency is possible. If the FLAG 224 of the latest DATA 221 is “DELETE” and the STATE 221 corresponding to the Tx process is “COMMIT”, it means that the latest DATA 221 has been deleted. Further, if the FLAG 224 of the latest DATA 221 is “GARBAGE”, it means that the latest DATA 221 is a target of garbage collection (GC).

FIG. 11 is a part including a difference from the sequence diagram shown in FIG. 8 in the sequence diagram of the process of atomically persisting a plurality of data. Hereinafter, differences from FIG. 8 will be mainly described.

The Tx processing system 250 sends the SQL command (UPDATE), the location of the pre-update data (DATA_pt), the UPDATE target data, and the ID to the TxMS 260 (340). The TxMS 260 receives them, and sends the processing instruction (UPDATE), the location of the data before update (DATA_pt), the data to be updated, and the ID to the NVMS_DATA management system 272 according to the received SQL instruction (UPDATE) (341) ). The NVMS_DATA management system 272 receives them, creates NVMS_DATA 220 according to the received processing instruction (UPDATE), and updates NEXT_pt 225 corresponding to the received DATA_pt to the location of the UPDATE target data (DATA 221). At this time, the Tx processing system 250 and the TxMS 260 do not have to wait for the NVMS_DATA management system 272 to end the processing.

The Tx processing system 250 sends the SQL command (DELETE), the location of the pre-update data (DATA_pt), and the ID to the TxMS 260 (350). The TxMS 260 receives them, and in accordance with the received SQL command (DELETE), sends the processing command (DELETE), the location of the pre-update data (DATA_pt), and the ID to the NVMS_DATA management system 272 (351). The NVMS_DATA management system 272 receives them and updates the NEXT_pt 225 corresponding to the received DATA_pt to the location of the DELETE target data (DATA 221) in accordance with the received processing command (DELETE). At this time, the Tx processing system 250 and the TxMS 260 do not have to wait for the NVMS_DATA management system 272 to end the processing.

This process makes it possible to update, insert or delete multiple data simultaneously.

When the UPDATE process is aborted, the NEXT_pt 225 of the pre-update data remains indicating the location of the UPDATE target data, but the STATE 221 identified from the STATE_pt 222 of the UPDATE target data becomes “ABORT”, so the management program 240 Shows that the UPDATE process failed. Furthermore, the management program 240 may initialize the NEXT_pt 225 of the pre-update data after the UPDATE process is aborted.

Further, when the commit is completed for the data after UPDATE or DELETE, the management program 240 may change the FLAG 224 of the data before update or before deletion to “GARBAGE”. The management program 240 may execute garbage collection on data corresponding to FLAG 224 “GARBAGE”.

FIG. 12 shows detailed examples of NVMS_STATE 210, NVMS_DATA 220 and NVMS_TABLE 230 shown in FIG. 9 and SYSTEM_PARAM 540 which is further information stored in the storage space 200.

Each configuration of NVMS_STATE 210, NVMS_DATA 220, and NVMS_TABLE 230 is the same as the configuration shown in FIG.

SYSTEM_PARAM 540 is stored in the nonvolatile space 202, for example. SYSTEM_PARAM 540 includes parameters necessary for the status and management of the system (for example, the computer system 110B or the management program 240). Specifically, for example, SYSTEM_PARAM 540 holds START_STATUS 541 for realizing normal activation and termination. START_STATUS 541 indicates whether the previous Tx process ends normally (“SUCCESS”) or abnormally ends (“FAILURE”). If START_STATUS 541 at the time of activation is “FAILURE”, it is understood that the previous termination was an abnormal termination (for example, an unintended termination or a termination failed).

FIG. 13 shows further functions that NVMS 270 has.

The NVMS 270 includes an NVMS_START management system 273 in addition to the NVMS_STATE management system 271 and the NVMS_DATA management system 272. The NVMS_START management system 273 manages activation and termination, and operates in cooperation with the NVMS_STATE management system 271 and the NVMS_DATA management system 272. The NVMS_START management system 273 guarantees normal startup and termination by controlling the START_STATUS 541 of the SYSTEM_PARAM 540.

FIG. 14 is a flowchart of the NVMS_START management system 273.

The NVMS_START management system 273 receives processing instructions such as start (START) and end (END) from the user 100 (computer system 110A) (S500). The NVMS_START management system 273 determines the received processing command (S510). The “activation” mentioned here is the activation of the system 273 (or the management program 240 including the system 273), and the “end” is the termination of the system 273 (or the management program 240 including the system 273).

If the processing instruction is START, the NVMS_START management system 273 determines whether or not the previous termination is normal (S520). Specifically, the failure determination is a determination of whether the START_STATUS 541 is “SUCCESS” or “FAILURE”.

When the START_STATUS 541 is “SUCCESS”, the NVMS_START management system 273 changes the START_STATUS 541 to “FAILURE” (S540), and ends the startup process (S550).

When the START_STATUS 541 is “FAILURE”, the NVMS_START management system 273 confirms and repairs the data abnormality as the area confirmation of the NVMS_STATE 210 (S521). Also, the NVMS_START management system 273 initializes the area of the NVMS_TABLE 230 (S522). Finally, the NVMS_START management system 273 maintains START_STATUS 541 at “FAILURE” (S540), and ends the startup process (S550). Even if S540 is not present, START_STATUS 541 remains “FAILURE”, so S540 may be skipped.

When the processing command is END, the NVMS_START management system 273 waits for the system termination processing (S530). When waiting for the system to end, the NVMS_START management system 273 confirms that at least the Tx process is not being executed. As a confirmation method during execution of Tx processing, for example, a method of confirming that all STATEs 212 in NVMS_STATE 210 are not “PROCESSING” or a method of confirming that there is no thread executing Tx processing can be cited. . If the end of the system is confirmed, the NVMS_START management system 273 changes the START_STATUS 541 to “SUCCESS” (S540) and ends the end process (S550).

FIG. 15 is a part including a difference from the flowchart shown in FIG. 6 in the flowchart of the NVMS_STATE management system 271. Hereinafter, differences from FIG. 6 will be mainly described.

The NVMS_STATE management system 271 determines the received processing instruction (S310). If the processing instruction is START or END, the NVMS_STATE management system 271 ends (S340).

If the processing instruction is UNDO, the NVMS_STATE management system 271 determines whether there is an undetermined STATE 212 in the NVMS_STATE 210 (S351).

If there is no undetermined STATE 212 as a result of S351, the NVMS_STATE management system 271 ends (S340).

If there is an undetermined STATE 212 as a result of S351, the NVMS_STATE management system 271 determines whether or not the STATE 212 is “PROCESSING” (S352). If the referenced STATE 212 is “PROCESSING”, the NVMS_STATE management system 271 changes the STATE 212 from “PROCESSING” to “ABORT” (S353). If the referenced STATE 212 is other than “PROCESSING”, S351 is executed again.

FIG. 16 shows a part of the flowchart of the NVMS_DATA management system 272 that includes a difference from the flowchart shown in FIG. 7 (or the flowchart including the part shown in FIG. 10). Hereinafter, differences from FIG. 7 (and FIG. 10) will be mainly described.

The NVMS_DATA management system 272 determines the received processing command (S410). If the processing instruction is START or END, the NVMS_DATA management system 272 ends (S440).

If the processing instruction is UNDO, the NVMS_DATA management system 272 initializes the NVMS_TABLE 230 (S460). This is because there is a possibility that unnecessary data may remain in the NVMS_TABLE 230 when the normal termination is not performed. By initialization of NVMS_TABLE 230, all entries (or only entries corresponding to Tx processing corresponding to STATE 212 “ABORT”) are deleted from NVMS_TABLE 230. After S460, the NVMS_DATA management system 272 ends (S440).

FIG. 17 is a sequence diagram of activation and termination.

The NVMS_START management system 273 is executed by a command from the user 100 (computer system 110A), and accepts START or END as a command.

When receiving a command (START) from the user 100 (360), the NVMS_START management system 273 determines whether the START_STATUS 541 is “SUCCESS” or “FAILURE”.

If START_STATUS 541 is “SUCCESS”, the NVMS_START management system 273 changes START_STATUS 541 to “FAILURE” and ends.

When the START_STATUS 541 is “FAILURE” (370), the NVMS_START management system 273 restores the NVMS_STATE 210 by sending a processing command (UNDO) to the NVMS_STATE management system 271 (371), and sends a processing command (UNDO) to the NVMS_DATA management system 272. The NVMS_TABLE 230 is initialized by sending (372). START_STATUS 541 is maintained at “FAILURE”. In the restoration (310) of the NVMS_STATE 210, the STATE 212 of the Tx process in which the STATE 212 is “PROCESSING” is changed to “ABORT”.

When receiving an instruction (END) from the user 100 (380), the NVMS_START management system 273 waits for the system to end (for example, waits for the end of all Tx processes). After detecting the system termination, the NVMS_START management system 273 changes START_STATUS 541 to “SUCCESS” and terminates.

Such processing makes it possible to realize normal startup and termination.

Hereinafter, the embodiments will be summarized. Note that in the general description, new descriptions such as modifications of the embodiment can be added as appropriate.

For example, an in-memory DBMS generally has a process of expanding (reading) a database from storage to a volatile memory and then reflecting a log file read and read from the storage to the database on the volatile memory at the time of failure recovery. I need it. It is conceivable that the database is not read at the time of failure recovery, but in that case, it is necessary to frequently access the storage in order to expand the necessary data into the memory by Tx processing.

Therefore, as Comparative Example 1, it is conceivable to place the database in NVM instead of storage. As a result, high-speed access can be expected, and as a result, high-speed processing for expanding the database into volatile memory or reflecting the log file to the database can be expected.

Furthermore, as Comparative Example 2, an in-memory DBMS may be adopted, and the database may be arranged in NVM instead of volatile memory. As a result, both the pre-update data and the post-update data remain in NVM, which is expected to eliminate the need for log files. By making the log file unnecessary, the process of reflecting the log file to the database at the time of failure recovery is eliminated, and as a result, the speed of failure recovery is expected.

However, since NVM cannot simultaneously write a plurality of data (for example, a plurality of data in a processor cache), there is a possibility that data consistency cannot be maintained. Specifically, for example, in the case where it is necessary to write data 1 to NVM and then write data 2 to NVM, a failure occurs after writing data 1 to NVM and before writing data 2 to NVM. Since data 2 is not written, inconsistency occurs. In other words, in the first comparative example, the data existing in the NVM is committed data, but in the second comparative example, uncommitted data also exists in the NVM. For this reason, it is difficult to adopt the comparative example 2, and it is considered that the NVM is adopted as an alternative storage device (that is, adopted as a log file storage destination) as in the comparative example 1.

Therefore, in the present embodiment, the management program 240 centrally manages a plurality of data states for each Tx process. That is, the management program 240 manages the state for each Tx process and associates a plurality of data with the Tx process. After writing a plurality of data to the non-volatile space 202 (for example, NVM), the management program 240 changes the state of Tx processing associated with the plurality of data from non-persistent (“PROCESSING”) to permanent (“COMMIT”) To do. Thereby, a plurality of data is made permanent at the same time regardless of the data amount.

In the present embodiment, the management program 240 observes the following two rules.
(Rule 1) It is necessary to write the state (“PROCESSING”) at the start of the Tx process in the nonvolatile space 202 before writing any data in the Tx process to the nonvolatile space 202. That is, after it is guaranteed that the state “PROCESSING” has been written in the nonvolatile space 202, it is necessary to write data in the Tx process into the nonvolatile space 202. This is because when data is first written in the nonvolatile space 202, the pointer (STATE_pt) indicating the destination of the data is unknown.
(Rule 2) When committing Tx processing, after confirming that all data related to the Tx processing has been written to the nonvolatile space 202, the status of the Tx processing is changed from “PROCESSING” to “COMMIT”. Need to be changed. Note that data need not be written into the nonvolatile space 202 every time SQL processing is performed. The data only needs to be in the non-volatile space 202 before committing the Tx process.

In this embodiment, the failure recovery process is completed simply by setting the state of the Tx process whose state is “PROCESSING” to “ABORT”.

As mentioned above, although one embodiment was described, it cannot be overemphasized that this invention can be variously changed in the range which is not limited to this embodiment and does not deviate from the summary. For example, the present invention can be applied to other types of environments (data management systems) that need to maintain the consistency of a plurality of data, for example, file systems, or KVS (Key Value Store).

110B: Computer system

Claims

A storage space including a non-volatile space;
A processor that accepts transaction processing instructions and writes data in the transaction processing according to the instructions to the non-volatile space,
Each time the processor receives a transaction processing instruction,
(A) A state associated with a transaction process according to the received instruction and writing a non-persistent state to the storage space;
(B) When ending the transaction process, change the transaction process state to a permanent state.
Data management system.
In (A), the processor writes a state associated with the transaction processing in the nonvolatile space before writing any data in the transaction processing according to the received instruction into the nonvolatile space.
The data management system according to claim 1.
The processor performs transaction processing every time data is inserted, updated or deleted.
In association with the state of the transaction processing written in the non-volatile space in (A) to the data,
Writing the data into the non-volatile space;
The data management system according to claim 2.
When the processor ends its transaction processing,
If all data in the transaction process is written in the non-volatile space, the state of the transaction process is changed to a permanent state.
The data management system according to claim 2.
The processor executes a process for changing each state of all target transaction processes to an abort state as a failure recovery process,
Each of all the target transaction processes is a transaction process associated with a state indicating a non-persistent state.
The data management system according to claim 4.
In the storage space, a system state indicating whether or not the previous termination has been normally performed is stored,
The processor executes the failure recovery process when the system state at the start indicates an abnormal state.
The data management system according to claim 5.
The non-volatile space stores state management information;
The state management information is information including a transaction process ID and a transaction process state for each transaction process.
The data management system according to claim 2.
The non-volatile space further stores data management information;
The data management information is information including, for each data in any transaction processing, data and an information element indicating a location in the state management information of a state of transaction processing for which the data is a processing target.
The data management system according to claim 7.
The data management information further includes, for each data in any transaction processing, an information element indicating the necessity of the data in the nonvolatile space, and an information element indicating the location of the data after the data is updated.
The data management system according to claim 8.
The non-volatile space further stores transaction management information;
The transaction management information is information including a transaction process ID and an information element indicating the location of the data in the data management information for each data in the transaction process for each transaction process.
The data management system according to claim 8.
The transaction management information includes an entry for each transaction process,
Each entry includes a transaction process ID and an information element indicating the location of the data in the data management information for each data in the transaction process,
The processor deletes an entry corresponding to a transaction process in a permanent state or a transaction in an abort state from the transaction management information.
The data management system according to claim 10.
In the storage space, a system state indicating whether or not the previous termination has been normally performed is stored,
When the system state at startup indicates an abnormal state, the processor changes each state of all target transaction processes to an abort state, and deletes the transaction management information as failure recovery processing And run
Each of all the target transaction processes is a transaction process associated with a state indicating a non-persistent state.
The data management system according to claim 10.
Having non-volatile memory,
The non-volatile space is a space based on the non-volatile memory,
The processor executes (A) and (B) by executing an in-memory DBMS (Database Management System).
The data management system according to claim 1.
Receive transaction processing instructions,
A state associated with transaction processing according to the received instruction, and writing a non-persistent state to a storage space including a nonvolatile space to which data is written in the transaction processing,
To end the transaction process, change the transaction process state to the permanent state.
Data management method.