CA1273434A - Method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system without burdening the system log - Google Patents
Method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system without burdening the system logInfo
- Publication number
- CA1273434A CA1273434A CA000528995A CA528995A CA1273434A CA 1273434 A CA1273434 A CA 1273434A CA 000528995 A CA000528995 A CA 000528995A CA 528995 A CA528995 A CA 528995A CA 1273434 A CA1273434 A CA 1273434A
- Authority
- CA
- Canada
- Prior art keywords
- transaction
- log
- tablespace
- load
- address space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1474—Saving, restoring, recovering or retrying in transactions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1405—Saving, restoring, recovering or retrying at machine instruction level
- G06F11/141—Saving, restoring, recovering or retrying at machine instruction level for bus or memory accesses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
- Debugging And Monitoring (AREA)
Abstract
METHOD FOR RESTARTING A LONG-RUNNING, FAULT-TOLERANT
OPERATION IN A TRANSACTION-ORIENTED DATA BASE
SYSTEM WITHOUT BURDENING THE SYSTEM LOG
Abstract of the Disclosure A restartable load without logging method permits the restart of a LOAD operation from the last COMMIT point without requiring the writing of images of loaded records to the log. Instead, the method logs only a minimal amount of information, recording positions within the data sets to be loaded and within the tablespace being loaded.
OPERATION IN A TRANSACTION-ORIENTED DATA BASE
SYSTEM WITHOUT BURDENING THE SYSTEM LOG
Abstract of the Disclosure A restartable load without logging method permits the restart of a LOAD operation from the last COMMIT point without requiring the writing of images of loaded records to the log. Instead, the method logs only a minimal amount of information, recording positions within the data sets to be loaded and within the tablespace being loaded.
Description
1273~34 METHOD FOR RESTARTING A LONG-RUNNING, FAULT-TOLERANT
OPERATION IN A TRANSACTION-ORIENT~D DATA BASE
SYSTEM WITnOUT BURDENING T~E SYSTEM LOG
Technical Field This invention relates to transaction-oriented data base systems, and more particularly, those systems having LOAD utilities and Recovery Managers which regulate system activity in the event of faults or interruptions.
Background As pointed out by C. J. Date, "An Introduction to Database Systems", Vol. 2, Addison-Wesley Publishing Co., copyright 1983, in Chapter 1 thereof, the purpose of a data base system is to carry out transactions. In this regard, a transaction is a unit of work. It consists of the execution of an application specified sequence of operations beginning with a special BE~IN TRANSACTION
operation and ending with either a COMMIT operation or a ROLLBACK operation. COMMIT is used to signal successful termination of the unit of work, while ROLLBACK is used to signal unsuccessful termination of work because of some exceptional condition. In transaction-oriented systems, a transaction such as transferring monev from one account to another is a single atomic operation. It either succeeds or fails. If it fails, then nothing should have changed;
that is, the effect should be as if it were never initiated.
Transaction-oriented systems usually include a Recovery Manager. A Recovery Manager is a subsystem component specializing in maintaining the atomic nature of transactions and reestablishing system operation. In order to reestabIish an information state of affairs to a 100 percent fidelitv, logging of all events occurs. The ~ .
~Z73~34 total log consists of a currentlv active online portion on direct access, plus an arbitrarv number of earlier portions in archival store.
There may be many events which cause a system to stop and thus require a subse~uent system restart. While the content~s of main memory and volati:Le buffers are lost, the data base on nonvolatile media is usually not damaged.
Transactions that were in progress at the time of failure must be rolled back since they were not completed. In order to identify which transactions to roll back, a search of the entire log ~rom the beginning would have to be made. This would be manifest by noting those transactions having a BEGIN TRANSACTION record but no termination, such as a COMMIT or other primitive. To avoid this, prior art utilizes checkpointing. This means that the contents of volatile memory representing transactions in proce.ss are copied out to the active log.
Tndeed, information constituting the checkpoint itself is made of record and wxitten to the log data set, and its address also duly noted in a RESTART file in nonvolatile storage. Each checkpoint record contains a list of all transactions active at the time of the checkpoint. Thus, at system restart time, the Recovery Manager can obtain the address of the most recent checkpoint record from the RESTART file. It then locates that checkpoint record in the log and proceeds to search forward through the log `~
from that point to the end. As a result of this process, the Recovery M~nager is able to determine both the transactions that need to be undone to effectuate ROLL~ACK
and the transactions that need to be redone to effectuate CO~IT in order to restore the data base to a correct state. To implement this, the Recovery Manager starts with two lists, an UNDO list and a REDO list. The UNDO
list initially contains all transactions listed in the checkpoint record. In contrast, the REDO list is initially empty. The Recovery Manager searches forward SA9-85-042 _~_ .
.
- 1~7343~L
through the log starting from the checkpoint record. If it finds a BEGIN TRANSACTION record for a given tranisaction, it adds that transaction to the UNDO list.
If it finds a COMMIT record for a given transaction, it moves that transaction from the UNDO to the REDO list.
When the Recovery Manager reaches the end of the log, the UNDO list and the REDO list identify, respectively, those transactions that must be undone and those which must be redone. Secondly, it goes forward through the log, redoing the transactions in the REDO list. Lastlv, the Recovery Manager works backward through the log again, undoing the transactions in the UNDO list. No new work can be accepted by the system until this process is complete.
Writing a change to the data base and writing the log record representing that change are two distinct operations. There is a possibilitv of failure occurring in the interval between the two. To enhance sa~ety, the log record is always written first. This is termed a "writeahead log protocol". That is, a transaction is not allowed to write a record to the physical data base until at least the UNDO portion of the corresponding log record has been written to the phvsical log, and a transaction is likewise not allowed to complete the COMMIT processing until both the REDO and UNDO portions of all log records for the transaction have been written to the physical log.
. .
In a transaction-oriented data base system of the type described above, all changes to the data base are written to a log in support of recoverv in the event of int~rruption. As mentioned, each transaction utilizes BEGIN, COMMIT, or ROLLBACK primitives in order to bound the transaction. In this regard, REDOs ensure transaction return to the most recent COMMIT point. In contra~st, UNDOs ensure return to the transaction BEGIN point.
.~' '~' ~ '' ,~, ~ ' ' " ' ' . " ,, ' '.
~ . ' ' ' ' ' . '. ;''' ',' .' '' .
1.273~34 Illustrative of transaction system log writing and utilization operations include:
(a) Gawlick et al, USP 4,507,751, "Method and Apparatus for Logging Journal Data Using a Log Write Ahead Data Set", issued March 26, 19~5;
(b) Paradine et al, USP 4,15~,517, "Journal Back-up Storage Control for a Data Processing System", issued June 26, 1979; and (c) Baker et al, USP 4,498,145, "Method and Apparatus for Assuring Atomicitv of Multirow Update Operations in a Database System", issued February 5, 1985.
The Gawlick, Paradine, and Baker patents respectivelv describe (a) the writing to log before record updating, (b) buffer dumping to a log, and (c) writing to a hard and soft log concurrently to assure multirow atomic updating.
Significantlv, the patented methods all relate to data movements, including those of loading which are paced or determined by the COMMI~ points of transactions.
In transaction-oriented systems of the relational data base type, it is the LOAD software utility which moves sequential data sets to a relational tablespace.
In the event of an interruption, it is also the LOAD
utilitv which must either restart from the beginning in order to avoid the overhead of log writing~ or alternatively, restart from the last COMMIT point ~i.e.
the end of the last transaction) and incur said log writing.
. . , :
- ~27~at34 The Invention It is an obje~t of this invention to devise a method for minimizing the number of steps involved in the restart of long-running, fault-tolerant operations in a transaction-oriented data base system. It is a related object that the method more particularly utilize a restartable LOAD utilitv from an intermediate COMMIT point without introducing undesirable operational characteristics.
The foregoing objects are satisfied by a method for executing a restartable LOAD operation in a transaction-oriented relational data base system in which all changesto the data base are typically written to a log in support of recovery in the event of interruption.
In this svstem, each transaction utilizes BEGIN, COMMIT, or ROLLBACK primitives to bound the transaction.
The REDO operations ensure transaction return to the most recent COMMIT point, while the UNDO operations ensure return to the transaction BEGIN point. The system includes a first address space containing sequential input data sets, and a second address space containing a relational tablespace. A LOAD operation moves the data sets from the first into the second address space by way of a series of nonoverlapping transactions. In this regard, the LOAD operation does not employ either before or after images in the log for any data set being loaded.
While multiple address spaces are described above, they are not nece.ssarily limiting.
The method of the invention comprises the steps of (a) reiteratively transferring a predeterminRd quantity of ~;
the data set to the end of the tablespace, recording the current position within the data set, and estahlishing another end position within the tablespace, the . . .
..
.: .
reiterative transfer continuing until the data set either becomes exhausted or the transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the LOAD operation from the last COMMIT point by emulating the REDO and UNDO without recourse to the before and after images of the tablespace from the log.
srief Description of the Drawin~
Figure 1 shows t~ansaction execution according to the prior art.
Figure 2 shows a timeline of transactions in states of completion with reference to checkpoint and system failure according to the prior art.
Figure 3 shows an example of the JCL and syntax for a LOAD LOG(NO) invocation.
Figure 4 shows an example of the JCL and syntax for th~ restart of the LOAD LOG!NO) invocation.
Figures 5 and 6 set out timelines of a data base sVstem with a LOAD operation in progress including transactions, force writing of buffers, checkpoints, and COMMIT points, with respect to the use of the LOAD utility according to the invention. More particularly, Figure 5 sets out the interruption of a LOAD utility invocation consisting of multiple transactions, while Figure 6 illustrates the restart and completion of the LOAD utility invocation from the last COMMIT point.
:::: .
.: :. :,:
~.273~34 Description of the Preferred Embodiment and Industrial Applicabilit~
Referring now to Figure 1, there is shown a timeline of various types of primiti~es (BEGIN, COMMIT, ROL~ACK~
which bound transactions. It should be observed that transactions cannot be nested. Not:e, a transaction starts.
with the BEGIN primitive and terminates with a COMMIT.
Implicitly, transactions are not lost, partially done, or done more than once. To assure both effectuation of a transaction and recovery in the event of interruption, transaction-oriented data base systems include a Recovery Manager. The Recoverv Manager initiates the transaction responsive to an external requei~t, opens Up the necessary access paths through service calls to the Data Manaqer and Buffer Manager, maintains logging through service calls to the Log Manager as the elements of the transaction proceed, and terminates the transaction at an appropriate CO~IT point. The Recovery Manager is also invoked whenever transaction processinq is interrupted.
Referring now to Figure 2, there is shown a timeline of transactions Tl-T5 in relation to checkpoint time Tc and system failure time Tf. It should be observed that a system failure has occurred at time Tf, and that the most recent checkpoint prior to time Tf was taken at time Tc. ; ii Note that transactions of type Tl were completed before time Tc. Transactions of type T2 were started prior to time Tc and completed after time Tc, and before time Tf.
Also, transactions of type T4 started after time Tc and completed before time Tf, together with transactions of type T2, are subject to REDO. In contrast, transactions of type T3 and T5 are subject to UNDO. Both T3 and T5 were not completed prior to system failure at time Tf.
: . .
.
- . ~
1273at34 As previously mentioned, the software LOAD utility moves sequential data sets from a first address space to a relational tablespace. It is not unusual for the LOAD
utilit,v to process large quantities of data. This takes a considerable amount of time. If, during this interval the LOAD operation is interrupted before completion, then to require a restart of the entire LOAD operation from the beginning becomec unsatisfactory from a user perspective.
One prior art solution involving a software LOAD
utility permitted the LOAD utilit,v to issue an internal COMMIT point after completing the processing of each predetermined block of buffered data. This guaranteed the ability to restart from the last internal CO~IT point by writing a copv of the loaded records to the log. Since the active portion of the log is a finite data set, this often forced archiving of the log during execution. On the other hand, if such were to be avoided, the only alternative to burdening the log writing operation was to restart from the beginning. Reference should be made to the following IBM publications for greater detail:
(1) SQL/Data System Concepts and Facilities for VM/SP, GH24-5065;
OPERATION IN A TRANSACTION-ORIENT~D DATA BASE
SYSTEM WITnOUT BURDENING T~E SYSTEM LOG
Technical Field This invention relates to transaction-oriented data base systems, and more particularly, those systems having LOAD utilities and Recovery Managers which regulate system activity in the event of faults or interruptions.
Background As pointed out by C. J. Date, "An Introduction to Database Systems", Vol. 2, Addison-Wesley Publishing Co., copyright 1983, in Chapter 1 thereof, the purpose of a data base system is to carry out transactions. In this regard, a transaction is a unit of work. It consists of the execution of an application specified sequence of operations beginning with a special BE~IN TRANSACTION
operation and ending with either a COMMIT operation or a ROLLBACK operation. COMMIT is used to signal successful termination of the unit of work, while ROLLBACK is used to signal unsuccessful termination of work because of some exceptional condition. In transaction-oriented systems, a transaction such as transferring monev from one account to another is a single atomic operation. It either succeeds or fails. If it fails, then nothing should have changed;
that is, the effect should be as if it were never initiated.
Transaction-oriented systems usually include a Recovery Manager. A Recovery Manager is a subsystem component specializing in maintaining the atomic nature of transactions and reestablishing system operation. In order to reestabIish an information state of affairs to a 100 percent fidelitv, logging of all events occurs. The ~ .
~Z73~34 total log consists of a currentlv active online portion on direct access, plus an arbitrarv number of earlier portions in archival store.
There may be many events which cause a system to stop and thus require a subse~uent system restart. While the content~s of main memory and volati:Le buffers are lost, the data base on nonvolatile media is usually not damaged.
Transactions that were in progress at the time of failure must be rolled back since they were not completed. In order to identify which transactions to roll back, a search of the entire log ~rom the beginning would have to be made. This would be manifest by noting those transactions having a BEGIN TRANSACTION record but no termination, such as a COMMIT or other primitive. To avoid this, prior art utilizes checkpointing. This means that the contents of volatile memory representing transactions in proce.ss are copied out to the active log.
Tndeed, information constituting the checkpoint itself is made of record and wxitten to the log data set, and its address also duly noted in a RESTART file in nonvolatile storage. Each checkpoint record contains a list of all transactions active at the time of the checkpoint. Thus, at system restart time, the Recovery Manager can obtain the address of the most recent checkpoint record from the RESTART file. It then locates that checkpoint record in the log and proceeds to search forward through the log `~
from that point to the end. As a result of this process, the Recovery M~nager is able to determine both the transactions that need to be undone to effectuate ROLL~ACK
and the transactions that need to be redone to effectuate CO~IT in order to restore the data base to a correct state. To implement this, the Recovery Manager starts with two lists, an UNDO list and a REDO list. The UNDO
list initially contains all transactions listed in the checkpoint record. In contrast, the REDO list is initially empty. The Recovery Manager searches forward SA9-85-042 _~_ .
.
- 1~7343~L
through the log starting from the checkpoint record. If it finds a BEGIN TRANSACTION record for a given tranisaction, it adds that transaction to the UNDO list.
If it finds a COMMIT record for a given transaction, it moves that transaction from the UNDO to the REDO list.
When the Recovery Manager reaches the end of the log, the UNDO list and the REDO list identify, respectively, those transactions that must be undone and those which must be redone. Secondly, it goes forward through the log, redoing the transactions in the REDO list. Lastlv, the Recovery Manager works backward through the log again, undoing the transactions in the UNDO list. No new work can be accepted by the system until this process is complete.
Writing a change to the data base and writing the log record representing that change are two distinct operations. There is a possibilitv of failure occurring in the interval between the two. To enhance sa~ety, the log record is always written first. This is termed a "writeahead log protocol". That is, a transaction is not allowed to write a record to the physical data base until at least the UNDO portion of the corresponding log record has been written to the phvsical log, and a transaction is likewise not allowed to complete the COMMIT processing until both the REDO and UNDO portions of all log records for the transaction have been written to the physical log.
. .
In a transaction-oriented data base system of the type described above, all changes to the data base are written to a log in support of recoverv in the event of int~rruption. As mentioned, each transaction utilizes BEGIN, COMMIT, or ROLLBACK primitives in order to bound the transaction. In this regard, REDOs ensure transaction return to the most recent COMMIT point. In contra~st, UNDOs ensure return to the transaction BEGIN point.
.~' '~' ~ '' ,~, ~ ' ' " ' ' . " ,, ' '.
~ . ' ' ' ' ' . '. ;''' ',' .' '' .
1.273~34 Illustrative of transaction system log writing and utilization operations include:
(a) Gawlick et al, USP 4,507,751, "Method and Apparatus for Logging Journal Data Using a Log Write Ahead Data Set", issued March 26, 19~5;
(b) Paradine et al, USP 4,15~,517, "Journal Back-up Storage Control for a Data Processing System", issued June 26, 1979; and (c) Baker et al, USP 4,498,145, "Method and Apparatus for Assuring Atomicitv of Multirow Update Operations in a Database System", issued February 5, 1985.
The Gawlick, Paradine, and Baker patents respectivelv describe (a) the writing to log before record updating, (b) buffer dumping to a log, and (c) writing to a hard and soft log concurrently to assure multirow atomic updating.
Significantlv, the patented methods all relate to data movements, including those of loading which are paced or determined by the COMMI~ points of transactions.
In transaction-oriented systems of the relational data base type, it is the LOAD software utility which moves sequential data sets to a relational tablespace.
In the event of an interruption, it is also the LOAD
utilitv which must either restart from the beginning in order to avoid the overhead of log writing~ or alternatively, restart from the last COMMIT point ~i.e.
the end of the last transaction) and incur said log writing.
. . , :
- ~27~at34 The Invention It is an obje~t of this invention to devise a method for minimizing the number of steps involved in the restart of long-running, fault-tolerant operations in a transaction-oriented data base system. It is a related object that the method more particularly utilize a restartable LOAD utilitv from an intermediate COMMIT point without introducing undesirable operational characteristics.
The foregoing objects are satisfied by a method for executing a restartable LOAD operation in a transaction-oriented relational data base system in which all changesto the data base are typically written to a log in support of recovery in the event of interruption.
In this svstem, each transaction utilizes BEGIN, COMMIT, or ROLLBACK primitives to bound the transaction.
The REDO operations ensure transaction return to the most recent COMMIT point, while the UNDO operations ensure return to the transaction BEGIN point. The system includes a first address space containing sequential input data sets, and a second address space containing a relational tablespace. A LOAD operation moves the data sets from the first into the second address space by way of a series of nonoverlapping transactions. In this regard, the LOAD operation does not employ either before or after images in the log for any data set being loaded.
While multiple address spaces are described above, they are not nece.ssarily limiting.
The method of the invention comprises the steps of (a) reiteratively transferring a predeterminRd quantity of ~;
the data set to the end of the tablespace, recording the current position within the data set, and estahlishing another end position within the tablespace, the . . .
..
.: .
reiterative transfer continuing until the data set either becomes exhausted or the transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the LOAD operation from the last COMMIT point by emulating the REDO and UNDO without recourse to the before and after images of the tablespace from the log.
srief Description of the Drawin~
Figure 1 shows t~ansaction execution according to the prior art.
Figure 2 shows a timeline of transactions in states of completion with reference to checkpoint and system failure according to the prior art.
Figure 3 shows an example of the JCL and syntax for a LOAD LOG(NO) invocation.
Figure 4 shows an example of the JCL and syntax for th~ restart of the LOAD LOG!NO) invocation.
Figures 5 and 6 set out timelines of a data base sVstem with a LOAD operation in progress including transactions, force writing of buffers, checkpoints, and COMMIT points, with respect to the use of the LOAD utility according to the invention. More particularly, Figure 5 sets out the interruption of a LOAD utility invocation consisting of multiple transactions, while Figure 6 illustrates the restart and completion of the LOAD utility invocation from the last COMMIT point.
:::: .
.: :. :,:
~.273~34 Description of the Preferred Embodiment and Industrial Applicabilit~
Referring now to Figure 1, there is shown a timeline of various types of primiti~es (BEGIN, COMMIT, ROL~ACK~
which bound transactions. It should be observed that transactions cannot be nested. Not:e, a transaction starts.
with the BEGIN primitive and terminates with a COMMIT.
Implicitly, transactions are not lost, partially done, or done more than once. To assure both effectuation of a transaction and recovery in the event of interruption, transaction-oriented data base systems include a Recovery Manager. The Recoverv Manager initiates the transaction responsive to an external requei~t, opens Up the necessary access paths through service calls to the Data Manaqer and Buffer Manager, maintains logging through service calls to the Log Manager as the elements of the transaction proceed, and terminates the transaction at an appropriate CO~IT point. The Recovery Manager is also invoked whenever transaction processinq is interrupted.
Referring now to Figure 2, there is shown a timeline of transactions Tl-T5 in relation to checkpoint time Tc and system failure time Tf. It should be observed that a system failure has occurred at time Tf, and that the most recent checkpoint prior to time Tf was taken at time Tc. ; ii Note that transactions of type Tl were completed before time Tc. Transactions of type T2 were started prior to time Tc and completed after time Tc, and before time Tf.
Also, transactions of type T4 started after time Tc and completed before time Tf, together with transactions of type T2, are subject to REDO. In contrast, transactions of type T3 and T5 are subject to UNDO. Both T3 and T5 were not completed prior to system failure at time Tf.
: . .
.
- . ~
1273at34 As previously mentioned, the software LOAD utility moves sequential data sets from a first address space to a relational tablespace. It is not unusual for the LOAD
utilit,v to process large quantities of data. This takes a considerable amount of time. If, during this interval the LOAD operation is interrupted before completion, then to require a restart of the entire LOAD operation from the beginning becomec unsatisfactory from a user perspective.
One prior art solution involving a software LOAD
utility permitted the LOAD utilit,v to issue an internal COMMIT point after completing the processing of each predetermined block of buffered data. This guaranteed the ability to restart from the last internal CO~IT point by writing a copv of the loaded records to the log. Since the active portion of the log is a finite data set, this often forced archiving of the log during execution. On the other hand, if such were to be avoided, the only alternative to burdening the log writing operation was to restart from the beginning. Reference should be made to the following IBM publications for greater detail:
(1) SQL/Data System Concepts and Facilities for VM/SP, GH24-5065;
(2) SQL/Data Svstem Planning and Administration for VM/SP, SH24-5043; and
(3) SQL/Data System Data ~ase Services Utility for VM/SP, SH24-5069.
To satisfy the objects of the invention, the method of the invention should restart from an intermediate COMMiIT point without flooding the log with data records.
This is accomplished by logging onlv a minimal amount of information, recording positions within the data sets to be moved and within the tablespace to be loaded.
~273~34 Referring now to Figure 3, there is shown the JCL and syntax for a LOAD LOG(NO) invocation, and in Figure 4, the JCL and synta~: for the restart of the LOAD LOG(NO) invocation. JCL is an acronym for Job Control Language.
This is a language facility used on large systems such as the IsM 370 to (a) name a ~eries of programs to be executed; (b) define the executing machine parameters; and (c) set priorities amon~ the programs and associate data sets with particular svstem functions. A LOAD LOG command admits of either a "yes" or "no" designation. If a "no"
designation is made, then execution of a LOAD LOG(NO) writes no REDO or UNDO records containing loaded data to the log. It does record, however, a minimal amount of information in the log at each COM~IIT point. The information identifies the utility's position within the data sets to be moved, within wor~ data sets, and within the tablespace being loaded at the time the COMMIT was issued. This information permits a LOAD LOG(NO~
invocation to reposition itself ~onsistently within these objects during restart.
A LOAD LOG(NO) invocation has no REDO log records to reconstruct the state of the tablespace during restart processing. Further, there is no assurance that commited tablespace page~s may be written prior to the beginning of the utility restart processing. Therefore, the LOAD
invocation itself must guarantee that all updated tablespace pages are written prior to the terminal phase (Phase II) of a two-phase CO~IIT operation.
Parenthetically, the first phase of a two-phase COMMIT
means that the svstem must force all log records involving a transaction to the log. If there are multiple participants in a transaction, each participant must indicate the state of affairs to a coordinating element.
In Phase II, a coordinator will broadcast a COMMIT to all of the transaction participants so as to complete processing. This can occur only if all of the ::
participants have arrived at the same completion state.
Otherwise, the transaction is aborted by all participants.
As mentioned, the LOAD LOG(NO) invocation must guarantee that all updated tablespace pages are written prior to Phase II of the COMMIT. This is accomplished by requiring the BUf fer Manager to use a synchronous force write operation during Phase I of the CO~IIT. This initiates the synchronous write of all updated system paqes that have not yet been written for a particular tablespace.
Suppose a LOAD LOG(NO~ invocation has no UNDO log record to enable it to reconstruct the state of the tablespace during restart processing. Also, updated system pages that have been written but not committed mav be present in the tablespace at the beginning of this utility restart. Since the associated records will be loaded again by the restarted LOAD LOG(NO) invocation, the LOAD utility itself must again guarantee that extra updated pages be removed from the tahlespace prior to restarting from the last COMMIT point. The utility can again reference the Data Manager to initiate a LOAD
protocol to remove uncommitted data from the tablespace.
This invocation provides the Data Manager with a position within the tablespace at the last COMMIT point. The Data Manager may then erase any extraneous data, and thus guarantee that the tablespace page control information is consistent with-the contents of the tablespace. At this point, the LOAD utility can restart from the last COMMIT
point.
The Data Manager protocols permit a LOAD LOG(NO) invocation to (a) request the position of the last record within the tablespace; or (b) request that all records beyond a certain point within the tablespace be deleted, and that the tablespace page control information, --. -: ,~ , :
: . ~
'' ~:
~.~73434 including the position of the last record within the tablespace, be updated accordingly.
The Buffer Manager protocol permits a LOAD LOG(NO) invocation to force write buffers to a tablespace for which it holds an exclusive lock.
Force writing buffers to the tablespace guarantees that a known quantitv of tablespace data has been written to disk. LOAD LOG(NO) uses this protocol to ensure that the minimum amount o data required for restarting at the last COMMIT point has always been written to disk prior to a request for restarting from the last COMMIT point.
A LOAD invocation executes in two address spaces: an application address space, and a relational data ~ase address space c~lled the Advanced Data Rase Manager Facility (ADMF) address space. The basic flow within ea~h address space for a LOAD LOG(NO) invocation is as follows:
Application Addxess Space - Request that an ADMF Load Utility module (DSNURWIT) perform ADMF address space initialization processing.
0 - Do until all unprocessed records in the input data set have been processed:
Fill an application address space buffer with unprocessed records from the input data set.
Request than an ADMF Load Utility module (DSNURW~F) load the input data records from the application address space buffer into the tablespace being loaded.
' ~273~34 If the number of application address space buffers pxocessed since the last COMMIT exceeds an internal thxeshold value or the entire input data set has been proces~sed:
Synchronize the input and work data sets.
Request that an ADMF Buffer Manager module (DSNBWFOR) force write the ADMF buffers for the tablespace being loaded to disk.
Transfer information about the current position within the tablespace being loaded (contained in the UCRA) from the ADMF
address space to the application address space.
LOG information about the current position of unprocessed records within the input data set, the current position within the tablespace being loaded, and the current positions within the work data sets.
COMMIT the changes made to the tablespace being loaded since the last COMMIT or since the start of processing.
- Request that an ADMF Load Utilitv module (DSNURWIT) perform ADMF address space termination processing.
ADMF Address Space DSNURWIT:
- If initialization pxocessing has been requested~
~ .. . . -~273~34 Initialize ADMF address Cpace control blocks (including a control block called the URTS) for LOAD Utility processing.
Initialize Buffer Manager and Data Manager control blocks and internal indicators for LOG(NO) processing.
Invoke Data Manager ~DSNISLOD) to position at the last record within the tablespace being loaded and to return the internal positioning information.
Record the internal positioning information in the URTS.
Allocate a positional information control block (also called the UCRA). ~ ;
If not a restarted invocation:
Copy the internal positioning ;
information from the URTS into the UCRA.
If a restarted invocation:
Restore the internal positioning information in the UCRA from the ~ `
information logged during the last LOAD invocation for the tablespace being loaded.
.. : - . :-. . . , "
' ~
.
lZ73~34 Copy the internal positioning information within the tablespace being loaded from the UCRA into the URTS in preparation for any required repositioning that the Data Manager may need to perform on the next Data Manager Start Load (DSNISLOD) invocation.
- If termination processing ha~s been requested:
Invoke Buffer Manager (DSNICLPS) to force close the tablespace being loaded, which includes force writing any buffers not previously written.
DSNURI~BF:
.
- Move the application address space buffer containing the input data records being loaded into an ADMF address space buffer.
- Invoke Data Manager (DSNISLOD) to perform Start Load processing including any required repositioning within the tablespace being loaded ~ `
that the Data Manager needs to affect.
::
- Do while input data records in the ADMF address space buffer remain to be processed:
Invoke Data Manager (DSNILOAD) to load the next input data record into the tablespace being loaded, without writing UNDO or REDO
records to the LOG for the input data record, and to record information about the current position within the tablespace being loaded into the URTS.
~."' .,;- ~ ' ' 1~73434 - Copv the information about the current position within the tablespace being loaded from the URTS
into the UCRA.
- Invoke Data Manager (DSNIENDL) to perform ~nd Load processing including the update of tablespace page control information in preparation for COMMIT.
Illustrative Example oi~ the Method The Load UtilitV Invocation Consider a Load Utility invocation that is to load information about employees into an EMPLOYEE table.
Assume that the input data set consists of 1,000,000 fixed-length employee records each 100 bytes in length, and that each record contains a unique employee number along with an employee's name, address, title, department number, and work location.
Further, a~sume that the application address space and ADMF address space buffers are of size 32K, and that the internal threshold value for COMMIT is 100. Also, assume that the tablespace page size is 4R.
Finally, assume that the EMPLOYEE table resides in the same tablespace as another table, and assume also that the EMPLOYEE table and the other table contain 2152 and 212 rows respectively prior to execution of the Load Utility invocation.
Figure 3 presents the JCL and syntax for a LOAD
LOG(NO) invocation that performs this task. Note that the SYSIN DD statement stream defines the Load Utility invocation, that the SYSREC DD statement defines the input .. . . . . ~
1.2 73L~34 data set, and that the SYSUTl DD statement defines a work data set used later for building indices.
Buffer Usage and Fre~uencv of CO~IT
The application address .space and ADMF address space buffers contain both data records and control length information. suffer length control information takes up four bytes per buffer and record length control information takes up four bvte.s per record. Therefore, each fillina of the 32K application adAress space buffer transfers 315 records from the input data set into the application address space memory, since cont~ol information takes up 1264 hytes.
Consequently, the Load Utility invocation fills the 32K application address space buffer 3175 times. Since the invocation commits after every 100 application address space buffers, the Load Utility invocation consists of 32 transactions. Thus, following an interruption, restart of the LOAD LOG (NO) invocation from an internal COMMIT point will logically occur at the beginning of one of the 32 transactions.
Checkpoint Information ana Use of the l.og Both LOAD LOG (NO) and LOAD LOG (YES1 invocations log information about the current position of unprocessed records within the input data set, the current po.sition within the tablespace being loaded, and the current positions within one or more work data sets. Log record control information and checkpoint control information require 52 bytes. Recording the current position within the tablespace requires 24 bytes. Recording the position within the input data .set or within the work data sets requires 76 bytes per access method services buffer. The log space requi:rements per log record for logging SAg-85-042 -16-'.:, '" , ` :
."
'.': ' `' . ': ', ' '' ,., ' .' , .. .... . . ..
.~.,; . .
~2734~4 positional information are similar for LOAD LOG(NO) and LOAD LOG(YES) invocations. ~owever, a LOG(~O) invocation writes log records for recording positional information one-tenth as often as a LOG(YES) invocation.
A LOAD LOG(NO) invocation does not write REDO or UND~
records to the log for the records being loaded. This results in a drastic reduction of loa usage for a LOAD
LOG(NO) invocation that loads large quantities of data. A
LOAD LOG(YES) invocation that accomplishes the same task as the LOAD LOG(NO) invocation of Figure 3 will write a log record for each page written to the tablespace. Since 40 input data set records fill a page, a load of 1,000,000 records will load appro~imately 25,000 pages. Since each log record requires 40 bytes of control information as well as an image of the page, a LOAD LOG!YES) invocation will consume appro~imately 103,400,000 additional bytes of the log.
Execution of the Load UtilitY
Initialization processing for the execution of the LOAD LOG(NO) invocation of Figure 3 obtains and initializes the required control blocks and structures including the URTS and the UCRA. Initialization proces.sing also establishes the starting position for the load as the first free position rollowing the 2364 records present in the tablespace. Tnitialization processing records this first record position both within the URTS
and the UCRA.
The Load Utility then transfers the first 315 records from the input data set to the application address space buffer. Since repositioning within the tablespace is not required, the loading of records begins at the position recorded within the URTS and UCRA. Load record processing places each successive record into the tablespace directly -, '', ' , '-'''~' ''' ' :: ' - ' . :
behind the last loaded record and records the current position in the URTS. When load record processing for the application address space buffer is complete, load processing copies the current record position within the tablespace being loaded from the U~TS into the UCRA.
The Load Utility proceeds in the same fashion for the next 31,185 records within the input data set. It reiteratively transfers 315 records from the input data set to the application address space buffer and then processes the contents of that buffer as it processed the first buffer.
After load record processing completes for the lOOth buffer, load processing prepares for the first commit.
Preparation for CO~MIT consists of force writing ADMF
buffers to disk and logging positioning information.
Preparation begins by force writing all ADMF buffers that contain any of the 31,500 most recently loaded records and that have not already been written. Load processing then gathers positional information. It notes its current 20 position in the tablespace as being at the 33,864th record, its current position in the input data set as being at the 31,500th record, and its corresponding position within the work data set. The position in the work data set depends on the number of indices defined on 25 the EMPLOYEE table and the number o~ bytes in the keys of the indices. After logging its current positional information, load commits the 31,500 most recently loaded records.
Load processing continues to commit after each 30 loading of 31,500 records until it loads the final application address space buffer and issues the final CO~5IT.
: `:
.. . .. .
. :-;, :: '' ' : . :
., :~2~3~34 Example of Interruption and Restart Processing Assume that interruption of the Load Utilitv occurs as the utility loads the 636,348th record. At this point, the Load Utility is loading the 48th record of the 2021st application address space buffer. The current position within the tablespace is at the 638,712th record. The current position within the input data set is at or beyond the 636,615th xecord. The position in the work data set depends on the number of indices defined on the EMPLOYEE
tablQ, on the nu~ber of bvtes in the keys of the indices, and on the fact that 2020 buffers have been successfully loaded prior to the interruption. Figure 5 illustrates the eYecution of the LOAD LOG~NO) invocation up to the occurrence of the interruption.
Figure 4 presents the JCL and syntax for the restart of the LOAD LOG(NO) invocation from an internal COMMIT
point. Note that the onl~ change to the original JCL and syntax is the addition of the RESTART parameter on the EXEC JCL statement.
Restart load record processing will begin loading ;
records into the tablespace from the start of the 2001st application address space buffer of the original invocation. In order to accomplish restart in this fashion, initialization processing reconstructs the internal processing state as it eYisted following the 20th COMMIT.
Restart processing begins with initialization processing. Initialization processing obtains and initializes the required control blocks and structures, including the URTS and the UCRA. Initialization processing then establishes the position of the last record within the tablespace that was written to the disk prior to the interruption. For the interrupted LOAD
~ ' :. , . ... . .
1;~73434 LOG(NO) invocation, thi.s position can be anywhere in the tablespace from the 632,364th record to the 638,712th record. Initialization processing then reads the logged information to obtain the position within the input data set, the position within the tablespace being loaded, and the position within the work data set at the time the 20th COMMIT occurred. Initialization processing records this information in the appropriate fields of the UCRA and URTS.
Following initialization, the Load IJtility opens the input data set and the work data set, restoring its current position within each to the position held following the completion of the 20th COM~IIT. The Load invocation then transfers the next 315 records from the input data set to the application address space buffer.
The first record in the application address space buffer will be the 630,001st record of the input data set. Since repositioning within the tablespace is now required, the Load Utility invokes the Data Manager to delete anv records present within the tablespace following the 632,364th record. Restart processing is then complete, and the execution of the Load invocation proceeds as usual, loading buffer after buffer of input data until it processes all records of the input data set. Figure 6 illustrates the execution of the restart of the LOAD
LOG(NO) invocation from the 20th COMMIT point to completion.
While only certain preferred features of this invention have been shown by way of illustration, many changes and modifications will occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.
.. ^
. . - ,.
. . ~ .
- : .
., . . -
To satisfy the objects of the invention, the method of the invention should restart from an intermediate COMMiIT point without flooding the log with data records.
This is accomplished by logging onlv a minimal amount of information, recording positions within the data sets to be moved and within the tablespace to be loaded.
~273~34 Referring now to Figure 3, there is shown the JCL and syntax for a LOAD LOG(NO) invocation, and in Figure 4, the JCL and synta~: for the restart of the LOAD LOG(NO) invocation. JCL is an acronym for Job Control Language.
This is a language facility used on large systems such as the IsM 370 to (a) name a ~eries of programs to be executed; (b) define the executing machine parameters; and (c) set priorities amon~ the programs and associate data sets with particular svstem functions. A LOAD LOG command admits of either a "yes" or "no" designation. If a "no"
designation is made, then execution of a LOAD LOG(NO) writes no REDO or UNDO records containing loaded data to the log. It does record, however, a minimal amount of information in the log at each COM~IIT point. The information identifies the utility's position within the data sets to be moved, within wor~ data sets, and within the tablespace being loaded at the time the COMMIT was issued. This information permits a LOAD LOG(NO~
invocation to reposition itself ~onsistently within these objects during restart.
A LOAD LOG(NO) invocation has no REDO log records to reconstruct the state of the tablespace during restart processing. Further, there is no assurance that commited tablespace page~s may be written prior to the beginning of the utility restart processing. Therefore, the LOAD
invocation itself must guarantee that all updated tablespace pages are written prior to the terminal phase (Phase II) of a two-phase CO~IIT operation.
Parenthetically, the first phase of a two-phase COMMIT
means that the svstem must force all log records involving a transaction to the log. If there are multiple participants in a transaction, each participant must indicate the state of affairs to a coordinating element.
In Phase II, a coordinator will broadcast a COMMIT to all of the transaction participants so as to complete processing. This can occur only if all of the ::
participants have arrived at the same completion state.
Otherwise, the transaction is aborted by all participants.
As mentioned, the LOAD LOG(NO) invocation must guarantee that all updated tablespace pages are written prior to Phase II of the COMMIT. This is accomplished by requiring the BUf fer Manager to use a synchronous force write operation during Phase I of the CO~IIT. This initiates the synchronous write of all updated system paqes that have not yet been written for a particular tablespace.
Suppose a LOAD LOG(NO~ invocation has no UNDO log record to enable it to reconstruct the state of the tablespace during restart processing. Also, updated system pages that have been written but not committed mav be present in the tablespace at the beginning of this utility restart. Since the associated records will be loaded again by the restarted LOAD LOG(NO) invocation, the LOAD utility itself must again guarantee that extra updated pages be removed from the tahlespace prior to restarting from the last COMMIT point. The utility can again reference the Data Manager to initiate a LOAD
protocol to remove uncommitted data from the tablespace.
This invocation provides the Data Manager with a position within the tablespace at the last COMMIT point. The Data Manager may then erase any extraneous data, and thus guarantee that the tablespace page control information is consistent with-the contents of the tablespace. At this point, the LOAD utility can restart from the last COMMIT
point.
The Data Manager protocols permit a LOAD LOG(NO) invocation to (a) request the position of the last record within the tablespace; or (b) request that all records beyond a certain point within the tablespace be deleted, and that the tablespace page control information, --. -: ,~ , :
: . ~
'' ~:
~.~73434 including the position of the last record within the tablespace, be updated accordingly.
The Buffer Manager protocol permits a LOAD LOG(NO) invocation to force write buffers to a tablespace for which it holds an exclusive lock.
Force writing buffers to the tablespace guarantees that a known quantitv of tablespace data has been written to disk. LOAD LOG(NO) uses this protocol to ensure that the minimum amount o data required for restarting at the last COMMIT point has always been written to disk prior to a request for restarting from the last COMMIT point.
A LOAD invocation executes in two address spaces: an application address space, and a relational data ~ase address space c~lled the Advanced Data Rase Manager Facility (ADMF) address space. The basic flow within ea~h address space for a LOAD LOG(NO) invocation is as follows:
Application Addxess Space - Request that an ADMF Load Utility module (DSNURWIT) perform ADMF address space initialization processing.
0 - Do until all unprocessed records in the input data set have been processed:
Fill an application address space buffer with unprocessed records from the input data set.
Request than an ADMF Load Utility module (DSNURW~F) load the input data records from the application address space buffer into the tablespace being loaded.
' ~273~34 If the number of application address space buffers pxocessed since the last COMMIT exceeds an internal thxeshold value or the entire input data set has been proces~sed:
Synchronize the input and work data sets.
Request that an ADMF Buffer Manager module (DSNBWFOR) force write the ADMF buffers for the tablespace being loaded to disk.
Transfer information about the current position within the tablespace being loaded (contained in the UCRA) from the ADMF
address space to the application address space.
LOG information about the current position of unprocessed records within the input data set, the current position within the tablespace being loaded, and the current positions within the work data sets.
COMMIT the changes made to the tablespace being loaded since the last COMMIT or since the start of processing.
- Request that an ADMF Load Utilitv module (DSNURWIT) perform ADMF address space termination processing.
ADMF Address Space DSNURWIT:
- If initialization pxocessing has been requested~
~ .. . . -~273~34 Initialize ADMF address Cpace control blocks (including a control block called the URTS) for LOAD Utility processing.
Initialize Buffer Manager and Data Manager control blocks and internal indicators for LOG(NO) processing.
Invoke Data Manager ~DSNISLOD) to position at the last record within the tablespace being loaded and to return the internal positioning information.
Record the internal positioning information in the URTS.
Allocate a positional information control block (also called the UCRA). ~ ;
If not a restarted invocation:
Copy the internal positioning ;
information from the URTS into the UCRA.
If a restarted invocation:
Restore the internal positioning information in the UCRA from the ~ `
information logged during the last LOAD invocation for the tablespace being loaded.
.. : - . :-. . . , "
' ~
.
lZ73~34 Copy the internal positioning information within the tablespace being loaded from the UCRA into the URTS in preparation for any required repositioning that the Data Manager may need to perform on the next Data Manager Start Load (DSNISLOD) invocation.
- If termination processing ha~s been requested:
Invoke Buffer Manager (DSNICLPS) to force close the tablespace being loaded, which includes force writing any buffers not previously written.
DSNURI~BF:
.
- Move the application address space buffer containing the input data records being loaded into an ADMF address space buffer.
- Invoke Data Manager (DSNISLOD) to perform Start Load processing including any required repositioning within the tablespace being loaded ~ `
that the Data Manager needs to affect.
::
- Do while input data records in the ADMF address space buffer remain to be processed:
Invoke Data Manager (DSNILOAD) to load the next input data record into the tablespace being loaded, without writing UNDO or REDO
records to the LOG for the input data record, and to record information about the current position within the tablespace being loaded into the URTS.
~."' .,;- ~ ' ' 1~73434 - Copv the information about the current position within the tablespace being loaded from the URTS
into the UCRA.
- Invoke Data Manager (DSNIENDL) to perform ~nd Load processing including the update of tablespace page control information in preparation for COMMIT.
Illustrative Example oi~ the Method The Load UtilitV Invocation Consider a Load Utility invocation that is to load information about employees into an EMPLOYEE table.
Assume that the input data set consists of 1,000,000 fixed-length employee records each 100 bytes in length, and that each record contains a unique employee number along with an employee's name, address, title, department number, and work location.
Further, a~sume that the application address space and ADMF address space buffers are of size 32K, and that the internal threshold value for COMMIT is 100. Also, assume that the tablespace page size is 4R.
Finally, assume that the EMPLOYEE table resides in the same tablespace as another table, and assume also that the EMPLOYEE table and the other table contain 2152 and 212 rows respectively prior to execution of the Load Utility invocation.
Figure 3 presents the JCL and syntax for a LOAD
LOG(NO) invocation that performs this task. Note that the SYSIN DD statement stream defines the Load Utility invocation, that the SYSREC DD statement defines the input .. . . . . ~
1.2 73L~34 data set, and that the SYSUTl DD statement defines a work data set used later for building indices.
Buffer Usage and Fre~uencv of CO~IT
The application address .space and ADMF address space buffers contain both data records and control length information. suffer length control information takes up four bytes per buffer and record length control information takes up four bvte.s per record. Therefore, each fillina of the 32K application adAress space buffer transfers 315 records from the input data set into the application address space memory, since cont~ol information takes up 1264 hytes.
Consequently, the Load Utility invocation fills the 32K application address space buffer 3175 times. Since the invocation commits after every 100 application address space buffers, the Load Utility invocation consists of 32 transactions. Thus, following an interruption, restart of the LOAD LOG (NO) invocation from an internal COMMIT point will logically occur at the beginning of one of the 32 transactions.
Checkpoint Information ana Use of the l.og Both LOAD LOG (NO) and LOAD LOG (YES1 invocations log information about the current position of unprocessed records within the input data set, the current po.sition within the tablespace being loaded, and the current positions within one or more work data sets. Log record control information and checkpoint control information require 52 bytes. Recording the current position within the tablespace requires 24 bytes. Recording the position within the input data .set or within the work data sets requires 76 bytes per access method services buffer. The log space requi:rements per log record for logging SAg-85-042 -16-'.:, '" , ` :
."
'.': ' `' . ': ', ' '' ,., ' .' , .. .... . . ..
.~.,; . .
~2734~4 positional information are similar for LOAD LOG(NO) and LOAD LOG(YES) invocations. ~owever, a LOG(~O) invocation writes log records for recording positional information one-tenth as often as a LOG(YES) invocation.
A LOAD LOG(NO) invocation does not write REDO or UND~
records to the log for the records being loaded. This results in a drastic reduction of loa usage for a LOAD
LOG(NO) invocation that loads large quantities of data. A
LOAD LOG(YES) invocation that accomplishes the same task as the LOAD LOG(NO) invocation of Figure 3 will write a log record for each page written to the tablespace. Since 40 input data set records fill a page, a load of 1,000,000 records will load appro~imately 25,000 pages. Since each log record requires 40 bytes of control information as well as an image of the page, a LOAD LOG!YES) invocation will consume appro~imately 103,400,000 additional bytes of the log.
Execution of the Load UtilitY
Initialization processing for the execution of the LOAD LOG(NO) invocation of Figure 3 obtains and initializes the required control blocks and structures including the URTS and the UCRA. Initialization proces.sing also establishes the starting position for the load as the first free position rollowing the 2364 records present in the tablespace. Tnitialization processing records this first record position both within the URTS
and the UCRA.
The Load Utility then transfers the first 315 records from the input data set to the application address space buffer. Since repositioning within the tablespace is not required, the loading of records begins at the position recorded within the URTS and UCRA. Load record processing places each successive record into the tablespace directly -, '', ' , '-'''~' ''' ' :: ' - ' . :
behind the last loaded record and records the current position in the URTS. When load record processing for the application address space buffer is complete, load processing copies the current record position within the tablespace being loaded from the U~TS into the UCRA.
The Load Utility proceeds in the same fashion for the next 31,185 records within the input data set. It reiteratively transfers 315 records from the input data set to the application address space buffer and then processes the contents of that buffer as it processed the first buffer.
After load record processing completes for the lOOth buffer, load processing prepares for the first commit.
Preparation for CO~MIT consists of force writing ADMF
buffers to disk and logging positioning information.
Preparation begins by force writing all ADMF buffers that contain any of the 31,500 most recently loaded records and that have not already been written. Load processing then gathers positional information. It notes its current 20 position in the tablespace as being at the 33,864th record, its current position in the input data set as being at the 31,500th record, and its corresponding position within the work data set. The position in the work data set depends on the number of indices defined on 25 the EMPLOYEE table and the number o~ bytes in the keys of the indices. After logging its current positional information, load commits the 31,500 most recently loaded records.
Load processing continues to commit after each 30 loading of 31,500 records until it loads the final application address space buffer and issues the final CO~5IT.
: `:
.. . .. .
. :-;, :: '' ' : . :
., :~2~3~34 Example of Interruption and Restart Processing Assume that interruption of the Load Utilitv occurs as the utility loads the 636,348th record. At this point, the Load Utility is loading the 48th record of the 2021st application address space buffer. The current position within the tablespace is at the 638,712th record. The current position within the input data set is at or beyond the 636,615th xecord. The position in the work data set depends on the number of indices defined on the EMPLOYEE
tablQ, on the nu~ber of bvtes in the keys of the indices, and on the fact that 2020 buffers have been successfully loaded prior to the interruption. Figure 5 illustrates the eYecution of the LOAD LOG~NO) invocation up to the occurrence of the interruption.
Figure 4 presents the JCL and syntax for the restart of the LOAD LOG(NO) invocation from an internal COMMIT
point. Note that the onl~ change to the original JCL and syntax is the addition of the RESTART parameter on the EXEC JCL statement.
Restart load record processing will begin loading ;
records into the tablespace from the start of the 2001st application address space buffer of the original invocation. In order to accomplish restart in this fashion, initialization processing reconstructs the internal processing state as it eYisted following the 20th COMMIT.
Restart processing begins with initialization processing. Initialization processing obtains and initializes the required control blocks and structures, including the URTS and the UCRA. Initialization processing then establishes the position of the last record within the tablespace that was written to the disk prior to the interruption. For the interrupted LOAD
~ ' :. , . ... . .
1;~73434 LOG(NO) invocation, thi.s position can be anywhere in the tablespace from the 632,364th record to the 638,712th record. Initialization processing then reads the logged information to obtain the position within the input data set, the position within the tablespace being loaded, and the position within the work data set at the time the 20th COMMIT occurred. Initialization processing records this information in the appropriate fields of the UCRA and URTS.
Following initialization, the Load IJtility opens the input data set and the work data set, restoring its current position within each to the position held following the completion of the 20th COM~IIT. The Load invocation then transfers the next 315 records from the input data set to the application address space buffer.
The first record in the application address space buffer will be the 630,001st record of the input data set. Since repositioning within the tablespace is now required, the Load Utility invokes the Data Manager to delete anv records present within the tablespace following the 632,364th record. Restart processing is then complete, and the execution of the Load invocation proceeds as usual, loading buffer after buffer of input data until it processes all records of the input data set. Figure 6 illustrates the execution of the restart of the LOAD
LOG(NO) invocation from the 20th COMMIT point to completion.
While only certain preferred features of this invention have been shown by way of illustration, many changes and modifications will occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.
.. ^
. . - ,.
. . ~ .
- : .
., . . -
Claims (11)
1. A method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system, all changes to the data base being written to a log in support of recovery in the event of interruption, each transaction utilizing BEGIN, COMMIT, or ROLLBACK
primitives to bound said transaction, REDOs ensuring transaction return to the most recent COMMIT point, while UNDOs ensuring return to the transaction BEGIN point, comprising the steps of:
(a) reiteratively moving a predetermined quantity of a sequential data set and appending said quantity to the end of a data base system formatted object, recording the current position within the data set and establishing another end position within the formatted object, said step continuing either until the data set becomes exhausted or a transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the operation from the last COMMIT point by emulating REDO and UNDO without recourse to the before and after images of the formatted object from the log.
primitives to bound said transaction, REDOs ensuring transaction return to the most recent COMMIT point, while UNDOs ensuring return to the transaction BEGIN point, comprising the steps of:
(a) reiteratively moving a predetermined quantity of a sequential data set and appending said quantity to the end of a data base system formatted object, recording the current position within the data set and establishing another end position within the formatted object, said step continuing either until the data set becomes exhausted or a transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the operation from the last COMMIT point by emulating REDO and UNDO without recourse to the before and after images of the formatted object from the log.
2. A method according to claim 1, wherein said system includes a first address space containing the data sets and a second address space containing the data base system formatted object.
3. A method according to claim 2, wherein the data base object is a relational tablespace.
4. A method according to claim 1, wherein said long-running, fault-tolerant operation is of the type in which data sets are moved from a first address space to a second address space by way of a series of nonoverlapping suboperations, said fault-tolerant operation not employing before or after images of any data being moved.
5. A method for executing a restartable LOAD
operation in a transaction-oriented relational data base system, all changes to the data base being written to a log in support of recovery in the event of interruption, each transaction utilizing BEGIN, COMMIT, or ROLLBACK
primitives to bound said transaction, REDOs ensuring transaction return to the most recent COMMIT point, while UNDOs ensuring return to the transaction BEGIN point, said system exhibiting a first address space containing sequential data sets and a second address space containing a relational tablespace, the LOAD operation moving the data sets from the first into the second address space by a series of nonoverlapping operations, comprising the steps of:
(a) reiteratively transferring a predetermined quantity of the data set from the first address space to the end of the tablespace in the second address space, recording the current position within the data set and establishing another end position within the tablespace, said step continuing until either the data set becomes exhausted or the transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the LOAD operation from the last COMMIT
point by emulating the REDO and UNDO without recourse to the before and after images of the tablespace from the log.
operation in a transaction-oriented relational data base system, all changes to the data base being written to a log in support of recovery in the event of interruption, each transaction utilizing BEGIN, COMMIT, or ROLLBACK
primitives to bound said transaction, REDOs ensuring transaction return to the most recent COMMIT point, while UNDOs ensuring return to the transaction BEGIN point, said system exhibiting a first address space containing sequential data sets and a second address space containing a relational tablespace, the LOAD operation moving the data sets from the first into the second address space by a series of nonoverlapping operations, comprising the steps of:
(a) reiteratively transferring a predetermined quantity of the data set from the first address space to the end of the tablespace in the second address space, recording the current position within the data set and establishing another end position within the tablespace, said step continuing until either the data set becomes exhausted or the transaction in progress becomes interrupted; and (b) in the event of transaction interruption, restarting the LOAD operation from the last COMMIT
point by emulating the REDO and UNDO without recourse to the before and after images of the tablespace from the log.
6. A method according to claims 1 or 5, wherein step (b) is further limited in that in the event of restart interruptions occurring while performing the restart step, generating an idempotent outcome by restarting the operation again as if the restart interruption had not occurred.
7. A method according to claims 1 or 5, wherein step (h) is further limited in that emulating REDO and UNDO includes restoring the processing state of the operation to the state existing just prior to the last COMMIT utilizing log positional information and log independent processing.
8. A method according to claims 1 or 2, wherein the source data set is a formatted object.
9. A method according to claim 8, wherein the formatted object is a relational table.
10. A method according to claim 1, wherein the restart of the operation selectively occurs at a COMMIT
point prior in time to the last COMMIT point.
point prior in time to the last COMMIT point.
11. A method according to claim 10, wherein the initialization COMMIT point of the operation is used as a bound to effectuate ROLLBACK of the operation.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US06/835,396 US4868744A (en) | 1986-03-03 | 1986-03-03 | Method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system without burdening the system log |
| US835,396 | 1986-03-03 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1273434A true CA1273434A (en) | 1990-08-28 |
Family
ID=25269410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000528995A Expired - Fee Related CA1273434A (en) | 1986-03-03 | 1987-02-04 | Method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system without burdening the system log |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US4868744A (en) |
| EP (1) | EP0236743B1 (en) |
| JP (1) | JPH0628043B2 (en) |
| CA (1) | CA1273434A (en) |
| DE (1) | DE3788444T2 (en) |
Families Citing this family (69)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS6310251A (en) * | 1986-07-01 | 1988-01-16 | Nec Corp | Control system for record indicator |
| DE3771603D1 (en) * | 1986-07-23 | 1991-08-29 | Siemens Ag | MODULAR STRUCTURED ISDN COMMUNICATION SYSTEM WITH EDUCATION AND DISPLAY OF ERROR TEXTS. |
| US5065311A (en) * | 1987-04-20 | 1991-11-12 | Hitachi, Ltd. | Distributed data base system of composite subsystem type, and method fault recovery for the system |
| JP2718031B2 (en) * | 1987-07-17 | 1998-02-25 | 株式会社日立製作所 | History information acquisition method |
| JPH01147727A (en) * | 1987-12-04 | 1989-06-09 | Hitachi Ltd | Fault restoring method for on-line program |
| US5123104A (en) * | 1988-04-08 | 1992-06-16 | International Business Machines Corporation | Method and apparatus for concurrent modification of an index tree in a transaction processing system utilizing selective indication of structural modification operations |
| US4945474A (en) * | 1988-04-08 | 1990-07-31 | Internatinal Business Machines Corporation | Method for restoring a database after I/O error employing write-ahead logging protocols |
| US5043866A (en) * | 1988-04-08 | 1991-08-27 | International Business Machines Corporation | Soft checkpointing system using log sequence numbers derived from stored data pages and log records for database recovery |
| US5278983A (en) * | 1988-09-30 | 1994-01-11 | International Business Machines Corporation | Boundary representation solid modeling system |
| US5159669A (en) * | 1988-12-15 | 1992-10-27 | Xerox Corporation | Automatically creating a second workspace operation record including history data and a unit ID based on a first workspace operation |
| EP0388155B1 (en) * | 1989-03-14 | 1998-02-25 | Canon Kabushiki Kaisha | Process status supervisory system |
| JPH02310665A (en) * | 1989-05-25 | 1990-12-26 | Toshiba Corp | Data restoration method for distributed transaction processing system |
| DE68927705T2 (en) * | 1989-06-13 | 1997-07-17 | Ibm | Process for removing unconfirmed changes to stored data by a database management system |
| US5095421A (en) * | 1989-08-17 | 1992-03-10 | International Business Machines Corporation | Transaction processing facility within an operating system environment |
| US5170480A (en) * | 1989-09-25 | 1992-12-08 | International Business Machines Corporation | Concurrently applying redo records to backup database in a log sequence using single queue server per queue at a time |
| US5170464A (en) * | 1990-01-26 | 1992-12-08 | International Business Machines Corporation | Method for rolling back an expert system |
| JP3293839B2 (en) * | 1990-05-16 | 2002-06-17 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Computer system that adjusts the commit scope to the unit of work |
| US5327532A (en) * | 1990-05-16 | 1994-07-05 | International Business Machines Corporation | Coordinated sync point management of protected resources |
| DE69126066T2 (en) * | 1990-06-29 | 1997-09-25 | Oracle Corp | Method and device for optimizing logbook usage |
| EP0465019B1 (en) * | 1990-06-29 | 1997-05-14 | Oracle Corporation | Method and apparatus for managing state identifiers for efficient recovery |
| US5317730A (en) * | 1991-01-11 | 1994-05-31 | International Business Machines Corporation | System for modifying persistent database based upon set of data elements formed after selective insertion or deletion |
| US5287501A (en) * | 1991-07-11 | 1994-02-15 | Digital Equipment Corporation | Multilevel transaction recovery in a database system which loss parent transaction undo operation upon commit of child transaction |
| US5313625A (en) * | 1991-07-30 | 1994-05-17 | Honeywell Inc. | Fault recoverable computer system |
| JP3590075B2 (en) * | 1992-01-20 | 2004-11-17 | 株式会社東芝 | Virtual storage data processing apparatus and method |
| EP0554854A3 (en) * | 1992-02-04 | 1996-02-28 | Digital Equipment Corp | System and method for executing, tracking and recovering long running computations |
| GB2273180A (en) * | 1992-12-02 | 1994-06-08 | Ibm | Database backup and recovery. |
| US6259446B1 (en) | 1992-12-23 | 2001-07-10 | Object Technology Licensing Corporation | Menu state system |
| JP3793226B2 (en) * | 1992-12-23 | 2006-07-05 | オブジェクト テクノロジー ライセンシング コーポレイション | Atomic command system |
| JP2557192B2 (en) * | 1993-03-15 | 1996-11-27 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Transaction processing synchronization method, transaction processing monitoring method, and transaction commit processing method |
| GB2281644A (en) * | 1993-09-02 | 1995-03-08 | Ibm | Fault tolerant transaction-oriented data processing. |
| US5530802A (en) * | 1994-06-22 | 1996-06-25 | At&T Corp. | Input sequence reordering method for software failure recovery |
| US5590277A (en) * | 1994-06-22 | 1996-12-31 | Lucent Technologies Inc. | Progressive retry method and apparatus for software failure recovery in multi-process message-passing applications |
| US5666514A (en) * | 1994-07-01 | 1997-09-09 | Board Of Trustees Of The Leland Stanford Junior University | Cache memory containing extra status bits to indicate memory regions where logging of data should occur |
| US5435004A (en) * | 1994-07-21 | 1995-07-18 | International Business Machines Corporation | Computerized system and method for data backup |
| US5596710A (en) * | 1994-10-25 | 1997-01-21 | Hewlett-Packard Company | Method for managing roll forward and roll back logs of a transaction object |
| US5644696A (en) * | 1995-06-06 | 1997-07-01 | International Business Machines Corporation | Recovering multi-volume data sets during volume recovery |
| SE506534C2 (en) * | 1995-06-15 | 1998-01-12 | Ericsson Telefon Ab L M | Way to determine content in restoration log |
| JPH11508070A (en) * | 1995-06-16 | 1999-07-13 | ルーセント テクノロジーズ | Checkpoint recovery system for execution control |
| US6105148A (en) * | 1995-06-16 | 2000-08-15 | Lucent Technologies Inc. | Persistent state checkpoint and restoration systems |
| US6044475A (en) * | 1995-06-16 | 2000-03-28 | Lucent Technologies, Inc. | Checkpoint and restoration systems for execution control |
| US5940612A (en) * | 1995-09-27 | 1999-08-17 | International Business Machines Corporation | System and method for queuing of tasks in a multiprocessing system |
| US5721918A (en) * | 1996-02-06 | 1998-02-24 | Telefonaktiebolaget Lm Ericsson | Method and system for fast recovery of a primary store database using selective recovery by data type |
| GB2311391A (en) * | 1996-03-19 | 1997-09-24 | Ibm | Restart and recovery of OMG compliant transaction systems |
| KR100198805B1 (en) * | 1996-11-22 | 1999-06-15 | 정선종 | A prevention method of dangling transaction with initializing transaction table |
| US5998952A (en) * | 1997-04-09 | 1999-12-07 | Trw Inc. | Method and apparatus for reducing torque ripple in an electric motor using anticasual filtering |
| US5925095A (en) * | 1997-05-12 | 1999-07-20 | International Business Machines Corporation | Method and a computer program product for client/server computing with support for both recoverable and non-recoverable clients |
| JPH1196062A (en) * | 1997-09-19 | 1999-04-09 | Hitachi Ltd | Directory access method |
| US6173292B1 (en) | 1998-03-04 | 2001-01-09 | International Business Machines Corporation | Data recovery in a transactional database using write-ahead logging and file caching |
| US6226651B1 (en) | 1998-03-27 | 2001-05-01 | International Business Machines Corporation | Database disaster remote site recovery |
| US6253212B1 (en) * | 1998-06-23 | 2001-06-26 | Oracle Corporation | Method and system for maintaining checkpoint values |
| US6351754B1 (en) | 1998-06-23 | 2002-02-26 | Oracle Corporation | Method and system for controlling recovery downtime |
| GB2343020A (en) | 1998-10-19 | 2000-04-26 | Ibm | Handling transaction failures in a transaction processing system |
| US6618822B1 (en) * | 2000-01-03 | 2003-09-09 | Oracle International Corporation | Method and mechanism for relational access of recovery logs in a database system |
| US6651073B1 (en) | 2000-05-23 | 2003-11-18 | International Business Machines Corporation | Method and apparatus for insuring database data integrity without data recovery logging |
| US6567928B1 (en) | 2000-05-23 | 2003-05-20 | International Business Machines Corporation | Method and apparatus for efficiently recovering from a failure in a database that includes unlogged objects |
| US6769074B2 (en) * | 2000-05-25 | 2004-07-27 | Lumigent Technologies, Inc. | System and method for transaction-selective rollback reconstruction of database objects |
| US6631374B1 (en) | 2000-09-29 | 2003-10-07 | Oracle Corp. | System and method for providing fine-grained temporal database access |
| US8650169B1 (en) | 2000-09-29 | 2014-02-11 | Oracle International Corporation | Method and mechanism for identifying transaction on a row of data |
| WO2004027604A2 (en) | 2002-09-23 | 2004-04-01 | Neos Financial Systems Limited | Transaction processing system |
| US7308456B2 (en) * | 2002-12-19 | 2007-12-11 | International Business Machines Corporation | Method and apparatus for building one or more indexes on data concurrent with manipulation of data |
| US7890466B2 (en) * | 2003-04-16 | 2011-02-15 | Oracle International Corporation | Techniques for increasing the usefulness of transaction logs |
| US7499953B2 (en) * | 2004-04-23 | 2009-03-03 | Oracle International Corporation | Online recovery of user tables using flashback table |
| US8918367B2 (en) * | 2004-04-30 | 2014-12-23 | Sap Se | Two phase commit emulation for non distributed transactions |
| US7720820B2 (en) * | 2005-07-12 | 2010-05-18 | Microsoft Corporation | Logless persistent components for enterprise applications |
| US8856083B2 (en) * | 2006-06-09 | 2014-10-07 | Oracle International Corporation | Framework to optimize delete all row operations on database objects to improve throughput, query ability and flashback |
| AU2014253672B2 (en) | 2013-04-19 | 2019-05-30 | Commonwealth Scientific And Industrial Research Organisation | Checking undoability of an API-controlled computing system |
| CN108052430B (en) * | 2017-11-30 | 2022-03-18 | 努比亚技术有限公司 | Mobile terminal restart positioning method, mobile terminal and computer readable storage medium |
| KR102326490B1 (en) * | 2020-03-25 | 2021-11-15 | 주식회사 티맥스 소프트 | Pipelining for step input dataset and output dataset |
| US11748212B1 (en) * | 2021-06-28 | 2023-09-05 | Gravic, Inc. | Method and apparatus for resolving automatic transaction facility (ATF) failures |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3564506A (en) * | 1968-01-17 | 1971-02-16 | Ibm | Instruction retry byte counter |
| US4459658A (en) * | 1982-02-26 | 1984-07-10 | Bell Telephone Laboratories Incorporated | Technique for enabling operation of a computer system with a consistent state of a linked list data structure after a main memory failure |
| US4648031A (en) * | 1982-06-21 | 1987-03-03 | International Business Machines Corporation | Method and apparatus for restarting a computing system |
| US4498145A (en) * | 1982-06-30 | 1985-02-05 | International Business Machines Corporation | Method for assuring atomicity of multi-row update operations in a database system |
| US4514846A (en) * | 1982-09-21 | 1985-04-30 | Xerox Corporation | Control fault detection for machine recovery and diagnostics prior to malfunction |
| US4521847A (en) * | 1982-09-21 | 1985-06-04 | Xerox Corporation | Control system job recovery after a malfunction |
| JPS59160899A (en) * | 1982-12-09 | 1984-09-11 | セコイア・システムス・インコ−ポレ−テツド | Memory backup system |
| US4697266A (en) * | 1983-03-14 | 1987-09-29 | Unisys Corp. | Asynchronous checkpointing system for error recovery |
| US4538265A (en) * | 1983-03-24 | 1985-08-27 | International Business Machines Corporation | Method and apparatus for instruction parity error recovery |
| US4641305A (en) * | 1984-10-19 | 1987-02-03 | Honeywell Information Systems Inc. | Control store memory read error resiliency method and apparatus |
| US4665520A (en) * | 1985-02-01 | 1987-05-12 | International Business Machines Corporation | Optimistic recovery in a distributed processing system |
| US4703481A (en) * | 1985-08-16 | 1987-10-27 | Hewlett-Packard Company | Method and apparatus for fault recovery within a computing system |
-
1986
- 1986-03-03 US US06/835,396 patent/US4868744A/en not_active Expired - Fee Related
-
1987
- 1987-01-29 JP JP62017469A patent/JPH0628043B2/en not_active Expired - Lifetime
- 1987-02-04 CA CA000528995A patent/CA1273434A/en not_active Expired - Fee Related
- 1987-02-05 EP EP87101585A patent/EP0236743B1/en not_active Expired - Lifetime
- 1987-02-05 DE DE3788444T patent/DE3788444T2/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| EP0236743A2 (en) | 1987-09-16 |
| JPS62206645A (en) | 1987-09-11 |
| EP0236743A3 (en) | 1989-09-27 |
| JPH0628043B2 (en) | 1994-04-13 |
| DE3788444D1 (en) | 1994-01-27 |
| US4868744A (en) | 1989-09-19 |
| DE3788444T2 (en) | 1994-06-23 |
| EP0236743B1 (en) | 1993-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1273434A (en) | Method for restarting a long-running, fault-tolerant operation in a transaction-oriented data base system without burdening the system log | |
| US6035379A (en) | Transaction processing for user data employing both logging and shadow copying | |
| EP0642079B1 (en) | Fault tolerant transaction-oriented data processing | |
| US4648031A (en) | Method and apparatus for restarting a computing system | |
| JP3672208B2 (en) | Hierarchical transaction processing method | |
| US4498145A (en) | Method for assuring atomicity of multi-row update operations in a database system | |
| US6732124B1 (en) | Data processing system with mechanism for restoring file systems based on transaction logs | |
| Mohan et al. | ARIES: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging | |
| EP0250847B1 (en) | Managing log data in a transaction-oriented system | |
| US6151607A (en) | Database computer system with application recovery and dependency handling write cache | |
| US5933838A (en) | Database computer system with application recovery and recovery log sequence numbers to optimize recovery | |
| US6578041B1 (en) | High speed on-line backup when using logical log operations | |
| US5946698A (en) | Database computer system with application recovery | |
| JPH06318165A (en) | Method for making data available in transaction adaptive system in restart after trouble | |
| WO2007016302A2 (en) | Direct-update software transactional memory | |
| JPH06332778A (en) | Method for transaction management | |
| JPH05108448A (en) | Backup and recovery of data | |
| Kumar et al. | Database recovery | |
| JP3135714B2 (en) | Checkpoint restart method | |
| Strom et al. | A recoverable object store | |
| Skoglund et al. | Transparent orthogonal checkpointing through user-level pagers | |
| EP0834128B1 (en) | Data set backup in a shared environment | |
| US7350104B2 (en) | Invocation of a follow on unit of work | |
| EP0817019A2 (en) | Method of stratified transaction processing | |
| Frantz et al. | Object-Oriented Transaction Processing in the KeyKOS Microkernel. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKLA | Lapsed |