JP5397076B2 - Job execution apparatus, job execution method, and job execution program - Google Patents

Description

  The present invention relates to a job execution device, a job execution method, and a job execution program.

  Generally, in a batch processing system, data is sequentially read from an input file according to a job, and the read data is processed and written to an output file. If a failure occurs during processing in such a system, the output data is returned to the state before the processing, and the processing is executed again to recover the data. Patent Document 1 below discloses a technique for restarting processing without returning output data to an initial state before processing even when a failure occurs.

JP-A-6-12267

  In the technique described in Patent Document 1, the number of data output during batch processing is stored. If a failure occurs, the input data is read again from the beginning, the input data for the number of stored output data items is skipped, the process is restarted from the next input data, and the output file is updated. ing.

  However, some batch processing systems employ a batch processing acceleration system (PREST) that can execute a plurality of jobs in parallel. In such a batch processing system, when a part of data is output in the preceding job, the subsequent job is executed in parallel using the output data. Therefore, during batch processing, the data processing state and the number of processes differ for each job executed in parallel. When a failure occurs in such a batch processing system, simply storing the total number of processing cases cannot grasp the data processing status and the number of processing items that differ for each job. It is necessary to return all undefined data to the initial state. That is, since the batch processing is started again from the first job for the data returned to the initial state, it takes time to recover the data after the failure recovery.

  The present invention has been made to solve the above-described problem, and can execute a job execution apparatus and a job that can shorten the data recovery time after failure recovery even when a plurality of jobs are executed in parallel. An object is to provide an execution method and a job execution program.

  A job execution apparatus according to the present invention is a job execution apparatus capable of executing a first job and a second job constituting a batch process in parallel, and a first job execution unit that executes the first job; A data storage unit for storing first data output as a result of execution of the first job by the first job execution unit, and a second unit for processing the first data stored in the data storage unit A second job execution unit that executes a job, a data information storage unit that stores data information including data specifying information for specifying the first data processed by the second job, and a failure has occurred A dump unit that dumps the first data stored in the data storage unit and the data information stored in the data information storage unit.

  The job execution method of the present invention is a method executed in an apparatus that can execute a first job and a second job that constitute a batch process in parallel, and the first job that executes the first job An execution step; a data storage step for storing first data output as a result of execution of the first job in the first job execution step; and a first data processing step for processing the first data stored in the data storage step. A second job execution step for executing the second job, a data information storage step for storing data information including data specifying information for specifying the first data processed in the second job, and a fault If it occurs, dump the first data stored in the data storage step and the data information stored in the data information storage step Including pump and step, the.

  The job execution program of the present invention causes a computer to execute each step included in the job execution method.

  According to the present invention, data recovery time after recovery from a failure can be shortened.

It is a block diagram which shows the function structure of the job execution apparatus in embodiment. It is a figure which shows typically the data flow in a job execution apparatus. 7 is a flowchart for explaining an operation when an agent unit transmits processing data to a subsequent job execution unit. 6 is a flowchart for explaining an operation when an agent unit receives processing result data from a preceding job execution unit. It is a flowchart for demonstrating operation | movement when a failure generate | occur | produces. 10 is a flowchart for explaining an operation when an agent unit receives a commit notification from a subsequent job execution unit.

  Hereinafter, preferred embodiments of a job execution device, a job execution method, and a job execution program according to the present invention will be described with reference to the accompanying drawings.

  First, the configuration of the job execution apparatus in the embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram illustrating a functional configuration of the job execution apparatus. FIG. 2 is a diagram schematically illustrating a data flow in the job execution apparatus. The job execution apparatus employs a batch processing acceleration system (PREST) that can execute a plurality of jobs in parallel.

  As illustrated in FIG. 1, the job execution apparatus 1 includes a plurality of agent units 10, a plurality of job execution units 20, an agent management unit 30, a memory 40, and a dump information file 50.

  The job execution unit 20 is provided for each job and executes the assigned job. In this embodiment, for convenience of explanation, a case where there are two jobs will be described. The job to be executed first is referred to as a preceding job (first job), and the job that executes processing using data output by the execution of this preceding job is referred to as a subsequent job (second job). The job execution unit 20 that executes the preceding job is referred to as the preceding job execution unit 20 (first job execution unit), and the job execution unit 20 that executes the subsequent job is the subsequent job execution unit 20 (second job execution unit). ) Since there are two jobs, the agent unit 10 is one.

  The agent unit 10 is provided between the job execution units 20 and controls data transfer processing from the preceding job to the succeeding job. The agent management unit 30 has a function of managing all the agent units 10 and transmits a failure occurrence notification indicating that a failure has occurred in each agent unit 10 when a failure occurs. Details of the agent unit 10 will be described later.

  The memory 40 includes a processing data storage area 41 (data storage unit) and a processing data information storage area 42 (data information storage unit). As shown in FIG. 2, the processing data storage area 41 stores processing data (first data) output from the preceding job execution unit 20. The processing data is stored in a identifiable manner in units of chunks (data groups). A chunk is a collection of a plurality of data and is a unit of transaction. In this embodiment, a chunk is formed with three data. A number for identifying data (data identification information) is counted up in order of input to data, and a number for identifying chunks (data group identification information) is counted up in order of input to data. The The processing data stored in the processing data storage area 41 is registered when output from the preceding job execution unit 20 and is deleted when a commit is received from the subsequent job execution unit 20.

  A processing data management table 421 and a subsequent processing data number 422 are stored in the processing data information storage area 42. The processing data management table 421 has, for example, a chunk number, a chunk head data number, and a chunk tail data number as data items. The chunk number item stores the chunk number to which the processing data output from the preceding job execution unit 20 belongs. In the item of the chunk head data number, a number (data specifying information) given to the head process data constituting the chunk is stored. In the item of the chunk last data number, a number (data specifying information) assigned to the last process data constituting the chunk is stored.

  In the processing data management table 421, registration and deletion are performed in units of processing data management records (data information) composed of the data items. The processing data management record is registered when the first processing data constituting the chunk is output from the preceding job execution unit 20. The chunk last data number of the processing data management record at the time of registration stores “-” indicating that no data exists yet. Thereafter, when the last process data constituting the chunk is output from the preceding job execution unit 20, the number of the process data is stored in the chunk last data number of the process data management record. That is, when “−” is stored in the chunk tail data number, this indicates that any of the processing data belonging to the chunk is being executed by the preceding job execution unit 20. Further, the processing data management record is deleted when the last processing data constituting the chunk is committed by executing the job in the subsequent job execution unit 20.

  The subsequent processing data number 422 stores a number (data information) given to the processing data being processed by the subsequent job execution unit 20 among the processing data stored in the processing data storage area 41.

  The agent unit 10 will be described in detail with reference to FIG. As shown in FIG. 1, the agent unit 10 includes a processing result data receiving unit 110, an end code checking unit 111, a processing data updating unit 112, a data request receiving unit 120, a processing target data management unit 121, a processing The data acquisition unit 123, the data transmission unit 124, the commit notification reception unit 130, the failure occurrence notification unit 140, the failure occurrence notification reception unit 141, the dump unit 142, and the data recovery unit 143 are included.

  The processing result data receiving unit 110 receives processing result data from the preceding job execution unit 20. The processing result data is output from the job execution unit 20 for each data input to the job execution unit 20. The processing result data includes processing data output as a result of executing the job and a job end code. The processing data stores information indicating whether the data is the tail data of the chunk. The end code stores information indicating whether or not the processing has ended normally. If a failure occurs during job execution, the processing result data does not include the processing data, and only the end code is included.

  The end code confirmation unit 111 determines whether or not the processing has ended normally based on the end code included in the processing result data received from the preceding job execution unit 20.

  The processing data update unit 112 stores the processing data received from the preceding job execution unit 20 in the processing data storage area 41 when the end code confirmation unit 111 determines that the processing has ended normally.

  The processing data update unit 112 determines whether the processing data included in the processing result data is the tail data of the chunk. When the processing data is the last data of the chunk, the processing data update unit 112 searches for a data information management record in which “−” is stored in the chunk last data number. The processing data update unit 112 stores the number of the processing data received from the preceding job execution unit 20 in the chunk last data number of the detected data information management record.

  The data request reception unit 120 receives a data request for requesting transmission of processing data from the subsequent job execution unit 20. When the data request receiving unit 120 receives a data request, the data request receiving unit 120 notifies the processing target data management unit 121 to that effect.

  The processing target data management unit 121 identifies data that has been processed immediately before by the subsequent job execution unit 20 and determines data to be transmitted to the subsequent job execution unit 20. Specifically, the processing target data management unit 121 refers to the subsequent processing data number 422 stored in the processing data information storage area 42, thereby determining the number of data processed immediately before by the subsequent job execution unit 20. Identify. The processing target data management unit 121 notifies the processing data acquisition unit 123 of a number obtained by adding 1 to the specified number as transmission target data information.

  The processing data acquisition unit 123 determines whether data corresponding to the transmission target data information received from the processing target data management unit 121 is stored in the processing data storage area 41. The process data acquisition unit 123 acquires data corresponding to the transmission target data information when data corresponding to the transmission target data information is stored. If the data corresponding to the transmission target data information is not stored, the processing data acquisition unit 123 waits for a predetermined time and then repeats the process of determining the presence or absence of data. If the data corresponding to the transmission target data information cannot be acquired even if the re-determination process is repeated a predetermined number of times, the process data acquisition unit 123 issues an error notification to that effect.

  The data transmission unit 124 transmits the data acquired by the processing data acquisition unit 123 to the subsequent job execution unit 20. The data transmission unit 124 transmits the error notification issued by the processing data acquisition unit 123 to the subsequent job execution unit 20.

  The commit notification receiving unit 130 receives a commit notification from the subsequent job execution unit 20. The commit notification is issued each time processing is confirmed in units of chunks in each job execution unit 20. The commit notification includes the number of the last data in the chunk.

  When the commit notification is received from the subsequent job execution unit 20, the processing target data management unit 121 refers to the chunk tail data number of the processing data management table 421, and this chunk tail data number is included in the commit notification. Delete the processing data management record that is less than or equal to the number of the last data in the chunk.

  When the commit notification is received from the subsequent job execution unit 20, the process data update unit 112 refers to the process data number stored in the process data storage area 41, and the process data number is used as the commit notification. Process data that is less than or equal to the last data number in the included chunk is deleted in units of chunks.

  When the end code confirmation unit 111 determines that the process has ended abnormally, the failure occurrence notification unit 140 transmits a failure occurrence notification indicating that a failure has occurred to the agent management unit 30.

  When the failure occurrence notification receiving unit 141 receives the failure occurrence notification, the failure occurrence notification receiving unit 141 stops the processing functions of the processing data update unit 112 and the processing target data management unit 121. When the failure occurrence notification receiving unit 141 receives the failure occurrence notification, the failure occurrence notification reception unit 141 sends an execution instruction for dump processing to the dump unit 142.

  The dump unit 142 displays the process data stored in the process data storage area 41, the process data management record stored in the process data management table 421, and the process data number stored in the subsequent process data number 422, respectively. Dump to dump information file 50.

  The data recovery unit 143 stores each data stored in the dump information file 50 in the processing data storage area 41, the processing data management table 421, and the subsequent processing data number 422, so that each data in the memory 40 is immediately before the failure. Restore to the state.

  Next, the operation of the job execution apparatus 1 in this embodiment will be described with reference to the drawings. With reference to FIG. 3, the operation when the agent unit transmits processing data to the subsequent job execution unit will be described.

  First, when the data request receiving unit 120 receives a data request from the subsequent job execution unit 20 (step S101), the data request receiving unit 120 notifies the processing target data management unit 121 that the data request has been received (step S102).

  Subsequently, the processing target data management unit 121 refers to the subsequent processing data number 422 stored in the processing data information storage area 42 and specifies the number of the data processed immediately before by the subsequent job execution unit 20 ( Step S103).

  Subsequently, the processing target data management unit 121 transmits a number obtained by adding 1 to the specified number to the processing data acquisition unit 123 as transmission target data information (step S104).

  Subsequently, the processing data acquisition unit 123 determines whether or not processing data corresponding to the transmission target data information received from the processing target data management unit 121 is stored in the processing data storage area 41 (step S105). When this determination is YES (step S105; YES), the processing data acquisition unit 123 acquires processing data corresponding to the transmission target data information from the processing data storage area 41 (step S106).

  Subsequently, the data transmission unit 124 transmits the processing data corresponding to the transmission target data information to the subsequent job execution unit 20 (step S107).

  Subsequently, the processing target data management unit 121 stores and updates the number of the processing data transmitted to the subsequent job execution unit 20 in the subsequent processing data number 422 (step S108).

  On the other hand, when it is determined in step S105 that the processing data corresponding to the transmission target data information is not stored in the processing data storage area 41 (step S105; NO), the processing data acquisition unit 123 determines the number of times of determination. Is counted up (step S109), and it is determined whether the number of determinations exceeds a predetermined number (step S110). If this determination is YES (step S110; YES), the processing data acquisition unit 123 issues an error notification and transmits it to the subsequent job execution unit 20 (step S111).

  On the other hand, when the number of determinations is equal to or less than the predetermined number in step S110 (step S110; NO), the processing data acquisition unit 123 waits for a predetermined time and then proceeds to step S105 described above.

  With reference to FIG. 4, the operation when the agent unit receives the processing result data from the preceding job execution unit will be described.

  First, when processing result data is received from the preceding job execution unit 20 (step S201), the end code confirmation unit 111 determines whether or not the processing has ended normally based on the end code included in the processing result data. Determination is made (step S202). When this determination is NO (step S202; NO), the failure notification unit 140 transmits a failure notification to the agent management unit 30 (step S203). Thereby, the agent management unit 30 transmits a failure notification to all the agent units 10, and each agent unit 10 performs processing at the time of failure.

  On the other hand, when it is determined in step S202 that the process has been normally completed (step S202; YES), the process data update unit 112 stores the process data included in the process result data in the process data storage area 41. (Step S204).

  Subsequently, the process data update unit 112 determines whether or not the process data included in the process result data is the tail data of the chunk (step S205). If this determination is NO (step S205; NO), the process ends.

  On the other hand, when it is determined in step S205 that the processing data is the tail data of the chunk (step S205; YES), the processing data update unit 112 stores “-” in the chunk tail data. A data information management record is searched (step S206). The process data update unit 112 stores and updates the number of the process data included in the process result data received in step S201 in the chunk end data of the detected data information management record (step S207).

  With reference to FIG. 5, the operation when a failure occurs will be described.

  First, when the failure occurrence notification receiving unit 141 receives the failure occurrence notification transmitted from the agent management unit 30 (step S301), the failure occurrence notification reception unit 141 stops the processing functions of the processing target data management unit 121 and the processing data update unit 112 ( Step S302).

  Subsequently, the failure occurrence notification receiving unit 141 sends an execution instruction for dump processing to the dump unit 142 (step S303).

  Subsequently, the dump unit 142 stores the processing data stored in the processing data storage area 41, the processing data management record stored in the processing data management table 421, and the processing data stored in the subsequent processing data number 422. Each number is dumped in the dump information file 50 (step S304).

  Subsequently, when the failure is resolved (step S305; YES), the data recovery unit 143 stores each data dumped in the dump information file 50 in the memory 40 (step S306). Thereby, each data of the memory 40 can be restored to the state immediately before the failure. In addition, it can be determined that the failure has been resolved, for example, by receiving a failure resolution notification transmitted from the agent management unit 30 in accordance with an input operation by the administrator.

  With reference to FIG. 6, the operation when the agent unit receives a commit notification from the subsequent job execution unit will be described.

  First, when the commit notification receiving unit 130 receives a commit notification from the subsequent job execution unit 20 (step S401), the commit notification receiving unit 130 notifies the processing target data management unit 121 of the number of the last data in the chunk included in the commit notification. (Step S402).

  Subsequently, the processing target data management unit 121 refers to the chunk tail data number of the processing data management table 421, and this chunk tail data number is equal to or less than the number of the tail data in the chunk included in the commit notification. The processing data management record is deleted (step S403).

  Subsequently, the processing target data management unit 121 notifies the processing data update unit 112 of the number of the last data in the chunk included in the commit notification (step S404).

  Subsequently, the processing data update unit 112 refers to the number of the processing data stored in the processing data storage area 41, and the processing data number is less than or equal to the number of the last data in the chunk included in the commit notification. Is deleted in units of chunks (step S405). In this way, the memory storage efficiency can be improved by deleting committed data.

  As described above, according to the job execution device 1 in the embodiment, when a failure occurs, it is possible to save unconfirmed processing data together with the processing state when a failure occurs for each job executed in parallel. Therefore, when the failure is recovered, the processing data of each job can be returned to the state at the time of the failure by restoring the saved data. In other words, by restarting the execution of each job after data recovery, it is possible to restart the process from the state at the time of occurrence of the failure. Therefore, even when a plurality of jobs are executed in parallel, the data recovery time after failure recovery can be shortened.

  Here, the job execution apparatus 1 is installed with a program for realizing the functions of the above-described units. By executing this program, the function of each unit described above can be realized.

  Note that the above-described embodiment is merely an example, and does not exclude various modifications and technical applications that are not explicitly described in the embodiment. That is, the present invention can be implemented by being modified into various forms without departing from the spirit of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Job execution apparatus, 10 ... Agent part, 20 ... Job execution part, 30 ... Agent management part, 40 ... Memory, 41 ... Processing data storage area, 42 ... Processing data information storage area, 50 ... Dump information file, 110 ... Processing result data receiving unit, 111... End code checking unit, 112... Processing data updating unit, 120... Data request receiving unit, 121... Processing target data management unit, 123. Commit notification receiving unit, 140... Failure occurrence notification unit, 141... Failure occurrence notification receiving unit, 142... Dump unit, 143... Data recovery unit, 421 .. processing data management table, 422.

Claims (6)

A job execution apparatus capable of executing in parallel a first job and a second job constituting a batch process,
A first job execution unit for executing the first job;
A data storage unit for storing first data output as a result of execution of the first job by the first job execution unit;
A second job execution unit for executing the second job for processing the first data stored in the data storage unit;
A data information storage unit for storing data information including data specifying information for specifying the first data being processed in the second job;
A dump unit that dumps the first data stored by the data storage unit and the data information stored by the data information storage unit when a failure occurs;
A job execution apparatus comprising:   When the second data output as a result of executing the second job by the second job execution unit is committed, the commit is started from the first data stored in the data storage unit. The job execution apparatus according to claim 1, further comprising a data deleting unit that deletes the first data corresponding to the second data. The data information further includes data group specifying information for specifying a data group composed of a plurality of the first data stored in the data storage unit,
The second job execution unit commits the second data for each data group,
When the second data is committed, the data deleting unit changes the first data stored in the data storage unit to the data group corresponding to the committed second data. The job execution apparatus according to claim 2, wherein the first data included is deleted.   4. The data recovery unit according to claim 1, further comprising a data recovery unit that stores the first data and the data information dumped by the dump unit in a storage area at the time of dumping. 5. Job execution device. A job execution method executed in an apparatus capable of executing a first job and a second job constituting a batch process in parallel,
A first job execution step for executing the first job;
A data storage step of storing first data output as a result of execution of the first job in the first job execution step;
A second job execution step for executing the second job for processing the first data stored in the data storage step;
A data information storing step for storing data information including data specifying information for specifying the first data being processed in the second job;
A dump step for dumping the first data stored in the data storage step and the data information stored in the data information storage step when a failure occurs;
Including a job execution method.   A job execution program for causing a computer to execute each step according to claim 5.

Images