JP2003208323A - Method, system and program for executing batch job - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a load to be applied to a system when many micro batch jobs are simultaneously executed by managing executing states and execution routes of the respective micro batch jobs, to enhance parallelism when micro batch jobs are executed and to shorten intervals to arrival of the last job in the system where many micro batch jobs are simultaneously executed. <P>SOLUTION: Parallel execution of many jobs in short time is enabled by controlling execution of the micro batch jobs generated by a micro batch job generation part by a micro batch job controller and further reducing the load to be applied to the system by the micro batch job by providing a buffer in the case of execution. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer system for executing a process by combining a plurality of batch jobs, and a system for managing the execution and schedule of the batch jobs.

[0002]

2. Description of the Related Art When a plurality of processes and a certain amount of time are required to obtain a certain output result on a computer system, it is common to schedule a plurality of batch jobs in combination and execute them on the system. This is because the batch job can routinely execute a plurality of programs having different processing contents in a fixed order. As a scheduling method for a plurality of batch jobs, there is a method disclosed in Japanese Patent Laid-Open No. 6-59913.

In this method, a plurality of batch jobs are input to the batch job scheduler together with their execution conditions and execution times in advance, and the batch jobs to be executed at the times are automatically extracted and their execution order is considered. You can start the batch job you created.

[0004]

As an example of a computer system using a batch job, when performing analysis work etc. using data in a backbone system database,
A system with a data warehouse (DWH) and a data mart (DM) can be mentioned. At this time, in order to be able to search data at high speed in response to the request of the user who conducts analysis work,
A method is used to execute batch jobs to convert the data stored in the backbone DB that occurred during a certain period in a batch at a time when the system has a relatively large amount of time, to store the data collectively in DWH and DM. ing. The reason why the batch job is executed at night when the system has a relatively large margin is that a batch job usually uses a lot of computer system resources and has a long processing time.

However, when the user wants to retrieve more fresh data, the core DB is not used at night when the system has a relatively large margin, but during the day when the system is normally operated for core business. It is necessary to reflect the above data in DWH and DM. At this time, in order to make the latest data available to the user as soon as possible, a large number of batch jobs targeting the data generated at a certain short fixed time are started, and the backbone DB of the basic system is repeatedly executed in a short time. A possible method is to reflect the data in DWH and DM. In such a case, it is necessary to manage the progress of batch jobs that have been started and are being executed concurrently, and adjust batch jobs that are to be executed concurrently in the system.

For example, performance degradation due to competition of system resources used by executing batch jobs in parallel,
It is necessary to prevent the data inconsistency and the like caused by the job generated by the data generated later overtaking the job generated by the data generated first. In the above conventional technique, a batch job is determined according to the execution time designated by the user in advance and the ending status of the preceding batch job.

For example, the batch job B is a method of controlling the execution of the batch job under the condition that the batch job A cannot be executed until the batch job A is completed. However, with this method, it is not possible to select the optimum batch job in consideration of not only the end status of the preceding batch job but also the batch job being executed and the system resource used by the batch job.

According to the present invention, when a plurality of batch jobs targeting a relatively small amount of data are started in parallel under conditions specified by the user, optimum batch job execution management satisfying these conditions is performed. It will be possible.

As shown in this example, in the present specification, a micro batch job is a job that has a relatively small amount of data to be processed and is completed in a short time under the condition specified by the user, as compared with a general batch job. Call.

An object of the present invention is to manage a large number of microbatch jobs simultaneously in a system in which a large number of microbatch jobs are simultaneously executed by managing the execution status and execution path of each microbatch job. Is to reduce the load on the system, improve parallelism when executing micro batch jobs, and shorten the arrival interval of final jobs.

[0011]

A microbatch job generation unit that generates a table and a job execution time buffer required for microbatch job execution management by using a microbatch job execution condition input by a user. This is achieved by providing a microbatch job controller that manages the created table and the microbatch job execution information in the system to determine the microbatch job to be executed.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to FIGS.

FIG. 1 is a conceptual diagram showing the relationship among a microbatch job controller 101, a microbatch job generator 102, and a microbatch job 103 according to an embodiment of the present invention.

Here, the micro-batch job is a batch job that is started at intervals less than daily depending on conditions such as a fixed time interval or the number of data items specified by the user, or a batch job that uses the result of the batch job. That is. The method of setting these conditions will be described later.

The user operates the microbatch job input unit 10
The execution conditions of the microbatch job are input through 4. The execution conditions will be described later. The microbatch job generation unit 102 manages the execution path of the microbatch job based on this information, and the job path management table 10
5, a concurrent execution job management table 106 that manages the possibility of simultaneous execution of a plurality of microbatch jobs, and a job activation condition table 107 that manages the activation conditions of a new microbatch job.

The execution conditions and execution contents of each microbatch job are stored in the job execution condition table 111.
In addition, the job execution time buffer 110 is generated by the micro batch job to be executed, and is used as the output destination of the micro batch job.

Microbatch job controller 101
Stores the execution status of the microbatch job in the job execution status management table 108 and the information of each microbatch job to be executed in the execution batch information management table 109. The microbatch job controller 101 uses the tables 108 and 109 and the tables 105, 106 and 107 created by the microbatch job generation unit 102,
111 is used to manage the microbatch job 103 executed in the system. Note that each of the above-mentioned tables may have a data structure other than the table, such as information connected by a pointer or tree-structured data, and these are information stored in a recording device accessible by a computer.

FIG. 2 shows an example of the configuration of a system adopting the present invention. Micro batch job controller 101,
The microbatch job generation unit 102 and the job execution buffer 110 are configured in the data warehouse system 201. Here, 101 is a condition that it is connected to a server that executes a microbatch job, and the location is not limited. It suffices that 110 is located in a place that can be used as a data path for a microbatch job. 10
Each function such as 1, 102, 110 may be in the same computer, or may be stored in a recording medium accessible via a network.

FIG. 2 shows that the input client 202
The data sent to the core system server 204 by the data input program 203 and stored in the core DB 206 is converted into the data first conversion unit 208, the data warehouse (D
WH) 211 and the data second conversion unit 213 to store in the data mart (DM) 216, so that the data retrieval program 219 of the information system client 218 can retrieve the data necessary for the analysis work. A house system 201 is shown.

Although the computer on which the data input program 203 operates is shown as the input client and the computer on which the data retrieval program operates is shown as the information client, the client program that inputs and outputs these information is different. This computer may be used, or the same computer may be used.

In a system such as that shown in FIG.
Normally, the backbone system D is used at night when the system has a relatively large margin.
Data extraction of the data generated in a certain period stored in B206, data conversion for storing in DWH,
A method that uses a batch job that reflects the converted data in the DWH 211, extracts the data from the DWH, converts the data for storage in the DM 216, totalizes the data, and executes the processing of reflecting the resulting data in the DM 216. Has been taken.

Microbatch job controller 101
Manages microbatch jobs executed in a computer system to which each computer of the core system server 204, the data warehouse server 209, and the data mart server 214 is connected.

Next, the microbatch job generation unit 102
Generates a job path management table 105, a simultaneous execution job management table 106, a job activation condition table 107, a job execution condition table 111, and a job execution buffer 110. These tables and buffers are used by the microbatch job controller 101 when executing a microbatch job.

The execution conditions input by the user are execution information of each microbatch job, simultaneous execution condition of the microbatch job, and start condition of the microbatch job. The execution condition from the user is the microbatch job input unit 1
04 accepts. Details of these execution conditions are shown below.

First, the execution information of each microbatch job input by the user is obtained from the microbatch job generation unit 102.
Shows the method of storing in the job execution condition table 111.

FIG. 3 shows an example of the job execution condition table 111. FIG. 3 shows an example in which the certain batch job described in FIG. 2 is stored as a plurality of micro batch jobs.

The job execution condition table 111 represents the microbatch jobs executed in the system shown in FIG. 2 as JOB1 to JOB6, and the information of each microbatch job and the place where the microbatch job is executed, etc. It represents the associated information.

The job execution condition table 11 shown in FIG.
Reference numeral 1 denotes an execution job number 307 that indicates the execution order of the microbatch job, a job execution location 308 that indicates the location to execute the microbatch job, a job result reflection destination 309 that indicates the reflection destination of the execution result of the microbatch job, and a microbatch job. Each field of the microbatch job execution program 310 indicating the execution content of The information in the field of the microbatch job execution program 310 may be a program necessary to execute the microbatch job corresponding to the field, or may be a pointer to a file in which the microbatch job execution program is described.

The user inputs the execution job number, job execution location, job result reflection destination, and microbatch job execution program to the microbatch job input unit 104 in the order of execution of each microbatch job. The microbatch job generation unit 102 stores the input execution conditions of the microbatch job in the microbatch job execution condition table 111.

Here, the microbatch job whose execution job number is JOB1 is the first job to be executed in this embodiment, is executed by the data extraction unit 207 in the core system server 204, and is the data from the core system DB 206. This is a microbatch job that executes extraction and transfers the result to the next process JOB2. The numbers shown after JOB indicate the execution order of the microbatch jobs.

The user operates the microbatch job input unit 1
From 04, JOB1 is specified as the job number, a core system server as the job execution location, a data extraction unit, JOB2 as the job result reflection destination, and a file that indicates the program that executes the extraction process shown here in the microbatch job execution program is specified. . The input of this condition is shown in 311.

JOB2 is executed on the data first conversion unit 208, executes data conversion for storing the data of the backbone DB 206 in the DWH 211, and outputs the result to JOB3.
It is a micro batch job to transfer to. An example in which this execution condition is input as in JOB1 is shown in 312. Similarly, JOB3 is a microbatch job that is executed by the data reflection unit 210 in the data warehouse server 209 and that reflects the data converted by JOB2 to the DWH211. An example in which this execution condition is input in the same manner as in JOB1 is shown in 313.

JOB4 is the data warehouse server 2
09 is executed by the data extraction unit 212 in the DWH 211.
It is a microbatch job that executes data extraction from the job and transfers the result to JOB5. An example in which this execution condition is input as in JOB1 is shown in 314.

JOB5 is executed on the second data conversion unit 213, executes data conversion for storing the data of DWH211 in DM216, totalization, etc., and outputs the result to JOB.
This is a microbatch job to be transferred to No. 6. An example in which this execution condition is input as in JOB1 is shown in 315.

JOB6 is a microbatch job which is executed by the data reflection unit 215 in the data mart server 214 and reflects the data converted by JOB5 in the DM 216. An example in which this execution condition is input as in JOB1 is shown in 316.

In the present embodiment, as described above, the microbatch jobs are executed one by one in the order of execution. However, for example, when JOB1 ends, JOB is triggered.
2. Jobs that can execute multiple jobs at the end of one job, such as simultaneous execution of JOB3, JOB1, J
There is a case in which one job can be executed upon completion of a plurality of jobs such that JOB3 cannot be executed unless two jobs of OB2 are completed. For example, in the former case, the job result reflection destination 309 of JOB1 is set to JOB2, J
Register with OB3. In the latter case, the job result reflection destination 309 of JOB1 may be registered by adding a branch number after the job number, such as JOB3-1 and JOB3-2.

Note that this is an example, and an identifier indicating a microbatch job, information indicating a program or a computer for executing the microbatch job, and information indicating a place for storing the execution result of the microbatch job are associated with each other. It is also possible to store, as the job execution condition, information including the above information.

Next, the conditions for simultaneous execution of the microbatch jobs input by the user are determined by the microbatch job generation unit 10.
2 shows a method of storing in the concurrent execution job management table 106.

FIG. 4 shows the concurrent job management table 10
6 has an execution job number 702 and a concurrent execution impossible job number 703 field indicating an execution job number that cannot be simultaneously executed when the microbatch job having the execution job number is executed.

The microbatch job generation unit 102 registers, in the simultaneous execution job management table 106, the information of the microbatch job which is designated by the user and cannot be executed simultaneously.

For example, in the system used in this embodiment, when a plurality of microbatch jobs are simultaneously executed, for example, a JO that writes data to the DWH
JO that reads and extracts data from B3 and DWH
Since exclusive control for processing B4 is frequently performed, a load is placed on the data warehouse server 209, which may affect the performance of data reflection and data extraction.
Further, by continuously processing a large number of microbatch jobs, the time required for the data reflection to the DWH and the data extraction processing becomes long, and there is also a problem that the timing of maintenance is eliminated.

Therefore, in this embodiment, the DWH211
In order to prevent the data reflection microbatch job JOB3 and the data extraction microbatch job JOB4 from being executed at the same time, the job execution number JOB3 cannot be executed concurrently, the job number JOB4 and the job execution number J
Specifies a job number JOB3 that cannot be executed concurrently when executing OB4.

Next, the microbatch job generation unit 102 sets the microbatch job start conditions input by the user.
Shows the method of storing in the job activation condition table 107.

FIG. 5 shows an example of the job activation condition table 107. The user operates the microbatch job input unit 10
From 4, input the microbatch activation conditions. The information input by the user is stored in the microbatch job generation unit 10
2 is stored in the job activation condition table 107.

The job activation condition table 107 shows the types of activation conditions of the microbatch job, the job activation condition type 804, the activation conditions of the microbatch job 805, the application of the activation condition types of the microbatch job. Has fields for a start time, a start time 806, an application end time, and an end time 807.

In each field corresponding to the entry of 802, the number of data items for determining the number of data items to be processed by JOB1 which is the first microbatch job, is determined. Stores the start condition.

The start condition is a data number start condition 802 for determining the number of times the micro batch job is to be started with respect to the number of occurrences of data to be processed by JOB1 which is the first micro batch job, and a fixed time. Every time, a time condition 803 for starting this microbatch job is input. In addition, the user needs to start the microbatch job activation, start time 805, end time of the microbatch job activation, and end time 8 according to these conditions.
Enter 06.

The information input by the user in this way and stored in the job activation condition table is used by the job execution management unit 114 of the microbatch job controller 101. This processing will be described with reference to FIG.

FIG. 6 is an example of the job path management table 105. The execution job group number 502 is a serial number of a combination of microbatch jobs, and the job execution starting point 50 is the job to be executed first in the combination.
3. Fields include an execution job type 504 that indicates a microbatch job of this job group, and a job result reflection part 505 that actually reflects the microbatch job result.

FIG. 7 shows a flow in which the microbatch job generation unit 102 generates the job route management table 105 and the job execution buffer 110 based on the information input by the user through the microbatch job input unit 104. .

Here, an example of inputting the contents of an example of the job execution condition table 111 shown in FIG. 3 is shown.

The user inputs the microbatch job input section 10
4, the microbatch job execution condition table 111 is input according to the above-described microbatch job execution conditions.
Is generated (401). Next, a combination of micro batch jobs that can be continuously executed is registered in the job path management table 105 (402). At this time, the microbatch job that can be continuously executed is that the output destination of the data output by the microbatch job is another microbatch job.

For example, in the job execution condition table 111 shown in FIG. 3, since the reflection destination of JOB1 is JOB2, it is determined that JOB1 and JOB2 can be continuously executed,
JOB1, J as a combination of micro batch jobs
Create OB2. Here, when the job result reflection destination is two jobs, the two jobs can be executed simultaneously at the job result reflection position. For example, if the result reflection destinations of JOB1 are JOB2 and JOB3, (JOB1, JOB
B2), (JOB1, JOB3), (JOB1, JOB
2, JOB3) is a combination of jobs that can be executed simultaneously. When a plurality of job result reflection destinations are one job, a plurality of reflection source jobs are always executed in combination.

However, the reflection source job and the reflection destination job are not combined. For example, the result reflection destination of JOB1 is JOB
When the result reflection destination of 3-1 and JOB2 is registered as JOB3-2, a combination including JOB1 and JOB2 is always created. Do not make a combination that separates JOB1 and JOB2.
Let 3 run. At this time, JOB1, JOB
No combination of 2 and JOB3 is made. The combination of microbatch jobs created in 402 is registered in the job path management table 105 (403). The information to be registered is the execution job group number, the job execution starting point, and the execution job type.

Next, for all the execution job groups in the job path management table 105, the reflection destination of the final job result in the execution job type is acquired from the job execution condition table 111 and registered as the job result reflection position ( 404). FIG. 6 shows an example of the job route management table 105 generated here. At this time, if the reflection destination is the next job, it is registered as the runtime buffer of the next job, such as the JOB2 runtime buffer.

Next, when executing a microbatch job, an inquiry is made to the user by displaying on the screen whether or not there is an unnecessary execution job group. Delete the micro batch job group. Instead of accepting an unnecessary job designation from the user, the selection criteria of the job group, such as the conditions under which the job group is selected or deleted, may be stored in advance in the computer.

For example, in the microbatch job execution method described later, when it is assumed that all microbatch jobs have the same time, the parallelism is improved when the microbatch jobs are not combined and executed one by one. , A micro batch job that combines JOBs is deleted (405).

FIG. 8 shows an example of deleting unnecessary execution job groups and updating the job path management table 105 in the processing of 405.

Finally, for all the job result reflection destinations in the job path management table 105 that are the execution time buffers, the data activation conditions in the job activation condition table 107 are applied to the server or the like where the reflection destination exists. The job execution time buffer 110 having a capacity capable of storing the data is generated (406).

Based on the microbatch job execution conditions input by the user to the microbatch job input section 104, the microbatch job generation section 102 is the information necessary for the microbatch job controller 101 to execute the microbatch job. Route management table 10
5, the concurrent execution job management table 106, the job activation condition table 107, the job execution time buffer 110, and the job execution condition table 111 are generated.

Next, a method for executing the microbatch job in the system shown in FIG. 2 by the microbatch job controller 101 will be described. First, the job execution status management table 108 and the execution job information management table 109 generated by the microbatch job controller 101 for managing the execution of the microbatch job will be described.

FIG. 9 shows the job execution status management table 108.
For example, the transition status of the table contents (job data to be processed by the microbatch job) during execution of the microbatch job is shown. In addition, (1) and (2) of FIG.
By displaying the table information on the screen in the order of (6), it is possible to make the data processing status of each microbatch job easy for the user to understand.

FIG. 9A has elements of a standby job 1202 indicating a currently executable microbatch job and a current status 1203 indicating a microbatch job currently being executed in the system. The first microbatch job to be executed is shown in the leftmost row of the table, the microbatch jobs to be executed sequentially from the left are shown thereafter, and the last microbatch job is shown in the rightmost row. In FIG. 9A, JOB1 (1204) to be executed first is shown, and JOB2 (1205), JOB3 (1206), and JOB are shown below.
4 (1207), JOB5 (1208), and finally JOB6 (1209) are shown in the order of execution.

Here, in each cell of the table, the job data number of the execution target data of the microbatch job activated by the new job generation step 112 described later is stored. This job data number is the job data number of the data to be executed when the first microbatch job (JOB1 in this embodiment) to be executed is activated.
It is represented by a serial number such that it is D1 and the execution target data of the microbatch job that is started next is JD2.

Details of the state transitions (1) to (6) in FIG. 9 will be described with reference to FIG.

Next, FIGS. 10 (1) to 10 (6) show transitions of table contents during execution of a microbatch job, which is an example of the execution job information management table 109. In FIG. 10 (1), the job data number 1302 of the new microbatch job input target generated by the new job generation step 112 and the time when the data used by the microbatch job executed first is last updated. It has a data last update time 1303, and a data location 1304 field that indicates where this data is currently stored.

FIG. 10A shows the job data number JD1.
The storage location of the data to be executed is the backbone DB, and the data is the data that was last updated at 10:40 in the backbone DB. Hereinafter, FIGS. 10 (2) to (6)
Is also the same.

The process of registering the information in the tables shown in FIGS. 9 and 10 will be described with reference to FIG. How the information in each table in FIGS. 9 and 10 changes will be described with reference to FIG.

FIG. 11 shows a processing flow of the new job generation unit 112 of the microbatch job controller 101.

First, the microbatch job controller 101 acquires the number of new data items that can be currently extracted from the data extraction unit 207 of the backbone DB. The number of new data items can be acquired from, for example, the transaction log of the backbone DB 206.

At 901, it is determined whether the new data number satisfies the activation condition of the data number condition of the job activation condition table 107. If not satisfied, in 902, it is determined whether the current time satisfies the start condition of the time condition of the job start condition table 107. If either one is satisfied, the process of 903 is executed, and if not satisfied, the processes of 901 and 902 are continued until they are satisfied. Incidentally, 901
5 is stored in the entry 802 in FIG. 5, and the time condition 902 is stored in the entry 803 in FIG.

At 903, the job data number determined by a serial number as the information of the microbatch job target data and the information of the microbatch job target data is determined and executed together with the last update time of the data targeted by the microbatch job. It is registered in the job information management table 109. For example, “JD1” is stored in the job data number of the execution job management table shown in FIG.
"10:40" is stored in the data last update time.

In this embodiment, since the data to be processed by the first microbatch job is in the backbone DB 206, this information is also registered in the execution job information management table 109. For example, as indicated by the entry 311 in FIG. 3, since the job execution place to be executed first in this embodiment is the data extraction unit of the core system server, the data to be processed by the microbatch job is the core. System D
It can be seen that it exists in B. Based on this, FIG.
A "backbone DB" is stored as information on the data location in (1).

At 904, the first job row in the waiting job column of the job execution status management table 108 (see FIG. 9).
In (1), the job data number “JD1” registered in 903 is registered in JOB1 line).

At 905, it is confirmed whether or not the end condition of the job activation condition table is satisfied, and if it is satisfied, the new job generation process is ended. An example of the job execution status management table 108 generated here is shown in FIG. 9A, and an example of the executed job information management table 109 is shown in FIG. 10A.

FIG. 12 shows a processing flow of the job activation management section 113 of the microbatch job controller 101.

In 1001, triggered by the fact that a new microbatch job has been generated or the already executed microbatch job has ended, the processing proceeds to the next processing 1002.

In 1002, it is determined whether or not all the data to be processed in the microbatch job have completed the microbatch jobs (JOB1 to JOB6 in this embodiment). The last update time of the data stored in the information management table 109 is notified. This allows the user to determine when the currently reflected data was created. If it is not finished, the job execution status management table 1 is shown in 1003 if it is not a new job.
The job data number is registered in the row of the microbatch job to be executed next in the queue job queue 08.

Which data is processed by which microbatch job can be determined by referring to the information in the job execution status management table, as shown in FIG. The final update time of the data can be determined by referring to the information in the execution job information management table, as shown in FIG. By notifying the computer used by the user of these pieces of information, the user can grasp the processing status of each data.

After the processing of the job execution status management 1003 or 1004 at this time, the job execution management unit 114 is activated. The processing of the job execution management unit 114 will be described next.

FIG. 13 shows a processing flow of the job execution management unit 114 of the microbatch job controller 101.

Here, the microbatch controller 1 executes the new job generation step 112 shown in FIG.
It is assumed that 01 starts the new microbatch jobs indicated by job data numbers JD1 and JD2 in this order at regular intervals. Here, FIG. 9A is an example of the transition of the registered contents of the job execution status management table 108 shown in FIG.
10 is an example of the transition of the registered contents of the execution job information management table 109 shown in FIG. 9 (6) and FIG.
This will be described with reference to (1) to FIG. 10 (6).

Here, FIG. 9 (1), FIG. 10 (1), and FIG.
(2) and FIG. 10 (2), FIG. 9 (3) and FIG. 10 (3), FIG. 9 (4) and FIG. 10 (4), FIG. 9 (5) and FIG. 10 (5),
9 (6) and 9 (6) show the contents of the job execution status management table 108 and the contents of the execution job information management table 109 at the same time.

By the processing of the new job generation unit 112 shown in FIG. 11, JD1 is registered in the next execution job row of the waiting job sequence of FIG. 9A, and by the processing of the job activation management unit 113 shown in FIG. It is assumed that the job execution management unit 114 shown in FIG. 13 is activated.

At 1101, the current status of the job execution status management table 108 is confirmed. Since the job data number is not registered in the current status column of FIG. 9 (1),
It can be confirmed that all micro batch jobs are available. Next, in 1102 and 1103, the data number of the standby job which is an empty job is selected. In FIG. 9A, there is only JD1 and JD1 is in JOB1 row 1204, so JOB1 is selected. In FIG. 9 (1), there is no currently executed microbatch job (1104).
Select B1 to JOB6 (1204 to 1209). Next, at 1107, an execution job group starting from the job to be executed is selected from the job path management table 105.

Here, according to the job path management table shown in FIG. 6, the execution job group whose origin is JOB1 is MBJ.
Select 1, MBJ2, MBJ3. Since there are no other waiting jobs (1108), MBJ3 having the largest number of executed microbatch jobs is selected (1109). here,
In the current status column 1203, the job data numbers are registered in the rows of JOB1, JOB2, and JOB3, which are micro batch jobs to be executed by MBJ3, and the execution job group MBJ3 is activated.
Delete JD1 from the queue of waiting jobs. This result is shown in FIG.
It becomes (2). At this time, the execution location of the JOB executed by the MBJ3 and the job execution program are acquired from the job execution condition acquisition table 111.

The data to be executed is acquired from the execution job information management table 109, and the job result reflection part is updated. Here, the data location line 1304 of FIG.
After determining that it is in the core DB, the reflection destination of the MBJ3 is acquired from the job result reflection portion 505 of the job route management table shown in FIG. 6 and updated. In this case, the DWH is acquired and the result is updated in the execution job information management table 106 (1112). The result is shown in FIG.

Next, before the MBJ3 finishes the job for the job data JD1, the microbatch job controller 101 causes the new job generation unit 1 shown in FIG.
It is assumed that 12 creates a new microbatch job.

Since this microbatch job is the second job, the job data number is JD2. The data targeted by the microbatch job at this time is the backbone DB
Then, assuming that the data is finally updated at time 11:30, at 903, the job data number “JD2”,
The data last update time “11:30” and the data location “backbone DB” are registered in the execution job information management table. 90
In 4, the JD2 is registered in the job execution status management table. The contents of the job execution status management table at this time are shown in FIG. 9C, and the contents of the job information management table are shown in FIG.
It shows in (3).

Job execution management step 11 shown in FIG.
In FIG. 4, although there is an empty microbatch job from (3) of FIG. 9 (1101), there is no same waiting microbatch job as the empty job (1102), so this processing ends.
That is, referring to the “current status” column of the table in FIG. 9C, since job data is not registered in the current status column for JOB4 to JOB6, JOB4 to JOB6 are “empty microbatch jobs”. It can be determined, and then, in the “waiting job” column of the table of FIG.
Referring to the columns corresponding to B4 to JOB6, it is determined that the job data is not registered in the column of the waiting job, and the process ends.

Next, it is assumed that MBJ3 finishes the microbatch job for the job data JD1. 9 is an example of the job execution status management table in 1003 of the job activation management step 113 shown in FIG.
JD1 is registered in the JOB4 row of the stand-by job sequence of (4). Since no new job has been generated, the contents of the execution job information management table at this time, FIG. 10 (4), are the same as those in FIG. 10 (3).

Based on the job execution status management table of FIG. 9 (4), the job execution management unit 114 has an empty job (1101), and the same waiting job JOB4 as the empty job,
It is determined that JOB1 is selectable (110
2). Here, since the job closest to the job final end point is JOB4, JOB4 is selected (110).
3). Next, since there is no currently executing job (1104), 11
As a free job at 06, from the row of FIG. 9 (4) which is an example of the job execution status management table, JOB1 to JOB
6 (1205 to 1209) is selected.

Next, at 1107, the execution job group starting from JOB4, MBJ7, MBJ8, and MBJ9, is selected from the job path management table shown in FIG. J
Since there is no waiting job in OB4 and JOB3, which is one before it, (1108), MBJ that has the largest number of executed jobs
9 Select. Here, the job data number is registered in the row of the microbatch job to be executed by MBJ9, JOB4, JOB5, and JOB6 in the execution job column of the job execution status management table, the execution job group MBJ9 is activated, and JD1 is set from the standby job column. It is erased (1111). As a result, the contents of the job execution status management table become as shown in FIG. 9 (5).

At this time, the data to be executed is DWH from the data location in the execution job information management table of FIG. 10 (4).
After determining that there is the above, the reflection destination of MBJ9 is acquired from the job result reflection portion 505 of the job route management table shown in FIG. 6 and updated (1112). In this case, the DM is acquired and the result is updated in the data location. As a result, the result of the execution job information management table is shown in FIG.

Here, based on FIG. 9 (5), which is an example of the job execution status management table, it is determined that there is still an empty microbatch job in the system, and the same standby microbatch job JOB1 as the empty microbatch job is found.
Is selectable (1102).

At 1104, the vacant job JOB3 and the job JOB4 currently being executed cannot be executed from the simultaneous execution job management table 106 shown in FIG.
Is removed from the empty microbatch job, and JOB1 and JOB2 remain as empty microbatch jobs (110
4). Since there are no empty microbatch jobs and standby microbatch jobs, MBJ2 is selected from the job path management table shown in FIG. 6 (1106, 11).
07, 1108, 1109).

Next, the MBJ2 is started, and the job execution status management table and the job information management table are updated in the same manner as the method described above (1111, 1112). The results are shown in FIGS. 9 (6) and 10 (6). At this time, M
By using the runtime buffer 110, the reflection destination of BJ2 can simultaneously execute the maximum number of jobs for JD1 and JD2 in parallel without imposing a load on the data warehouse server 209.

Next, as an example of the job execution status management table 108 shown in FIG. 14, JOB3 and J in FIG.
An execution method when a waiting job is continuous like OB4 is shown.

In this case, JOB4 can execute MBJ7, MBJ8, and MBJ9 from the job route management table shown in FIG. 6 (1107), but the micro batch job immediately before the micro batch job to be executed is the waiting micro batch job. Exists (1109), select the MBJ7 with the smallest number of executed jobs (1110),
Run. In this way, the parallelism of the execution of the microbatch job of the entire system is enhanced by advancing while limiting the microbatch job to be executed when there is a waiting microbatch job subsequently.

At this time, for the waiting job JOB3,
From the concurrent job management table 106 shown in FIG.
Because it cannot be executed when B4 is activated (110
4), it is determined whether there is a next waiting job (1105),
It ends because there is no. FIG. 14B shows an example of the job execution status management table 108 at this time.

In the above-mentioned embodiment, the embodiment is described between the computers connected via the network. However, the embodiment described above is applied to a batch job executed in a certain computer. You may apply.

As described above, by advancing the microbatch jobs in the system, it is possible to improve the parallelism of the microbatch job execution and shorten the arrival interval of the final microbatch jobs.

Further, by using the control section (functions such as a microbatch job controller and a microbatch job generation section) for executing a plurality of computers based on the execution definition information of the batch job shown in this embodiment, From the job execution conditions defined by a plurality of micro batch jobs, an identifier indicating the micro batch job is read from the storage unit, a plurality of combinations of the identifiers indicating the extracted micro batch jobs are generated, and read from the storage unit. By selecting an executable combination from the generated combinations of the identifiers based on the concurrently executed job management information and instructing the computer to execute the microbatch job based on the selected combination, Executing jobs on a computer or multiple computers connected via a network It is possible to perform the execution of the job smoothly between the computer. Further, this allows the administrator to grasp the execution status of the job and the status of the data processed by each job more finely than the conventional batch job.

Further, according to the present invention, the microbatch job generation unit causes the batch job to be executed as a microbatch job based on the condition designated by the user, and the microbatch job controller can execute a plurality of jobs simultaneously in parallel. By doing so, it becomes possible to execute many jobs in a short time.

Thus, in a data warehouse system or the like in which the input data is processed in several steps and used for analysis work of the information system client, the data with higher freshness can be used.

Further, by providing the run-time buffer, it is possible to reduce the load on the system when a large number of microbatch jobs are simultaneously executed and execute the jobs.

Further, the microbatch job controller notifies the user application such as the information system client of the last update time of the data targeted by the microbatch job, so that the user can determine the current data being used. It becomes possible to judge whether there is any.

[0108]

According to the present invention, the microbatch job generation unit executes a job as a plurality of microbatch jobs based on the specified conditions, and the microbatch job controller simultaneously executes a plurality of microbatch jobs in parallel. By enabling it, it becomes possible to execute many jobs in a short time.

[Brief description of drawings]

FIG. 1 is a diagram showing an example of an embodiment of the present invention.

FIG. 2 is a configuration example of a system that implements the present invention.

FIG. 3 is a diagram showing an example of job execution conditions.

FIG. 4 is an example of a concurrent execution job management table.

FIG. 5 is an example of a job activation condition table.

FIG. 6 is an example of a job route management table.

FIG. 7 is an example of a processing flow of a microbatch generation unit.

FIG. 8 is an example of a job route management table.

FIG. 9 is an example of transition of table contents when executing a microbatch job of the job execution status management table.

FIG. 10 is an example of transition of table contents when executing a microbatch job of the execution job information management table.

FIG. 11 is an example of a processing flow of a new job generation step.

FIG. 12 is an example of a processing flow of a job activation management step.

FIG. 13 is an example of a processing flow of a job execution management step.

FIG. 14 is an example of transition of table contents when executing a microbatch job of the job execution status management table.

[Explanation of symbols]

101 ... Micro batch job controller 102 ... Micro batch job generation unit 103 ... Micro batch job 104 ... Batch job input section 105 ... Job route management table 106 ... Concurrent job management table 107 ... Job start condition table 108 ... Job execution status management table 109 ... Execution job information management table 110 ... Job execution buffer 111 ... Job execution condition table 112 ... New job generation unit 113 ... Job start management unit 114 ... Job execution management unit

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Yuichi Yagawa             1099 Ozenji, Aso-ku, Kawasaki City, Kanagawa Prefecture             Ceremony company Hitachi Systems Development Laboratory (72) Inventor Yasuharu Namba             1099 Ozenji, Aso-ku, Kawasaki City, Kanagawa Prefecture             Ceremony company Hitachi Systems Development Laboratory (72) Inventor Shoichi Minami             890 Kashimada, Sachi-ku, Kawasaki City, Kanagawa Stock             Hitachi, Ltd. Financial Systems Division F-term (reference) 5B085 BA01                 5B098 AA10 GA03 GC09 GD02 GD14

Claims (12)

[Claims] 1. A method of executing a batch job in a control unit for executing a plurality of computers based on execution definition information of the batch job, wherein the control unit executes the job by defining the batch job by a plurality of micro batch jobs. The identifier indicating the microbatch job is read from the storage unit from the condition, a plurality of combinations of the identifiers indicating the extracted microbatch jobs are generated, and the combination is generated based on the concurrent job management information read from the storage unit From the combination of the identifiers,
A method for executing a batch job, which comprises selecting an executable combination and instructing a computer to execute a microbatch job based on the selected combination. 2. The job execution condition stores an identifier indicating a microbatch job, information indicating a program for executing the microbatch job or information indicating a computer for executing the microbatch job, and an execution result of the microbatch job. 2. The method for executing a batch job according to claim 1, further comprising information in which the information indicating the storage unit to be stored or the identifier indicating the microbatch job using the execution result of the microbatch job is associated. 3. The job activation condition includes at least the type of job activation condition, the number of job occurrences, the job start time,
2. The batch job execution method according to claim 1, further comprising any one of job end times. 4. The concurrent job management information includes information designating a microbatch job that cannot be executed concurrently with a certain microbatch job.
How to execute the described batch job. 5. A batch job execution system in a computer having a recording unit, wherein the computer reads out job execution conditions stored in the recording unit and designating a certain batch job as a plurality of microbatch jobs, An identifier indicating a microbatch job included in the read job execution condition is extracted, a combination of identifiers indicating the extracted microbatch job is created, and the concurrent execution job management information stored in the recording unit is read,
Based on the read concurrent execution job management information,
From the combination of identifiers indicating the created microbatch job, a combination of identifiers indicating a microbatch job is selected, the job starting condition stored in the recording unit is read, and a combination of identifiers indicating the selected microbatch job is selected. A batch job execution system comprising: a processing unit that transmits an instruction to execute a microbatch job based on the read job activation condition. 6. The job execution condition stores an identifier indicating a microbatch job, information indicating a program for executing the microbatch job or information indicating a computer for executing the microbatch job, and an execution result of the microbatch job. 6. The batch job execution system according to claim 5, wherein the batch job execution system includes information associating information indicating a storage unit or an identifier indicating a microbatch job that uses the execution result of the microbatch job. 7. The job start condition includes at least the type of job start condition, the number of job occurrences, the job start time,
6. The batch job execution system according to claim 5, including any one of job end times. 8. The concurrent job management information includes information designating a microbatch job that cannot be executed concurrently with a certain microbatch job.
Execution system of the described batch job. 9. When a microbatch job is executed based on a combination of identifiers indicating the selected microbatch job and the read job starting condition, execution status information of the executed microbatch job is acquired. 6. The batch job execution system according to claim 5, wherein the micro batch job is scheduled based on the execution status information. 10. When a microbatch job is executed based on a combination of identifiers indicating the selected microbatch job and the read job activation condition, the data to be processed by the executed microbatch job is stored. 6. The batch job execution system according to claim 5, wherein update time information is acquired and update time information of the acquired data is notified. 11. A program for executing a batch job in a computer having a recording unit, wherein the program reads job execution conditions stored in the recording unit, which specify a certain batch job as a plurality of microbatch jobs, An identifier indicating a microbatch job included in the read job execution condition is extracted, a combination of the identifiers indicating the extracted microbatch jobs is created, the concurrent execution job management information stored in the recording unit is read, and Based on the read concurrent execution job management information, a combination of identifiers indicating the micro batch job is selected from the combination of identifiers indicating the created micro batch job, and based on the combination of identifiers indicating the selected micro batch job. , Micro batch jobs Batch job execution program for causing the row. 12. A program for executing a batch job in a computer having a recording unit, wherein the program reads job execution conditions stored in the recording unit and designating a certain batch job as a plurality of microbatch jobs, An identifier indicating a microbatch job included in the read job execution condition is extracted, a combination of the identifiers indicating the extracted microbatch jobs is created, the concurrent execution job management information stored in the recording unit is read, and Based on the information that specifies the microbatch jobs that are not executed at the same time and that is included in the read concurrent execution job management information, the combination of the identifiers indicating the microbatch jobs is selected from the combination of the identifiers indicating the created microbatch jobs, and Selected microbatch job A microbatch job is executed based on the combination of the identifiers shown, and information that associates the identifier of the data processed by the executed microbatch job with the update time of the data and the information indicating the storage location of the data is stored. An execution program for a batch job, which acquires and notifies the acquired information.