JP2526501B2 - A traveling control method for large-volume data processing in computer systems. - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a traveling control system for mass data processing according to a load condition of a computer system which can perform online processing and mass data processing by batch processing.

[0002]

2. Description of the Related Art In a computer system, when a large amount of data processing is edited by batch processing, a large amount of resources such as a memory, a CPU and a disk that constitute the computer system are consumed. If there are other subsystems such as online processing that share resources on the computer system, those other subsystems cannot secure sufficient resources and the throughput of the entire computer system decreases. However, performance problems such as delay in response time will occur.

In a computer system in which online processing and batch processing coexist, it is generally necessary to guarantee the responsiveness of online processing. Therefore, in order to solve the above performance problem, priority is given to online processing. As a method of allocating resources, for example, a method such as dispatching priority for setting the priority of CPU usage, a dedicated memory area, and introduction of a high-speed disk dedicated for online processing is adopted. Further, there is a case where a method for suppressing the occurrence of the above performance problem is adopted by dividing the operating hours of online processing and batch processing.

However, generally, in a system in which online processing and batch processing coexist, unless it is a system in which accounting business and development business coexist (in such a system, the responsiveness of online processing is guaranteed. (There is no need to do this), online processing and batch processing are not related, and since batch processing exists as a background job started from online processing,
Throughput must be guaranteed to some extent in batch processing itself, and it is not possible to adopt a method such as dividing operating hours.

Since it is a very expensive general-purpose computer system, finite resources are used as efficiently as possible,
It is a common desire among users of computer systems that they want to exhibit the maximum performance of the computer system, and it is a natural right as a user. Therefore, allocating resources only for online processing is a luxury and hard-to-use method for users because they want to guarantee the responsiveness of online processing. We would like to implement it by using resources efficiently as much as we do not.

The technique disclosed in JP-A-59-49645 is
In a computer system where online processing and batch processing coexist, the load on the computer system is monitored, and when the load exceeds a certain amount, the batch processing being executed is interrupted in steps and the batch processing is using The resources of the computer system are released to reduce the load on the computer system, and when the load exceeds a certain amount, the suspended batch processing is restarted.

[0007]

Problems to be Solved by the Invention JP-A-59-4964
In the technique disclosed in Japanese Patent No. 5, the interruption point and the resumption point of the batch processing are between the steps constituting the batch processing. However, in batch processing for editing a large amount of data, the processing amount of one step is naturally large and the processing time is long. In the technique of Japanese Patent Laid-Open No. 59-49645, it is difficult to interrupt such a batch processing step once it is executed. Therefore, the load of the computer system becomes saturated until the step is completed and interrupted. It causes performance problems. In such a state, the batch processing step itself does not finish easily, resulting in performance problems such as a reduction in throughput of the entire computer system and a delay in response time.

The technique of the present invention guarantees the responsiveness of online processing in a computer system in which such online processing and batch processing for editing a large amount of data coexist, and the batch processing does not cause a performance problem. The purpose is to be able to use resources efficiently.

[0009]

A travel control method for processing a large amount of data in a computer system according to a first aspect of the present invention comprises an input / output device for inputting a job control language of a batch job in which a user of the computer system processes a large amount of data. A job management subsystem for translating a job control language in an operating system, controlling execution of each step constituting a job, and allocating resources, and an individual step of the batch job submitted from the input / output device. A business main module that is a corresponding program that executes reference / update processing for a large amount of data files and issues a commit command at the end of the processing unit,
By the commit command issued from this business main module, the restart information file in which the journal information of the business main module at that time is stored and the operating status of the computer system resource usage and job execution status are monitored. A performance monitoring subsystem to record, a performance reference value storage file that stores two types of reference values as a performance reference value, that is, a warning value and a limit value for the resource usage rate of the computer system, and the input from the input / output device. In response to the translation time of the job control language of the batch job that processes a large amount of data by the job management subsystem, the job management subsystem is activated by using the identification number of the business main module as a parameter, and the job management subsystem is activated at regular intervals. Use of resources recorded by the performance monitoring subsystem When the resource usage rate exceeds the limit value of the performance reference value in the performance reference value storage file, the job management subsystem is compared with the limit value of the performance reference value in the performance reference value storage file. Is forcibly interrupted by the business main module, the identification number of the business main module is stored in the work area, and the resource usage rate recorded by the performance monitoring subsystem is recorded at regular intervals. Corresponds to the identification number stored in the work area when the resource usage rate is lower than the warning value of the performance reference value in the performance reference value storage file and compared with the warning value of the performance reference value in the performance reference value storage file. A program for restarting the business main module based on the journal information of the business main module stored in the restart information file. It has a large amount of data processing control main that.

A travel control system for processing a large amount of data in a computer system of the second invention is the travel control system for processing a large amount of data in the computer system of the first invention.
A large-volume data processing batch job registration table in which the identification names of the batch jobs that perform large-volume data processing are registered in advance, an online job that performs online processing, and the identification names of batch jobs that are started from this online job are Only the step of determining whether or not it is registered in the large-volume data processing batch job registration table and configuring the batch job with the registered identification name is the business main module at the time of translation, and the business main module is identified. It has a job management subsystem that activates the large-volume data processing control main using the number as a parameter.

[0011]

An embodiment of the present invention will be described in detail below with reference to the drawings.

Referring to FIG. 1, the first embodiment of the present invention is a running control method for processing a large amount of data in a computer system. A computer system user inputs a job control language of a batch job for processing a large amount of data. I / O device 1, translation of job control language in operating system, execution control of each step constituting a job,
And a program corresponding to each step of the batch job submitted from the input / output device 1 and a job management subsystem 2 for executing resource allocation work, and executes reference / update processing for a large amount of data file 4. , The business main module 3 which issues a commit command at the end of the processing unit, and the restart information in which the journal information of the business main module 3 at that time is stored by the commit command issued from this business main module 3. There are two types of standards: a file 5, a performance monitoring subsystem 6 that monitors and records the usage status of computer system resources and job execution status in the operating system, and a warning value and a limit value for the resource usage rate of the computer system. A performance standard value storage file 7 that stores values as performance standard values Identification of the business main module 3 by the job management subsystem 2 in response to the time of translation by the job management subsystem 2 of the job control language of the batch job that processes a large amount of data input from the input / output device 1. The resource usage rate is activated by using the number as a parameter, and the resource usage rate recorded by the performance monitoring subsystem 6 is compared with the limit value of the performance reference value in the performance reference value storage file 7 at regular time intervals. When the performance standard value of the standard value storage file 7 exceeds the limit value, the business main module 3 recognized by the job management subsystem 2 is forcibly interrupted and the identification number of the business main module 3 Is stored in the work area 8 and the resource usage rate recorded by the performance monitoring subsystem 6 is recorded at regular intervals. Compared with the warning value of the performance standard value of the performance standard value storage file 7, if the resource usage rate is lower than the warning value of the performance standard value of the performance standard value storage file 7, the identification number stored in the work area 8 is used. Based on the journal information of the business main module 3 stored in the corresponding restart information file 5, it comprises a mass data processing control main 9 which is a program for restarting the business main module 3. There is.

Next, a running control method for large-volume data processing in a computer system, which is the operation of the first embodiment of the present invention, will be described with reference to FIGS. 1 and 2.

First, the user of the computer system inputs the job control language of a batch job for processing a large amount of data from the input / output device 1.

This job control language is translated by the job management subsystem 2 which is one subsystem of the operating system, but since it is a standard function of the operating system, detailed description will be omitted. Only the functions of the parts relating to the present invention will be described below.

The job management subsystem 2 translates the job control language input from the input / output device 1 and decomposes it into several steps designated by the job control language. In the present embodiment, the program corresponding to one of the steps is the business main module 3, and the job management subsystem 2 activates this business main module 3 after performing processing such as resource allocation for each step. (Step 21).

The business main module 3 repeats the process of inputting a record from the large-volume data file 4 in which a large amount of data is stored, editing it, and then outputting it to the large-volume data file 4 again (step 31). .
In the process of editing a large amount of data, the business main module 3 issues a commit command in units of input / output processing of one record to determine the updated state of the data (step 32).

After issuing the commit command, the business main module 3 processes the number of input / output records processed with the data files in the large volume data file 4 at that time, the storage address of the record, the key information, and the exclusive control. The information and the like are stored as a journal in the restart information file 5 together with the identification number of the business main module 3 (step 33).

The restart information file 5 stores a journal for each identification number of the business main module 3, and a journal having the same identification number is updated instead of being added.

The job management subsystem 2 translates the job control language from the input / output device 1 and, in response to the time when the job control language is decomposed into several steps designated by the job control language, the job main subsystem corresponding to the step is executed. The mass data processing control main 9 is started using the identification number of the module 3 as a parameter (step 22).

The mass data processing control main 9 is
Of the resource usage rates recorded by the performance monitoring subsystem 6 at regular intervals, the CPU usage rate and the memory page fault occurrence rate are referenced in this embodiment (step 91).

The performance monitoring subsystem 6 is one subsystem of the operating system, and records the use status of each resource constituting the computer system, for example, resources such as CPU, memory, and external storage device at regular time intervals,
Then, the execution status of a group of jobs executed on the computer system is recorded when the jobs are completed.

The large data processing control main 9 is
The CPU usage rate and the memory page fault occurrence rate referred to in step 91 are compared with the limit value of the performance reference value stored in advance in the performance reference value storage file 7 (step 92).

The performance reference values stored in the performance reference value storage file 7 in this embodiment are, for example, a CPU utilization limit value of 90%, a warning value of 75%, and a page fault occurrence limit value of (job The multiplicity × 3) times / second and the warning value are (job multiplicity × 2) times / second, which can be arbitrarily designated by the user of the computer system in advance.

The large data processing control main 9 is
In step 92, both the CPU usage rate and the page fault occurrence rate of the memory exceed the limit value of the performance reference value in the performance reference value storage file 7 (step 9).
3), the job management subsystem 2 in step 22
The business main module 3 corresponding to the identification number passed from the computer is forcibly interrupted (step 94). Step 9
If the CPU usage rate and the page fault occurrence rate of the memory are neither above the limit value of the performance reference value in the performance reference value storage file 7 in 3, the process returns to step 91.

The large data processing control main 9 is
In the interruption processing of the business main module 3 in this step 94, the journal information at the time of issuing the final commit command stored in the restart information file 5 is input by using the identification number of the business main module 3 as a key, and the business main module 3 is input. The resource updated after the main module 3 issues the commit instruction is restored to the state at the time of issuing the commit instruction (step 95), and the business main module 3
Releases the exclusive control of the resource targeted by the exclusive control (step 96).

The large data processing control main 9 is
After the restoration process of step 95 is completed, the identification number of the interrupted business main module 3 is recorded in the work area 8 (step 97).

The large data processing control main 9 is
When the identification number of the work main module 3 is recorded in the work area 8, the CPU usage rate and the page fault occurrence rate of the memory recorded by the performance monitoring subsystem 6 are referred to at regular time intervals (step 98), and the performance standard is preset. The performance reference value stored in the value storage file 7 is compared with the warning value (step 99).

The large data processing control main 9 is
In step 99, both the CPU usage rate and the page fault occurrence rate of the memory are lower than the warning value of the performance reference value in the performance reference value storage file 7 (step 9
A), the work main module 3 corresponding to the identification number recorded in the work area 8 is restarted (step 9B). If neither the CPU usage rate nor the page fault occurrence rate of the memory is below the warning value of the performance reference value in the performance reference value storage file 7 in step 9A, the process returns to step 98.

The large data processing control main 9 is
In the restart process of the business main module 3 in step 9B, the journal information at the time of issuing the final commit command stored in the restart information file 5 is input using the identification number of the business main module 3 as a key,
Exclusive control is performed on the resource for which exclusive control was performed when the business main module 3 issued the commit command (step 9C).

Thereafter, in the business main module 3, similarly to step 31, a record is input from the large amount of data stored in the large amount data file 4, edited, and then stored again in the large amount data file 4. The above process is repeated to issue a commit command in units of input / output process of one record to determine the updated state of data.

As described above, the processing of the traveling control method for processing a large amount of data in the computer system, which is the operation of the first embodiment of the present invention, is completed.

The running control method for large-volume data processing in the computer system according to the first embodiment of the present invention guarantees the responsiveness of online processing in a computer system in which online processing and batch processing for editing a large amount of data coexist. In addition, batch processing also has an effect that resources can be efficiently used to the extent that performance problems do not occur.

Referring to FIG. 3, a second embodiment of the present invention, a computer system of a large amount of data processing run control system, a large amount of data processing in the computer system of the first invention of the running control system in advance. The large-volume data processing batch job registration table 10 for registering the identification name of the batch job that executes data processing, the online job 11 that executes online processing, and the identification name of the batch job that is started from this online job 11 are
Only the step of determining whether or not the batch job having the registered identification name is registered in the large-volume data processing batch job registration table 10 is set as the business main module 3 at the time of translation, and the business main module is further processed. A job management subsystem 2 for activating the large-volume data processing control main 9 using the identification number 3 as a parameter is configured.

FIG. 3 shows the processing of the job management subsystem 2 which is the operation of the second embodiment of the present invention and which is a feature of the running control system for processing a large amount of data in the computer system.
Will be described with reference to FIG.

When the online job 11 activates a batch job, the job management subsystem 2 for translating the batch job has a large amount of data in which the identification name of the batch job is registered beforehand with the identification name of the batch job for executing the large amount of data processing. It is determined whether or not it is registered in the processing batch job registration table 10 (step 201). If the identification name is registered in the large-volume data processing batch job registration table 10 in step 201, only the step forming the batch job of the identification name is started as the business main module 3 at the time of translation (step 20).
2). After step 202 ends, the process returns to step 31.

Further, the job management subsystem 2 activates the large volume data processing control main 9 by using the identification number of the business main module 3 as a parameter (step 203). After step 203 ends, the process returns to step 91.

In the travel control method for large-volume data processing in the computer system according to the second embodiment of the present invention, various batch jobs executed from online processing are preliminarily identified by batch job IDs for executing large-volume data processing. By comparing the registered large-volume data processing batch job registration table, it is possible to limit the batch jobs for which running control is performed, so that the performance of the entire computer system can be managed more reliably than in the first embodiment, and the job can be performed. Has the effect that can be carried out.

[0039]

As described above, the running control method for large-volume data processing in the computer system of the present invention improves the responsiveness of online processing in a computer system in which online processing and batch processing for editing a large amount of data coexist. It has the effect that it can be guaranteed, and that batch processing can be efficiently performed by using resources to the extent that performance problems do not occur.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a flow chart showing processing of a job management subsystem 2, a job main module 3, and a large amount of data processing control main 9 according to the first embodiment of this invention.

FIG. 3 is a block diagram showing a second embodiment of the present invention.

FIG. 4 is a flow chart showing the processing of the job management subsystem 2 according to the second embodiment of this invention.

[Explanation of symbols]

 1 I / O device 2 Job management subsystem 3 Job main module 4 Large data file 5 Restart information file 6 Performance monitoring subsystem 7 Performance reference value storage file 8 Work area 9 Large data processing control main 10 Large data processing batch job registration table 11 online jobs

Claims (2)

(57) [Claims] 1. An input / output device into which a user of a computer system inputs a job control language of a batch job for processing a large amount of data, translation of the job control language in an operating system, execution control of each step constituting a job, and A job management subsystem that performs resource allocation work, and a program that corresponds to each step of the batch job that is input from the input / output device. Task main module that issues a commit command at the end of the operation, the restart information file that stores the journal information of that task main module at that time by the commit command issued from this job main module, and the computer in the operating system. System resources A performance monitoring subsystem that monitors and records usage status and job execution status, and a performance standard value storage file that stores two types of standard values, a warning value and a limit value, for the resource usage rate of the computer system as performance standard values. And, in response to the time of translation by the job management subsystem of the job control language of the batch job that processes a large amount of data input from the input / output device, the job management subsystem determines the identification number of the business main module. It is activated as a parameter and compares the resource usage rate recorded by the performance monitoring subsystem at regular time intervals with the limit value of the performance reference value in this performance reference value storage file,
When the resource usage rate exceeds the limit value of the performance standard value in the performance standard value storage file, the business main module recognized by the job management subsystem is forcibly interrupted and the business main module is forcibly suspended. The identification number of the resource is stored in the work area, and the resource usage rate recorded by the performance monitoring subsystem at regular time intervals is compared with the warning value of the performance reference value in the performance reference value storage file. Is below the warning value of the performance standard value in the performance standard value storage file, based on the journal information of the business main module stored in the restart information file corresponding to the identification number stored in the work area. And a mass data processing control main that is a program for restarting the business main module. Cruise control system of large amounts of data processing in the system. 2. A large-volume data processing batch job registration table in which an identification name of the batch job that executes large-volume data processing is registered in advance, an online job that executes online processing, and a batch job that is started from this online job. Only the step of determining whether or not the identification name of is registered in the large-volume data processing batch job registration table and configuring the batch job of the registered identification name is the business main module at the time of translation. 2. The travel control method for mass data processing in a computer system according to claim 1, further comprising a job management subsystem for activating the mass data processing control main using the identification number of the business main module as a parameter.