JP2012221374A - Batch job management server, batch job processing system and batch job execution method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a batch job processing system and a batch job execution method for not concentrating I/O timing to a shared storage medium of divided jobs within a range satisfying re-execution time when a fault occurs, in the batch job execution method using a plurality of file storage media of different characteristics.SOLUTION: A storage medium ratio to respective media is determined from a storage medium performance on the basis of divided job information in a job net, storage medium information and parameter information indicating a condition in job execution, and the divided jobs are allocated to the shared storage medium from the one of the longer re-execution time on the basis of the storage ratio. When determined that a re-execution time threshold of the parameter information is exceeded, return to a re-execution start job of the divided job having the longest re-execution time is performed, and the storage medium ratio is determined again.

Description

  The present invention relates to a batch job management server, a batch job processing system, and a batch job execution method in a computer system, and more particularly to a batch job system and execution method in a computer system provided with a plurality of file storage media having different characteristics in the system. About.

  In various corporate information systems including financial institutions, batch processing systems that process a large amount of data collectively are widely used. In recent years, in order to reduce system cost, batch processing systems that execute batch jobs in parallel using a large number of inexpensive servers have been constructed. Local jobs such as HDD (Hard Disk Drive), SSD (Solid State Drive), and memory are used to execute batch jobs. In batch processing systems that use multiple job execution servers, Therefore, a shared file server for sharing data is also used. As the number of job execution servers increases, access to the shared file server is concentrated and becomes a bottleneck, so the performance of the shared file server must be improved, such as increasing the number of hard disk drives, resulting in higher costs. There is a problem. Therefore, it is necessary to use a plurality of storage media having different characteristics, such as a local storage medium and a storage medium on a shared file server.

  Japanese Patent Application Laid-Open No. 2004-151820 discloses a method of re-executing a job or executing an alternative job in consideration of the remaining time until a predetermined completion time when a failure occurs in the execution of a batch job.

  In Patent Document 2, from the viewpoint of determining an arrangement location based on performance information of a storage medium, a plurality of access commands in an issuance standby state are analyzed, and access is performed so as to shorten the access time of the storage medium alone. A method for scheduling the issue order of commands is disclosed.

  Non-Patent Document 1 discloses a general technique for sharing a file using a local HDD.

JP 2009-181497 A JP 2009-163499 A

Google File System (Gemawat S. et al. “The Google File System, Proc. SOSP '03, pp. 29-43, 2003, http://portal.acm.org/citation.cfm?id=1165389.945450)

  Among the tasks performed by batch processing, there is a task that must be completed by a predetermined time, such as batch processing of data accumulated in the business hours of the previous day by the business start time of the next day. However, the amount of data handled in batch processing may vary, and the time required to execute a batch job varies depending on the amount of data. Further, when a failure occurs in the batch processing system, the recovery process is generally performed by re-execution of the job, but this delays the completion time of the batch job. Even in such a situation, the batch job must be completed at a predetermined time.

  The batch processing system which is the subject of the present invention relates to a batch job system and execution method using a plurality of file storage media having different characteristics. In such a batch processing system, a large number of job execution servers having a local storage are used. Further, since it is necessary to exchange data between job execution servers, a shared file server (shared storage medium or shared storage) for sharing data is provided. Local storage media (local storage) such as HDDs, SSDs, and memories provided in each job execution server are inexpensive, but have low reliability and cannot take over data to other servers in the event of a server failure. On the other hand, a shared storage medium (shared file server) is generally highly reliable but expensive. Also, as the number of job execution servers increases, access to the shared file server is concentrated, which is likely to become a bottleneck.

  If only local storage is used to execute a batch job, the batch job execution result of that server cannot be accessed when a server failure occurs, so the entire batch job must be re-executed on another server, and the job re-execution time is long. It will be time.

  On the other hand, when a batch job is executed using only a shared storage medium (shared storage), other servers can take over job execution when a server failure occurs, but access from many job execution servers is concentrated on the shared storage Therefore, there is a possibility that the file access speed is lowered and the normal batch job execution time becomes long.

  In each of the above prior art documents including Non-Patent Document 1, sufficient consideration is not given to the fact that accesses from a large number of job execution servers are concentrated on the shared storage medium.

  Patent Document 1 discloses a method of executing a job again or executing an alternative job when a failure occurs in the execution of a batch job. If the remaining time is insufficient, skip the job. Alternatively, since a simple alternative job is executed, there is a possibility that subsequent batch jobs and operations will be affected. Further, there is a possibility that I / O (Input / Output) timings are concentrated on the shared storage of the divided jobs.

  In Patent Document 2, although the data storage unit includes a plurality of storage media that store data in a redundant form, it does not handle storage media having different characteristics. In addition, there is no disclosure regarding the processing when a failure occurs in the execution of a batch job. In a storage medium that is not shared between computing nodes, there is a problem that data is lost in the event of a failure and additional re-execution time is required. Also, when applied to a job, the job output file becomes the input file for the next job, so there is a problem that the order cannot be changed.

  Non-Patent Document 1 does not disclose any countermeasures when a failure occurs in the execution of a batch job.

  As described above, in each of the above prior art documents, in the batch job execution system and method using a plurality of file storage media having different characteristics, the shared storage of divided jobs within a range satisfying the re-execution time at the time of failure There is no disclosure of a system or method that does not concentrate I / O (Input / Output) timing.

  An object of the present invention is to provide a batch job execution system and method using a plurality of file storage media having different characteristics in the present invention to a shared storage medium for divided jobs within a range satisfying the re-execution time at the time of failure. The present invention provides a batch job processing system and a batch job execution method that do not concentrate the I / O timing.

The representative ones of the present invention are as follows. A batch job management server that manages execution of batch jobs in a batch job processing system, wherein the batch job processing system includes at least one batch job execution server connected to the batch job management server via a network, and characteristics A plurality of different file storage media, the file storage media including one or more local storage media and one or more shared storage media,
The batch job management server
A job net that is a group of a series of jobs, job information that represents information on a divided job that is divided into jobs to be executed in parallel, storage medium information that represents information on the file storage medium, and conditions for job execution An information input unit for accepting setting of parameter information representing
A storage medium ratio determining unit that determines a storage ratio in each storage medium from the storage medium performance of the storage medium information;
A job storage medium determination unit that allocates to the shared storage medium from a divided job having a long re-execution time based on the storage ratio;
A re-execution time calculation unit for calculating the re-execution time of the divided job;
It is determined whether or not the re-execution time threshold of the parameter information has been exceeded, and when it is determined that the re-execution time threshold has been exceeded, the optimal return to the re-execution job of the split job having the longest re-execution time And an arrangement pattern generation unit.

  According to the present invention, in a batch job execution system and method using a plurality of file storage media having different characteristics in execution of a batch job, the file to each medium of the job within a range satisfying the re-execution time at the time of failure It is possible to provide a batch job processing system and a batch job execution method for controlling the storage ratio, and it is possible to reduce costs such as reducing the number of disks of the shared file server.

1 is a block diagram showing a hardware configuration of a computer system that executes a batch job according to a first embodiment of the present invention. FIG. 3 is a job net diagram illustrating an example of a job net in the first embodiment. 10 is a table showing job information in the first embodiment. 6 is a table showing file storage medium information in the first embodiment. The table showing the input parameter in Example 1. FIG. FIG. 6 is a diagram illustrating an example of a generation process of a job re-execution time management table that manages a job re-execution time in the first embodiment. 3 is a table showing a compression method in the first embodiment. 10 is a table for managing job compression information according to the first embodiment. 3 is a flowchart showing overall processing in the first embodiment. 6 is a diagram illustrating an example of a generated job re-execution time management table in Embodiment 1. FIG. FIG. 3 is a diagram illustrating a concept of allocation of a local storage medium and a shared storage medium to a divided job across a plurality of stages in the first embodiment. 5 is a flowchart illustrating processing for determining a storage ratio in each medium from file storage medium performance in the first embodiment. 9 is a flowchart illustrating processing for assigning a shared storage medium (shared storage) from a divided job having a long re-execution time based on a storage ratio in the first embodiment. 6 is a flowchart illustrating processing for correcting medium performance from a CPU time and an I / O amount of a job according to the first exemplary embodiment. 3 is a flowchart illustrating processing for correcting medium performance based on a compression method according to the first exemplary embodiment. The table which manages the re-execution time of the job which becomes 2nd Example of this invention. 9 is a flowchart showing overall processing in Embodiment 2. The block diagram which shows the principal part of the hardware constitutions of the computer system which performs a batch job which becomes the 3rd Example of this invention. 10 is a diagram illustrating an example of a storage medium information table in Embodiment 3. FIG.

  Generally, a batch job is composed of a plurality of jobs having dependency relationships. A certain business process is realized as a flow of a series of jobs, and a group of the series of jobs is called a job net. When local storage and shared storage are used separately as file storage media in divided job units within a batch job, all divided jobs issue I / O to the shared storage, and all other divided jobs issue I / O to the local storage. O is issued, and there is a problem that the I / O timing of each divided job is concentrated.

  Therefore, according to the present invention, when processing a job net, the ratio of each storage medium of the divided job is determined based on the medium performance of each medium of the shared storage medium (shared storage) and the local storage, and the rerun has a long re-execution time. By assigning a job to a shared storage medium, a means and method for reducing concentration of I / O timing on the shared storage medium are provided.

  That is, the batch job management server of the present invention includes at least one batch job execution server having a file storage medium, and a shared file server that has a file storage medium and processes a file access request from the batch job execution server Is determined via the storage medium performance based on parameter information indicating job division, job information in the job net, and parameter information indicating job execution conditions. Based on the ratio, the job is assigned to the shared storage medium from the divided job having a long re-execution time. When it is determined that the parameter information re-execution time threshold is exceeded, the process returns to the re-execution start job of the divided job having the longest re-execution time, and the storage medium ratio is determined again.

When the batch job management server assigns a divided job to each storage medium and the re-execution time of any of the divided jobs exceeds a preset job re-execution time threshold, the batch job management server sets the shared storage medium write weight. By increasing the storage ratio, the storage ratio in the shared storage medium is increased, the divided job having the longest re-execution time is returned, and the reallocation is executed. Therefore, it is possible to alleviate the concentration of access to the shared file server in divided job units, and even when a failure occurs, a batch job can be executed while satisfying a predetermined completion time.
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

A batch job processing system and a batch job execution method according to the first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram showing a hardware configuration of a computer system that executes a batch job, that is, a batch job processing system, according to the first embodiment of the present invention.

  In FIG. 1, a network 101 includes a batch job execution server 110 that executes batch jobs, a shared file server 130 that shares files between job execution servers, and a batch job management server 150 that manages execution of batch jobs. Is connected.

  The job execution server 110 is a CPU 111 that performs arithmetic processing, a memory 112 that stores programs and data, a communication interface 113 that communicates with other servers via the network 7, and a file storage medium that stores files 116. A local disk hard disk 115, a local solid state disk 116, and a disk controller 114 for controlling the local disk are provided. The file is stored in one or more of the file 117 on the local disk, the file 118 on the local solid state disk, and the file 119 on the memory.

  The memory 112 of the job execution server 110 includes a job program 120 that performs job processing, a job execution control unit 121 that controls job execution, a compression / decompression unit 122 that compresses and decompresses files 116, 117, and 118, A file access control unit 121 that controls file access of the job program exists as a program.

  The shared file server 130 includes a CPU 131 that performs arithmetic processing, a memory 132 that stores programs and data, a communication interface 133 that communicates with other servers via the network 101, and the storage system 140 via the storage network 102. A storage interface 134 for accessing is provided.

  The memory 132 of the shared file server 130 stores a file 143 of the hard disk 141 as a file storage medium of the storage system 140 or a solid state disk 142 as a file storage medium in response to a request from the file access control unit 123 of the job execution server 110. A file server control unit 135 that reads and writes the file 144 exists as a program.

  The storage system 140 is connected to the storage interface 134 of the shared file server 130 via the storage network 102 and includes a redundant hard disk 141 or a redundant solid state disk 142.

  The batch job management server 150 includes a CPU 151 that performs arithmetic processing, a memory 152 that stores programs and data, a communication interface 153 that communicates with other servers via the network 101, a local disk 155 that stores files, A disk controller 154 for controlling the local disk is provided.

  In the memory 152 of the batch job management server 150, an optimum arrangement pattern generation unit 161 and a job execution control unit 168 that controls batch job execution in the job execution server exist as programs. Each unit of the batch job processing system uses memory and file data, and executes various programs loaded in the memory by the CPU, thereby realizing their processing functions.

The optimal arrangement pattern generation unit 161 implements the following functions by executing a program. That is, information input for inputting batch job information, information on a medium for storing a file, parameters for setting execution conditions of the batch job, and compression information are input to the optimum arrangement pattern generation unit 161. Unit 162, a storage medium ratio determining unit 163 that determines a ratio of a medium that stores a file, a job storage medium determining unit 164 that determines which storage medium each divided job stores a file, and a batch job There is a processing time calculation unit 165 that calculates the processing time of the job and a re-execution time calculation unit 166 that calculates a processing time for re-executing the job in the event of a failure. Reference numeral 167 denotes a compression method determination unit that sets a compression method for compressing and storing a batch job in a storage medium.
The local disk 155 also includes a job information table 171 that stores job information of batch jobs, a storage medium information table 172 that stores characteristic information of file storage media, and a parameter table that stores information on execution conditions of batch jobs. 173, an execution job server for each of the divided jobs, a storage medium, a job re-execution time management table 174 for managing re-execution time, a compression information table 175 for storing characteristic information of a method for compressing files, and a job A job compression method management table 176 that manages the compression method to be used is stored.

  In the above configuration, the storage medium ratio determining unit 163 determines the storage medium ratio of the divided jobs according to the input job information table 171, storage medium information table 172, and parameter information table 173. The job storage medium determination unit 164 manages the re-execution time by referring to the information in the job re-execution time management table 174, that is, the optimum arrangement pattern, according to the determined storage medium ratio, and determines the storage medium of the divided job. decide. Regarding the management of the re-execution time, the result of calculating the processing time of the job by the processing time calculating unit 165 and the result of calculating the re-execution time of the job by the re-execution time calculating unit 166 are used. Further, the storage medium ratio determination unit 163 may further refer to the input job information table 171 and the compression information table 175 to determine the storage medium ratio. Furthermore, the optimum arrangement pattern generation unit 161 includes a progress monitoring unit 169 that monitors the progress status of the batch job, and the storage medium ratio determination unit 163 stores the storage medium according to the monitoring result obtained by the progress monitoring unit 169. The proportion may be corrected.

  The job execution control unit 119 of the job execution server 110 activates the job program 118 according to an instruction from the job execution control unit 168 of the batch job management server 150, and executes the job based on the optimum file arrangement pattern. The job program 116 reads or writes the file 117 stored in the local disk 115, the file 118 stored in the local solid state disk 116, or the file 119 stored in the memory 112 via the file access control unit 121. To perform business processing. Alternatively, the job program 118 is stored in the file 143 stored in the local disk 141 of the storage system 140 or the local solid state disk 142 via the file access control unit 121 and the file server control unit 135 of the shared file server 130. The file 144 is read and written, and business processing is performed.

  In FIG. 1, only two job execution servers and shared file servers are shown, but the present invention is not limited to this, and the present invention can be applied to any number of job execution servers and shared file servers. It is. The storage medium of the job execution server 110 is illustrated as a local hard disk 115, a local solid state disk 116, and a memory 112. However, the present invention is not limited to this, and the present invention can be applied to any storage medium. . Although the storage medium of the storage system 140 shows a local hard disk 141 and a local solid state disk 142, the present invention is not limited to this, and the present invention can be applied to any storage medium.

A certain business process is realized as a flow of a series of jobs, and a group of the series of jobs is called a job net. FIG. 2 is a job net diagram illustrating an example of a job net.
The first input file of the job net is called a start input file 200, and the last output file is called a result output file 240. A file used for passing data between jobs is called an intermediate file 220.

  The input file for job A is divided according to the number of job execution servers 110 and the number of CPUs, and job A is executed in parallel using different input files on a plurality of job execution servers 110. Similarly, job B to job F are executed in parallel on a plurality of job execution servers 110. The job A 210 is divided and includes a divided job A 1 211, a divided job A 2 212, and a divided job A 3 213. The start input file 200 is divided into start input files 201, 202, and 203. The result output file 240 is divided into result output files 241, 242, and 243. The intermediate file 220 is divided into result output files 221, 222, and 223.

  As an example of input file division, in payment processing at a financial institution, data is divided using an account number or area number as a key, and processing is assigned to a job execution server. The key for dividing the data is called an input data division attribute. The division attribute is determined according to the job processing contents of each job.

  FIG. 2 shows a state where the divided job A1 211 reads the input file 201 serving as input data, performs business processing such as data processing and aggregation, and outputs the result to the intermediate file 221. Further, the intermediate file 221 output from the divided job A1 211 is read and processed by the divided job B1 and the divided job D1, and the result is output to the intermediate file. Similarly, the divided job F1 reads and processes the output result files of the divided job C1 and the divided job E1, and outputs the result. In this way, each divided job reads and processes one or more input files, and writes one or more output files.

  FIG. 3 is an example of a job information table 171 in which information related to the job net in FIG. 2 is represented by a table. When there are a plurality of jobs, a name (A to F) is assigned to each job.

  In the job information table 171, for each job net name 301, here the job net A, the job name 302 included in the job net, the preceding job name 303 of each job, the succeeding job name 304, and the job input / output file must be shared. The sharing mandatory column 305 indicating whether the input file is uncompressed, the output date compression column 306 indicating whether the input file is uncompressed, the output uncompressed column 307 indicating whether the output file is uncompressed, and the CPU time of the job. A CPU time column 309 that represents the job, an I / O amount column 309 that represents the job I / O amount, and a job division number column 310 that represents the number of job divisions are included.

  In the case of JOB_A, since it is the first job in the job net, there is no preceding job, and since the output file of JOB_A is used by JOB_B and JOB_D, the subsequent job names are “JOB_B, JOB_D”. In the case of JOB_F, since the output files of JOB_C and JOB_E are used, the preceding job is “JOB_C, JOB_E”, and the succeeding job name is none because it is the last job in the job net.

FIG. 4 shows an example of the storage medium information table 172.
The storage medium information table 172 includes a medium name column 401, a medium type column 402, a server failure takeover column 403, and a medium performance (throughput) column 404. The medium performance (throughput) column 404 includes a read throughput column 405, a write throughput column 406, and a read / write throughput 407, and any value (the value of the column 405 + 406 or the value of the column 407) may be used. In the following example, read / write throughput 407 is used.

  The medium name 401 is, for example, “local HDD” or “shared file server HDD”. The medium type 402 represents the type of the file storage medium. When the type is local, it indicates that only one job execution server can access the stored file. When the type is shared, it indicates that an arbitrary job execution server can access the stored file.

  In the server failure takeover column 403, the server failure takeover permission / inhibition status indicates whether a file can be taken over to another job execution server when a failure occurs in a job execution server. In the case of local, since each job execution server holds the file, the file cannot be taken over to another job execution server, and the takeover is “impossible”. In the case of a shared file system, since the file can be accessed by another job execution server, the takeover is “permitted”.

  The medium performance in the column 404 represents the read speed and write speed (throughput) of the file storage medium. When the medium type is shared, it is the total throughput when accessed from all job execution servers.

  In the present embodiment, an example using five storage media of a local HDD, a local SSD, a local memory, a shared HDD, and a shared SSD is shown, but any number of storage media may be used. Also, other storage media other than those described above may be used.

  In the present invention, “a plurality of file storage media having different characteristics” is determined whether the characteristics are the same or different depending on whether or not the server failure takeover column 403 can be taken over at the time of server failure. . Further, media having substantially different media performance also correspond to “a plurality of file storage media having different characteristics”.

FIG. 5 shows an example of the input parameter table 173.
The input parameter table 173 includes a request processing time 511, a re-execution time threshold 512, a job execution server number 513, a CPU core number 514, and a shared write weight initial value 515.

  The requested processing time 511 represents the processing time that the batch job should complete when no failure occurs during the execution of the batch job. The re-execution time threshold 512 represents a processing time that can be accepted for re-execution when a failure occurs during the execution of a batch job. The number of job execution servers 513 represents the number of job execution servers used for executing batch jobs. The number of CPU cores 514 represents the number of CPU cores in the job execution server. The shared write weight initial value 515 represents an initial value of the shared write weight used for processing for determining the storage medium ratio described later. The shared write weight update 516 represents an update value of the shared write weight.

FIG. 6 shows an example of the job re-execution time management table 174 that is in the generation process in the optimum arrangement pattern generation unit 161.
The job re-execution time management table 174 includes a job name column 601, an item column 602, and divided job columns 603, 604, 605, 606, 607, and 608. The item column 602 includes an execution server 612, a storage medium 613, and a re-execution time 914. Lines 612, 613, and 614 indicate that the divided job 1 (divided job A1) of job A is that the execution server is the job execution server 1, the storage medium is the shared HDD, and the re-execution time is 0 minutes. A method for determining the contents of the job re-execution time management table will be described later.

FIG. 7 shows an example of the compression method information table 175.
The compression method information table 175 includes a compression method column 1101, a compression rate column 1102, a compression throughput column 1103, a decompression throughput column 1104, and a compression / decompression throughput column 1105. The compression throughput column 1105 is a throughput when the compression process and the decompression process are performed once. Here, as the values used for correcting the medium performance, the values of the compression throughput column 1103 and the decompression throughput column 1104 or the values of the compression / decompression throughput column 1105 are used. In this embodiment, the value of the compression / decompression throughput column 1105 is used.
For example, the row 1111 represents that the compression rate of the compression method 1 is 0.25, the compression throughput is 300 MB / second, the decompression throughput is 600 MB / second, and the compression / decompression throughput is 400 MB / second.

FIG. 8 shows an example of a job compression information management table 176 for managing which compression method is used by a job.
The job compression information management table 176 includes a job net name column 1201, a job name column 1202, a CPU time column 1203, an I / O amount column 1204, a local HDD column 1205, a local SSD column 1206, a local memory column 1207, and a shared HDD column 1208. , And a shared SSD column 1209. The job net name column 1201, job name column 1202, CPU time column 1203, and I / O amount column 1204 have the same contents as the job information table shown in FIG. In the local HDD column 1205, the local SSD column 1206, the local memory column 1207, the shared HDD column 1208, and the shared SSD column 1209, which compression method is used is set by the compression method determination unit 167.
For example, in the case of job A, the line 1211 indicates that the local HDD uses the compression method 2, the local SSD uses the compression method 1, the local memory uses the non-compression, the shared HDD uses the compression method 2, and the shared SSD uses the compression method 1. Yes.

FIG. 9 shows an overall processing flow of calculating the optimum file layout pattern by the optimal layout pattern generation unit 161 in the batch job execution method.
First, in step S701, the system operation manager sets job information, storage medium information, parameter information, and input parameters. These are stored in the job information table 171, the storage medium information table 172, and the parameter information table 173 by the information input unit 161. Next, in step S702, the optimum arrangement pattern generation unit 161 refers to the job re-execution time management table 174 and assigns the divided jobs evenly to the job execution servers. That is, with reference to the value of the job division number column 310 of the job information table 171 and the job execution server number 513 of the parameter information table, the job division number is divided by the job execution server number.

  For example, since the number of divisions of job A is 100 in FIG. 3 and the number of job execution servers is 5 in FIG. 5, the number of divided jobs per job execution server is 20, and divided jobs A1 to A20 are shown. The job execution server 1, the divided jobs A21 to A40 to the job execution server 2, the divided jobs A41 to A60 to the job execution server 3, the divided jobs A61 to A80 to the job execution server 4, and the divided jobs A81 to a100 Are assigned to the job execution server 5.

  Next, in step S703, the next job is selected. In step S704, the storage medium ratio determination unit 163 determines the storage ratio of each medium from the file storage medium performance. Details of the processing in step S704 will be described later. In step S705, the job storage medium determination unit 164 allocates a divided job having a long re-execution time to the shared storage based on the storage ratio determined in step S704. Details of the processing in step S705 will be described later. In step S706, the parameter information table 173 is referred to. In step S <b> 707, the optimum arrangement pattern generation unit 161 determines whether the re-execution time of the job in the job re-execution time management table 174 exceeds the re-execution time threshold of the parameter information table 173. If it is determined Yes in step S707, in step S708, the optimum arrangement pattern generation unit 161 increases the shared write weight, which is a value to be corrected in the direction of increasing the write ratio to the shared storage, and the input parameter table 173. In step S709, the optimum arrangement pattern generation unit 161 updates the re-execution time in the job having the longest re-execution time, that is, the divided job in which the re-execution time of the job is maximum. Return to the 0 minute job and repeat the process.

  For example, when it is determined that the re-execution time threshold has been exceeded in job F, the divided job having the longest re-execution time is the re-execution time of 4 minutes in the job re-execution time management table 174 shown in FIG. And the job whose re-execution time before that is 0 is job D (here, job E (not shown) is not 0).

  Next, when it is determined No in step S707, the optimal arrangement pattern generation unit 161 determines whether the job is a final job in step S710. When it determines with No in the said step S710, it returns to the said step S703 and repeats a process. If it is determined Yes in step S710, in step S711, the processing time calculation unit 165 calculates the processing time of the entire job, and determines whether the processing time of the entire job is within the requested processing time. If it is determined Yes in step S711, in step S712, the optimal arrangement pattern generation unit 161 outputs the job re-execution time management table 174 as the arrangement method, and ends the process. When it is determined No in step S711, the optimal arrangement pattern generation unit 161 outputs no arrangement method in step S713, and ends the process.

  An example of the job re-execution time management table generated in this way is shown in FIG. In the job re-execution time management table of FIG. 6 generated based on the initial value, the job F of the divided job 42 has a re-execution time of 4 minutes. In the present embodiment, a process for changing the storage ratio of the storage medium by the optimal arrangement pattern generation process and increasing the ratio of the storage medium to the shared storage for the job that causes the CPU of the job execution server to become a bottleneck. Do. For example, in the job re-execution time management table 174 of FIG. 10, the storage medium for job F is changed to the shared SSD, and the re-execution time is 0 minutes.

FIG. 11 shows an example of storage medium allocation to jobs A and B.
The re-execution time of the divided job written on the local storage medium is longer than the re-execution time of the divided job written on the shared storage medium. Therefore, in the present invention, for a divided job written on a shared storage medium (shared HDD), such as the divided job A01 of job A, the divided job B01 of the next stage job B is stored locally. For a divided job that is stored in a medium (local memory) and written to a local storage medium (local SSD) like the divided job A02 of job A, the divided job of job B at the next stage is reversed. B02 is controlled so as to be stored in a shared storage medium (shared SSD), so that the re-execution time of each divided job over a plurality of stages is made uniform. However, the storage in the shared storage medium and the local storage medium is not repeated alternately as the individual jobs progress. For example, in the divided job A22 of job A, the divided job B22 of job B at the next stage is stored in the local SSD with respect to the divided job written in the local memory. In this way, the allocation of each divided job to the storage medium is determined based on the number of storage media of the shared system that can be allocated, and is controlled to be uniform as a whole in multiple stages.

  Next, some steps in the processing flow for calculating the optimal pattern for file arrangement described in FIG. 12 will be described in more detail.

FIG. 12 shows a processing flow for determining the storage ratio in each medium from the file storage medium performance in step S704.
First, in step S801, the storage medium ratio determination unit 163 refers to the storage medium information table 172, and selects the next medium in step S802. In step S803, the storage medium ratio determination unit 163 corrects the medium performance based on the compression method. In step S804, the storage medium ratio determination unit 163 corrects the medium performance from the CPU time and I / O amount of the job.

  Details of step S803 and step S804 will be described later. The steps S803 and S804 are not essential for realizing the present invention, and may be omitted.

  In step S805, the storage medium ratio determination unit 163 determines whether the medium type of the storage medium information table 172 for the medium is shared. When it is determined Yes in step S805, the storage medium ratio determination unit 163 sets the new performance of the medium to the old medium performance × the shared write weight. When it is determined No in step S805, the storage medium ratio determination unit 163 sets the new performance of the medium as old media performance × number of job execution servers. When step S806 or step S807 is completed, it is determined in step S808 whether the medium is the final medium. When it determines with No in said S808, it returns to said step S802 and repeats a process. If it is determined Yes in step S808, the storage medium ratio determination unit 163 refers to the job information table 171 in step S809. In step S810, the storage medium ratio determination unit 163 determines whether or not the sharing essential column of the job information table 171 is output for the job. If the job is determined to be Yes in step S810, in step S811, the storage medium ratio determination unit 163 corrects the medium for which the medium type in the storage medium table 172 is shared for the job. The performance ratio of each medium is set as the storage medium ratio, and the process ends. If it is determined No in step S810, in step S812, the storage medium ratio determination unit 163 sets the corrected performance ratio of each medium as the storage medium ratio for all the media for the job, The process ends.

  For example, in the file storage medium table 172 of FIG. 4, when the read / write performance is used, the performance ratio for all media is as follows: local HDD: local SSD: local memory: shared file server HDD: shared file server SSD = 40: 200 : 400: 40: 200 = 1: 5: 10: 1: 5. The performance ratio with respect to a medium whose medium type is shared is shared file server HDD: shared file server SSD = 40: 200 = 1: 5.

  When the re-execution time threshold is exceeded, the shared write weight becomes 2, so the new medium performance of the shared file server HDD and the shared file server SSD is set to double. That is, local HDD: local SSD: local memory: shared file server HDD: shared file server SSD = 40: 200: 400: 40 × 2: 200 × 2 = 1: 5: 10: 2: 10.

  FIG. 13 shows a processing flow for assigning a shared storage medium from a divided job having a long re-execution time based on the storage ratio in step S705.

  First, in step S <b> 901, the job storage medium determination unit 164 refers to the job re-execution time management table 174 and the storage medium information table 172. In step S902, the job storage medium determination unit 164 selects the last job execution server written in the shared storage medium whose medium type is shared in the storage medium information table 172 in the previous job. In step S903, the next storage is selected, and in step S904, the number of divisions of the selected storage medium is calculated by the storage ratio × the number of divisions. For example, when the performance ratio for all media is 1: 5: 10: 1: 5, the ratio of the local HDD is 1 / (1 + 5 + 10 + 1 + 5) = 1/22.

  Next, in step S905, the next job execution server is selected. In step S906, in the divided job executed by the job execution server, the divided job with the longest re-execution time of the previous job is selected, and the storage medium is set. To do. In step S907, it is determined whether the medium type of the storage medium is shared. If it is determined Yes in step S907, the re-execution time of the selected job is set to zero. If it is determined No in step S907, the re-execution time of the selected job is set to the re-execution time of the previous job + the CPU time of the selected job / the number of divisions. When step S908 or step S909 is completed, it is determined in step S910 whether the number of divided jobs on the storage medium has been exceeded. When it determines with No in the said step S910, it returns to the said step S905 and repeats a process. If it is determined Yes in step S910, it is determined in step S911 whether or not it is the final storage medium. When it determines with No in the said step S911, it returns to the said step S903 and repeats a process. A process is complete | finished when it determines with Yes in the said step S911.

  For example, in the job re-execution time management table of FIG. 6, the job execution server 1 is selected for job A, the shared HDD is assigned to the divided job 1 having the longest re-execution time, and the re-execution time is set to 0 minutes. Next, the job execution server 2 is selected, and the above processing is repeated for the divided job 21 having the longest execution time. In the divided job 22 of job B, since the CPU time of job B is 3 minutes and the number of divided jobs is 100, the re-execution time is set to 10/100 minutes + 3/100 minutes = 13/100 minutes.

  In the present embodiment, the storage medium ratio is changed using the shared write weight, but it may be changed by any other method. For example, the number of divided jobs assigned to the shared HDD and shared SSD may be increased by adding a dummy divided job that does not perform any processing.

  In the present invention, the storage medium of the input file and output file of each divided job is determined by the above procedure, and the batch job is executed. By calculating a file arrangement pattern that satisfies the request processing time when a failure occurs in addition to the normal time, the request processing time can be satisfied even when a job occurs and the job is re-executed.

  Next, a method of correcting the medium performance from the CPU time and I / O amount of the job and changing the storage medium ratio will be described. In step S804 in FIG. 12, this is realized by further adding a process for correcting the medium performance from the CPU time and I / O amount of the job.

  FIG. 14 shows a processing flow for correcting the medium performance from the CPU time and I / O amount of the job in step S804.

  First, in step S1001, the storage medium ratio determination unit 163 determines whether or not I / O amount ÷ CPU time is less than the CPU-I / O ratio correction threshold. If it is determined Yes in step S1001, the storage medium ratio determination unit 163 determines in step S1002 whether the medium type of the storage medium is shared. If it is determined Yes in step S1002, in step S1003, the storage medium ratio determination unit 163 sets the new performance of the medium to the old medium performance × CPU-I / O ratio correction weight, and ends the process. When it is determined No in step S1001 or when it is determined No in step S1002, in step S1004, the storage medium ratio determination unit 163 sets the new performance of the medium to the old medium performance and ends the process. To do.

  In this embodiment, the CPU-I / O ratio correction threshold is set to 5, and the CPU-I / O ratio correction weight is set to 3. For example, in the job information table shown in FIG. 3, since the job A has a CPU time of 10 minutes and an I / O amount of 30 GB, I / O amount ÷ CPU time = 30 GB ÷ 10 minutes = 3, and the step S1002 Is determined as Yes. Therefore, the new performance of the shared HDD and shared SSD media is tripled, and the shared HDD is 40 × 3 = 120 MB / sec, and the shared SSD is 200 × 3 = 600 MB / sec.

  In the present invention, the CPU time and the I / O amount of the job are used to correct the storage medium performance according to the above procedure, and the CPU-I / O ratio is low, that is, the job execution is performed by changing the storage medium ratio. For jobs where the CPU of the server becomes a bottleneck, the ratio of the storage medium to the shared storage is increased and the CPU-I / O ratio is high, that is, the I / O of the job execution server becomes the bottleneck. For such jobs, the proportion of the storage medium in the shared storage can be reduced.

  Next, a method for correcting the medium performance based on the compression method and changing the storage medium ratio will be described. In step S803 in FIG. 12, this is realized by further adding processing for correcting the medium performance based on the compression method.

FIG. 15 shows a processing flow for correcting the medium performance based on the compression method of step S803.
First, in step S1301, the information system administrator inputs the contents of the compression information table 175. In step S1302, the compression method determination unit 167 refers to the compression information table 175 and the job compression information management table 176. In step S1303, the compression method determination unit 167 determines whether or not I / O amount / CPU time is less than the compression threshold value 1.

  If it is determined Yes in step S1304, all storage media are set to uncompressed. When it is determined No in step S1303, in step S1305, the compression method determination unit 167 determines whether I / O amount ÷ CPU time is less than the compression threshold value 2. If it is determined Yes in step S1305, in step S1306, the HDD is set to the compression method 1, and the SSD and the memory are set to uncompressed. If it is determined No in step S1305, in step S1307, the compression method determination unit 167 determines whether I / O amount ÷ CPU time is less than the compression threshold 3. If it is determined Yes in step S1307, in step S1308, the HDD is set to compression method 2, the SSD is set to compression method 1, and the memory is set to non-compression. If it is determined No in step S1307, the HDD and SSD are set to the compression method 2 and the memory is set to the compression method 1 in step S1309.

  When step S1304, step S1306, step S1308, or step S1309 is completed, in step S1310, the compression method determination unit 167 determines whether the medium is compressed in the selected job. (Whether it is other than uncompressed) is determined. When it is determined Yes in step S1310, in step S1311, the new performance of the medium is set to the medium performance / compression ratio, and the process ends. If it is determined No in step S1310, in step S1312, the compression method determination unit 167 sets the new performance of the medium to the old performance of the medium, and ends the process.

  In this embodiment, the compression threshold 1 is set to 1, the compression threshold 2 is set to 3, and the compression threshold 3 is set to 10. For example, in the job compression information management table shown in FIG. 14, since the job A has a CPU time of 10 minutes and an I / O amount of 30 GB, I / O amount ÷ CPU time = 30 GB ÷ 10 minutes = 3. In S1007, it is determined as Yes. Therefore, Step S1308 is executed, the local HDD column 1205 and the shared HDD column of the job compression information management table are set to the compression method 2, the local SSD column 1203 and the shared SSD column 1209 are set to the compression method 1, and the local The memory column 1207 is set to uncompressed.

  In this embodiment, the compression method is determined in four stages, but it may be an arbitrary stage. Further, the compression method may be determined by distinguishing between local storage and shared storage. For example, in step S1303, the local storage may be determined by I / O amount / CPU, and the shared storage may be determined by (I / O amount / CPU) × number of job execution servers.

As described above, in the present embodiment, the optimum file arrangement satisfying a predetermined completion time by using each local disk and shared file server of the job execution server as a file storage medium for each divided job constituting the job net. A pattern is generated, and a storage medium is allocated so that the I / O timing of the divided jobs to the shared storage is not concentrated. Based on the optimal file arrangement pattern generated in this way, a batch job is executed. Thereby, when a server failure occurs, the batch job execution result of the server can be accessed, so that the job re-execution time is shortened. Also, each batch job is executed using both the local disk and the shared file server, and the I / O timing to the shared storage of the divided jobs is not concentrated, so access from multiple job execution servers is shared A reduction in file access speed concentrated on the file server is also suppressed, and a predetermined completion time can be satisfied. Thus, according to this embodiment, in the execution of a batch job, the re-execution time at the time of failure is reduced. The I / O timing of the divided jobs to the shared storage can be prevented from being concentrated within the range to be satisfied.

  In the first embodiment, the storage medium for the divided job is determined and the process is terminated. However, there is a case where the processing does not end in the calculated processing time due to re-execution at the time of failure or processing time fluctuation due to data amount fluctuation. Therefore, this embodiment shows a method of monitoring the progress of job execution, correcting the storage medium ratio based on the monitoring result, and changing the storage medium of the divided jobs.

  A second embodiment of the present invention will be described with reference to FIGS. FIG. 16 is a job re-execution time management table according to the second embodiment of the present invention. FIG. 17 is an overall process flowchart in the second embodiment.

  In the second embodiment, the system configuration diagram shown in FIG. 1 includes a progress information monitoring unit 169 that monitors the progress information of the divided job and sets the end time of the divided job in the job re-execution time management table 174.

  FIG. 16 is a job re-execution time management table 174 corresponding to FIG. 6 of the first embodiment. The job execution server 1411, the storage medium 1412, and the re-execution time 1413 correspond to the job execution server 612, the storage medium 613, and the re-execution time 614 shown in FIG. In the present embodiment, it further has an end time 1414 for setting the end time of the job monitored by the progress monitoring unit 169.

FIG. 17 shows a processing flow for calculating an optimum pattern for file arrangement in the batch job execution method corresponding to FIG. 9 of the first embodiment.
First, in step S1502, the optimum arrangement pattern generation unit 161 executes step S701 and step S702 in FIG. In step S1503, the progress monitoring unit 169 monitors the progress of the divided job, and sets the job end time to the end time shown in FIG. In step S1504, the progress monitoring unit 169 determines whether all divided jobs have been completed. When it determines with No in the said step S1503, it returns to the said step S1503 and repeats a process. If it is determined Yes in step S1503, in step S1505, the progress monitoring unit 169 calculates the difference between the end time of the divided job and the end time of the first divided job, and the shared storage and local Calculate the average storage delay time.

  In step S1506, the progress monitoring unit 169 determines whether the average delay time of the shared storage is larger than the average delay time of the local storage. If it is determined Yes in step S1506, in step S1507, the progress monitoring unit 169 sets the new performance of the medium to old medium performance × progress monitoring weight 1 for the storage medium in which the medium type = shared. . If it is determined No in step S1506, in step S1508, the progress monitoring unit 169 sets the new medium performance to old medium performance × progress monitoring weight 2 for the storage medium in which the medium type = shared. . When step S1507 or step S1508 is completed, in step S1509, the progress monitoring unit 169 selects a job that has not started all the divided jobs.

  In step S1510, the optimal arrangement pattern generation unit 161 executes steps S703 to S714 in FIG. In step S1511, the progress monitoring unit 169 determines whether all jobs have been completed. When it determines with No in the said step S1511, it returns to the said step S1503 and repeats a process. If it is determined Yes in step S1511, the process ends.

  In this embodiment, the progress monitoring weight 1 is set to 0.8 and the progress monitoring weight 2 is set to 1.5. For example, when all the divided jobs of job A in FIG. 16 are completed, the first divided job is 10:00 of divided job 1, and the differences are 2 minutes for divided job 2 and 3 minutes for divided job 21, respectively. The divided job 22 is 2 minutes, the divided job 41 is 5 minutes, and the divided job 41 is 2 minutes. The average delay time of the shared storage is (0 minutes + 3 minutes) ÷ 2 = 1.5 minutes in order to calculate the average of the divided job 1 and the divided job 21. Similarly, the average delay time of the local storage is (2 minutes + 2 minutes + 5 minutes + 2 minutes) ÷ 4 = in order to calculate the average of divided job 2, divided job 22, divided job 41, and divided job 42. 2.75 minutes. Therefore, in step S1506, it is determined No, the new performance of the shared HDD and shared SSD medium is increased by 1.5 times, the shared HDD is 40 × 1.5 = 60 MB / sec, and the shared SSD is 200 × 1. 5 = 300 MB / sec. Thereby, the storage medium ratio is also changed. In step S1509, since job B has already been started, it is not selected and job C (job D if job C is also partly divided) is selected.

  As described above, according to this embodiment, in the execution of a batch job, the progress of job execution is monitored, the storage medium ratio is corrected based on the monitoring result, and the storage medium of the divided job is changed. As long as the re-execution time is satisfied, the I / O timing of the divided jobs to the shared storage can be prevented from being concentrated.

Next, a third embodiment of the present invention will be described with reference to FIG.
In the first and second embodiments, the shared file server 130 and the storage system 140 realize file sharing between the job execution server 110 and other job execution servers. Unlike the first or second embodiment, the present embodiment shows a method for determining a storage medium for divided jobs when the local storage in the job execution server 110 is shared. For example, a file can be shared using the local HDD of Non-Patent Document 1. Another feature is that other computers have a copy of the data so that the data is not lost in the event of a failure.

  FIG. 18 is a block diagram illustrating a hardware configuration of a computer system that executes a batch job, that is, a batch job processing system according to the third embodiment. FIG. 18 corresponds to FIG. 1 in the first embodiment. The job execution server 110 corresponds to the job execution server 110 illustrated in FIG. 1, and further includes a file server control unit 1601 corresponding to the file server 135 illustrated in FIG. 1. The batch job execution server 150 corresponds to 150 shown in FIG. 1, and the storage medium information table 1672 corresponds to the storage medium information table 172 shown in FIG.

FIG. 19 is an example of the storage medium information table 1672.
Columns 401 to 407 correspond to columns 401 to 407 shown in FIG. A row 1711 indicates that the value of the medium type column 402 of the local HDD is local, and a row 1712 indicates that the value of the medium type column 402 of the local HDD is shared. Both are local HDDs, but in the case of sharing, data is often copied to other job execution servers, so the media performance often decreases. As described above, even the same storage medium may be shared or may not be shared.

  The storage medium ratio is determined based on the storage medium information table 1672 shown in FIG. The determination method is the same as in the first embodiment.

  Also in this embodiment, in the execution of a batch job, the I / O timing to the shared storage of the divided jobs can be prevented from being concentrated within a range that satisfies the re-execution time at the time of failure.

DESCRIPTION OF SYMBOLS 101 ... Network, 102 ... Storage network, 110 ... Job execution server, 111 ... CPU, 112 ... Memory, 113 ... Communication interface, 114 ... Disk controller, 115 ... Local hard disk (HDD), 116 ... Local solid state disk (SSD) DESCRIPTION OF SYMBOLS 117 ... Local HDD file, 118 ... Local SSD file, 119 ... Local memory file, 120 ... Job program, 121 ... Job execution control part, 122 ... Compression / decompression execution part, 123 ... File access control part, 130 ... Shared file server 131 ... CPU, 132 ... memory, 133 ... communication interface, 134 ... storage interface, 135 ... file server control unit, 140 ... storage system, 141 ... store Storage system SSD, 143 ... Storage system HDD file, 144 ... Storage system SSD file, 150 ... Batch job management server, 151 ... CPU, 152 ... Memory, 153 ... Communication interface, 154 ... Disk controller, 155 ... Local disk 161 ... Optimal arrangement pattern determining unit 162 ... Information input unit 163 ... Storage medium ratio determining unit 164 ... Job storage medium determining unit 165 ... Processing time calculating unit 166 ... Re-execution time calculating unit 167 ... Compression method determination unit, 168 ... job execution control unit, 169 ... progress information monitoring unit, 171 ... job information table, 172 ... file storage medium table, 173 ... parameter table, 174 ... job re-execution time management table, 175 ... compression information Buru, 176 ... job compression method management table.

Claims (17)

A method for executing a batch job by a batch job processing system,
The batch job processing system includes:
At least one batch job execution server is connected to the batch job management server via a network, and includes a plurality of file storage media having different characteristics, and the file storage media include one or more local storage media And one or more shared storage media,
A job net that is a group of the series of jobs, job information that represents information of a divided job that is divided into jobs to be executed in parallel, storage medium information that represents information of the file storage medium, and job execution A step of accepting setting of parameter information representing a condition;
Determining a storage ratio to each file storage medium from storage medium performance of the storage medium information;
Assigning a split job having a long re-execution time to the shared storage medium based on the storage ratio;
Determining whether the re-execution time of each of the divided jobs exceeds a re-execution time threshold of the parameter information;
When it is determined that the re-execution time exceeds the re-execution time threshold, returning to the re-execution start job of the divided job having the longest re-execution time;
A batch job execution method comprising: The batch job execution method according to claim 1, wherein
Referring to a job re-execution time management table generated to manage the divided jobs, and equally assigning the divided jobs to the batch job execution server;
When it is determined that the re-execution time exceeds the re-execution time threshold, the write correction to the shared storage medium is corrected to increase before the step of returning to the re-execution start job of the divided job. Increasing a shared storage medium write weight that is a value to be
A batch job execution method further comprising: The batch job execution method according to claim 2,
The step of determining the storage ratio to each file storage medium from the storage medium performance,
Referring to the storage medium information table;
Determining whether the value of the medium type column of the storage medium information table is shared;
When it is determined that the value of the medium type column is shared, the new performance of the file storage medium is set to old media performance × shared write weight;
When it is not determined that the value of the medium type column is shared, setting the new performance of the file storage medium to old media performance × number of job execution servers;
A step of referring to the job information table;
Determining whether the value of the share mandatory column of the job information table is output;
Determining a corrected performance ratio of each file storage medium as a storage medium ratio for a file storage medium whose medium type is shared when it is determined that the value of the share mandatory column is output; ,
Determining the performance ratio of each corrected file storage medium as the storage medium ratio for all file storage media when it is not determined that the value of the share essential column is output;
A batch job execution method comprising: The batch job execution method according to claim 2,
The step of allocating to the shared storage medium from a divided job having a long re-execution time based on the storage ratio includes:
Referring to the job re-execution time management table and the storage medium information table;
Selecting the last job execution server written in the shared storage medium that is shared by the previous job in the job re-execution time management table;
Selecting the next file storage medium;
Based on the storage medium ratio determined by the step of determining the storage ratio to each file storage medium from the storage medium performance, calculating the division number of the selected storage medium ratio by the storage medium ratio x job division number;
Selecting the next job execution server;
Selecting a divided job having the longest re-execution time of the previous job in the divided job executed by the selected job execution server, and setting the selected file storage medium;
Determining whether the medium type of the selected file storage medium is shared;
When it is determined that the medium type is shared, the step of setting the re-execution time of the selected divided job to 0;
When the medium type is not determined to be shared, the step of setting the re-execution time of the selected divided job to the re-execution time of the previous job + the CPU time of the selected job / the number of divisions;
Determining whether the number of divided jobs of the selected file storage medium has been exceeded;
A batch job execution method comprising: The batch job execution method according to claim 3,
Correcting the storage medium performance of the storage medium information table from the CPU time and I / O amount of the job information table;
A batch job execution method further comprising: The batch job execution method according to claim 5,
The step of correcting the storage medium performance from the CPU time of the job and the I / O amount determines whether or not the value of the I / O amount of the job ÷ the CPU time of the job corrects the storage medium performance. Determining whether it is less than a CPU-I / O ratio correction threshold that is a threshold to
Determining whether the medium type of the storage medium is shared;
When it is determined that the medium type is shared, the CPU-I / O ratio correction weight representing the weight for correcting the storage medium performance is used to set the new performance of the file storage medium to the old medium performance × CPU−. Setting I / O ratio correction weights;
When it is not determined that the CPU-I / O ratio correction threshold is less than the threshold, or when the medium type is not determined to be shared, the new performance of the file storage medium is set to the old media performance;
A batch job execution method comprising: The batch job execution method according to claim 3,
Correcting the storage medium performance based on a compression method;
A batch job execution method further comprising: The batch job execution method according to claim 7,
The step of correcting the performance of the storage medium based on the compression method includes:
Receiving a setting of a compression information table representing compression method information;
Referring to the compression information table and a job compression information management table generated to manage the compression method used by each job;
Determining whether or not the value of the I / O amount of the job / the CPU time of the job is less than a compression threshold 1 in the CPU time and the I / O amount of the job information table;
When it is determined that the compression threshold is less than 1, all the file storage media are set to uncompressed;
If it is not determined that the compression threshold is less than 1, some of the file storage media are set to compression;
Determining whether or not to compress and store in the file storage medium in the selected job;
If it is determined to compress, the new performance of the file storage medium is set to media performance ÷ compression rate; and
If it is not determined to compress, setting the new performance of the file storage medium to the old media performance;
A batch job execution method comprising: The batch job execution method according to claim 2,
Monitoring the progress of the divided jobs and setting a job end time in a job re-execution time management table;
Determining whether all the divided jobs have been completed;
Calculating a difference between an end time of the divided job and an end time of the first divided job, and calculating an average delay time of the shared storage medium and the local storage medium;
Determining whether the average delay time of the shared storage medium exceeds the average delay time of the local storage medium;
When it is determined that the average delay time of the shared storage medium exceeds the average delay time of the local storage medium, the medium type is shared by using a progress monitoring weight representing a weight for correcting the storage medium performance. For a medium, setting the new performance of the medium to old medium performance × progress monitoring weight 1;
When it is not determined that the average delay time of the shared storage medium exceeds the average delay time of the local storage medium, the new performance of the medium is set to the old medium performance × progress monitoring weight for the medium whose medium type is shared. A step of setting to 2,
Selecting a job in which all the divided jobs are not started; and
A batch job execution method further comprising: And at least one batch job execution server connected to the batch job management server via a network, and a plurality of file storage media having different characteristics, the file storage media including one or more local storage media and one A batch job processing system including the above shared storage medium,
The batch job management server
A job net that is a group of a series of jobs, job information that represents information on a divided job that is divided into jobs to be executed in parallel, storage medium information that represents information on the file storage medium, and conditions for job execution An information input unit for accepting setting of parameter information representing
A storage medium ratio determination unit for determining a storage ratio of each storage medium information from the storage medium performance to each file storage medium;
A job storage medium determination unit that allocates to the shared storage medium from a divided job having a long re-execution time based on the storage ratio;
A re-execution time calculation unit for calculating the re-execution time of the divided job;
It is determined whether or not the re-execution time exceeds the re-execution time threshold of the parameter information, and when it is determined that the re-execution time threshold is exceeded, re-execution of the divided job having the longest re-execution time An optimal placement pattern generator that returns to the start job;
A batch job processing system comprising: The batch job processing system according to claim 10,
A compression method determination unit that determines a compression method used by the job;
A storage medium ratio determination unit for correcting the storage medium ratio based on the determined compression method;
A batch job processing system comprising: The batch job processing system according to claim 10,
A progress information monitoring unit that monitors the progress of the divided jobs and sets a job end time in a job re-execution time management table;
A storage medium ratio determination unit that corrects the storage medium ratio based on the job end time;
A batch job processing system comprising: The batch job processing system according to claim 10,
The batch job processing system includes:
The one or more batch job execution servers, and a shared file server for processing a file access request from the batch job execution server,
The batch job execution server includes at least one of a local disk, a local solid state disk, and a memory as the local storage medium.
The shared file server includes at least one of a hard disk and a solid state disk as the shared storage medium. The batch job processing system according to claim 10,
The batch job processing system includes:
One or more batch job execution servers including the file storage medium are connected to the batch job management server via the network;
The batch job execution server includes one or more local storage media and one or more shared storage media as the file storage media. A batch job management server that manages the execution of batch jobs in a batch job processing system,
The batch job processing system includes at least one batch job execution server connected to the batch job management server via a network, and a plurality of file storage media having different characteristics. In what includes the above local storage media and one or more shared storage media,
The batch job management server
A job net that is a group of a series of jobs, job information that represents information on a divided job that is divided into jobs to be executed in parallel, storage medium information that represents information on the file storage medium, and conditions for job execution An information input unit for accepting setting of parameter information representing
A storage medium ratio determining unit that determines a storage ratio in each storage medium from the storage medium performance of the storage medium information;
A job storage medium determination unit that allocates to the shared storage medium from a divided job having a long re-execution time based on the storage ratio;
A re-execution time calculation unit for calculating the re-execution time of the divided job;
It is determined whether or not the re-execution time threshold of the parameter information has been exceeded, and when it is determined that the re-execution time threshold has been exceeded, the optimal return to the re-execution job of the split job having the longest re-execution time An arrangement pattern generation unit;
A batch job management server comprising: The batch job management server according to claim 15,
A compression method determination unit that determines a compression method used by the job;
A storage medium ratio determination unit for correcting the storage medium ratio based on the determined compression method;
A batch job management server comprising: The batch job management server according to claim 15,
A progress information monitoring unit that monitors the progress of the divided jobs and sets a job end time in a job re-execution time management table;
A storage medium ratio determination unit that corrects the storage medium ratio based on the job end time;
A batch job management server comprising:

Images