JP7471091B2 - JOB EXECUTION SUPPORT SYSTEM AND JOB EXECUTION SUPPORT METHOD - Google Patents

Description

The present invention relates to a job execution support system and a job execution support method.

Public cloud services that provide cloud computing environments are widely available. One of the features of public cloud services is that the fee is set according to the specifications of the machine (virtual machine) used and the duration of service use. Therefore, it is preferable that the configuration of cloud batch jobs set by the user is optimized to reduce processing time, etc., in order to keep costs down (for example, by achieving short batch processing times on machines with lower specifications).

In this regard, as a method for improving the efficiency of job operations, Patent Document 1 discloses a technology for improving the operating rate of an output means compared to the conventional technology when executing a job that includes a pre-processing for generating information and a post-processing for outputting the information generated by the pre-processing to an output means. In addition, Patent Document 2 describes a technology for optimizing the use of cloud resources to prevent excessive costs.

JP 2012-35587 A JP 2014-501994 A

However, Patent Document 1 aims to improve the operation rate of jobs, not to shorten the processing time of the jobs themselves. Furthermore, Patent Document 2 has a configuration for reducing the load on resources by adjusting the timing of job launch, but does not have the primary objective of improving the efficiency of processing time. In this way, it can be said that optimization in cloud computing has not been sufficiently considered.

The present invention was made in light of this current situation, and its purpose is to provide a job execution support system and a job execution support method that are capable of optimizing job execution.

To solve the above problem, one aspect of the present invention includes a storage device that stores a database having predetermined data items, an intermediate file dependency analysis process that identifies data output by or input to each job in a process target program that executes a plurality of jobs that execute one or a plurality of processing instructions, thereby identifying processing dependencies between each of the jobs based on the data, and grouping each of the jobs in the process target program based on the identified dependencies to create job groups that can be executed in parallel with each other , an access destination dependency analysis process that identifies the data items in the database that are the access destination of each job, and whether the access form to the data related to the data items is registration, update, deletion, or reference, and an access destination to the database of jobs belonging to each of the job groups that are executed in parallel with each other based on the identified access forms. As a result, when a combination of a first job group and a second job group executed in parallel with each other is identified in which the databases to be accessed are the same but the data items to be accessed by registering, updating or deleting data in the database do not overlap, an access destination dependency analysis process divides the database into a first database including data items accessed by only the first job group by registering, updating or deleting data, and a second database including data items accessed by only the second job group by registering, updating or deleting data, and a job execution process that executes the job groups that can be executed in parallel with each other in parallel based on the program to be processed.

Another aspect of the present invention is an information processing device including a storage device storing a database having predetermined data items , the information processing device including a processing target program executing a plurality of jobs each executing one or a plurality of processing instructions, and an intermediate file dependency analysis process for identifying data output by each job in the processing target program or data input to each job, thereby identifying processing dependencies between each of the jobs based on the data, and grouping each job in the processing target program based on the identified dependencies to create job groups that can be executed in parallel with each other; an access destination dependency analysis process for identifying a data item in the database that is the access destination of each job, and whether the access form to the data related to the data item is registration, update, deletion, or reference; and an access destination dependency analysis process for identifying a data item in the database that is the access destination of each job, and whether the access form to the data related to the data item is registration, update, deletion, or reference. and when a combination of a first job group and a second job group executed in parallel with each other is identified in which the databases to be accessed are the same but the data items to be accessed by registering, updating or deleting data in the databases do not overlap, the method performs an access destination dependency analysis process to divide the databases into a first database including data items accessed only by the first job group by registering, updating or deleting data, and a second database including data items accessed only by the second job group by registering, updating or deleting data, and a job execution process to execute the job groups that can be executed in parallel with each other in parallel based on the program to be processed.

The present invention allows for the optimization of job execution.

Problems, configurations, and advantages other than those described above will become clear from the description of the embodiments below.

FIG. 1 is a diagram illustrating an example of a configuration of a job execution support system according to an embodiment of the present invention. FIG. 10 is a diagram illustrating an example of a configuration of an optimization program. 11 is a flowchart illustrating an example of a job execution support process. 13 is a flowchart illustrating details of an analysis process and a job parallelization process. FIG. 13 is a diagram illustrating an example of an intermediate file analysis table. 11A and 11B are diagrams illustrating an example of a structure analysis table and an execution order and processing unit of each job. FIG. 13 is a diagram showing an example of an access destination analysis table which is data created by the access destination dependency analysis process. FIG. 4 is a diagram illustrating an example of a job program. 13 is a flowchart illustrating details of a job group creation process. 10 is a flowchart illustrating details of a subsequent job specifying process. 11A and 11B are diagrams showing an example of a job group in the batch processing program 120 specified by the job group creation process, and an example of a corresponding structure analysis table. 11 is a flowchart illustrating details of a program reconfiguration process. 11 is a flowchart illustrating an example of an operation test process. 11 is a flowchart illustrating an example of an initial learning process. 13 is a flowchart illustrating details of an information presentation process. FIG. 13 illustrates an example of a batch processing operation status screen. 13 is a flowchart illustrating details of a business operation process and a relearning process.

An example of an embodiment of the present invention is described below with reference to the drawings.

Figure 1 is a diagram illustrating an example of the configuration of a job execution support system according to this embodiment.

The job execution support system 1 includes a cloud system 10 that stores one or more service programs 104, and a user system 50 that is used by a user who executes the service programs 104.

The user system 50 is, for example, an information processing system installed in a company's business premises or a data center, and performs a specified task using the service program 104. Note that a program equivalent to the service program 104 of the cloud system 10 is also installed in the user system 50 as a local environment (on-premise environment).

The cloud system 10 includes an AP server 100, a DB server 200, and a Web server 300. The cloud system 10 is, for example, an information processing system managed by a specific service provider.

Of these, the AP server 100 includes a memory 101 such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a CPU 102, a storage 103 such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a communication device 108 that communicates with the DB server 200 and the Web server 300. Although not shown, the AP server 100 also includes input devices such as a keyboard and a touch panel, and output devices such as a display and a touch panel.

The AP server 100 stores a service program 104, a batch processing program 120, a management program 125, and an optimization program 130 in the storage 103.

The service program 104 is a group of programs provided by the cloud system 10 to the user system 50, and is composed of programs (processing instructions) necessary for the user to perform various tasks. The service program 104 is executed on a virtual machine (not shown) created by the AP server 100. The cloud system 10 executes the service program 104 on a virtual machine having specifications specified by the user system 50 from a list that defines the virtual machine specifications (machine performance).

The batch processing program 120 (the program to be processed) is provided to execute each service program 104 at a timing specified by the user (e.g., monthly). Specifically, the batch processing program 120 has multiple processing units (hereinafter referred to as jobs) that include one or more processing commands (processing commands for the service programs 104).

The batch processing program 120 may be, for example, a specific program that was executed in an on-premise environment in the user system 50 and uploaded to the cloud system 10, or may be a program that was originally stored in the cloud system 10.

Each job of the batch processing program 120 is composed of an execution section which executes one or more service programs 104, and an SQL section which accesses one or more tables (not shown) stored in the AP server 100 or the DB server 200. Note that a table here is a data group such as a database, and in this embodiment, is a database consisting of one or more records having one or more data items (columns).

The batch processing program 120 is also capable of executing each job in parallel over time (parallel processing is possible). Specifically, the batch processing program 120 is executed based on a job definition, which is information that defines each job and the timing of their execution.

The format of the batch processing program 120 is not particularly limited, and may be, for example, a script, which is a text file created in a specific language or according to rules, or an executable program.

When the specified settlement time arrives, the cloud system 10 bills the user system 50 for a specified usage fee calculated based on the execution time (processing time) of each program (batch processing program 120, service program 104) executed in the cloud system 10 and the specifications of the configured virtual machine, etc.

Next, the management program 125 includes a construction unit that constructs a virtual machine that executes the batch processing program 120, and a job execution unit that executes the batch processing program 120 on the virtual machine.

The optimization program 130 is a group of programs for optimizing the execution of the service program 104 by the batch processing program 120. That is, the optimization program 130 optimizes the batch processing program 120 so as to reduce the usage fee for the cloud system 10 or to shorten the processing time in the cloud system 10. This makes it possible to optimize the processing, for example, when a program that performed batch processing in the user system 50 is migrated to the cloud system 10, which is a public client. Details of the optimization program 130 will be described later.

Next, the DB server 200 includes a memory 206 such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a CPU 207, a storage 201 such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a communication device 205 that communicates with the AP server 100. Although not shown, the DB server 200 may also include an input device such as a keyboard or a touch panel, and an output device such as a display or a touch panel.

The DB server 200 stores in the storage 201 a batch processing area 202 that stores data necessary for executing the batch processing program 120 and the above-mentioned tables, an analysis result storage area 203 that stores data output by the optimization program 130 (e.g., data on the analysis results of the batch processing program 120 by the optimization program 130), and a prediction model storage area 204 that stores a processing time prediction model described below.

The Web server 300 includes a memory 303 such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a CPU 304, a storage 301 such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a communication device 305 that communicates with the AP server 100.

The Web server 300 stores in the storage 301 data (quote presentation screen data 302) of a display screen showing the analysis results of the batch processing program 120 by the optimization program 130. Although not shown, the Web server 300 may also include an input device such as a keyboard or a touch panel, and an output device such as a display or a touch panel.

The AP server 100, the DB server 200, and the Web server 300 are communicatively connected to each other via a wired or wireless communication network 5 such as a local area network (LAN), a wide area network (WAN), the Internet, a dedicated line, etc. The user system 50 and the cloud system 10 are also communicatively connected to each other via a similar communication network 6.

Next, we will explain the details of the optimization program 130.

<Optimization program>
2 is a diagram illustrating an example of the configuration of the optimization program 130. The optimization program 130 includes an analysis program 150, a conversion program 160, a prediction model learning program 170, and an information presentation program 180.

The analysis program 150 analyzes the batch processing program 120. Specifically, the analysis program 150 includes an intermediate file dependency analysis unit 151, a table dependency analysis unit 152, and a structure analysis unit 153.

The intermediate file dependency analysis unit 151 identifies data output by each job in a target program (batch processing program 120) that executes multiple jobs that execute one or more processing instructions, or data input to each of the jobs (hereinafter, files containing such data will be referred to as intermediate files), and thereby identifies processing dependencies between each job based on that data, and identifies jobs in the target program that will be executed in parallel with each other, based on the identified dependencies.

In this embodiment, at least one of the jobs accesses some (e.g., some columns) or all of the data in a specific data group (table).

In addition, the job execution unit of the management program 125 executes the jobs identified by the intermediate file dependency analysis unit 151 in parallel.

The table dependency analysis unit 152 analyzes the access destinations of the data groups (tables) of each job identified by the intermediate file dependency analysis unit 151 and executed in parallel with each other, and when it identifies a combination of a first job and a second job that are executed in parallel with each other and that access the same data group but the data portions of the accessed data group do not overlap, it divides the data group into a first data group that includes a data portion accessed only by the first job, and a second data group that includes a data portion accessed only by the second job.

The structural analysis unit 153 identifies the execution unit and execution order of each job in the batch processing program 120.

Next, the conversion program 160 modifies the batch processing program 120 based on the results of the table dependency analysis unit 152. Specifically, the conversion program 160 includes a program reconfiguration unit 161.

The program reconfiguration unit 161 modifies the program to be processed (batch processing program 120) so that the first job identified by the table dependency analysis unit 152 accesses a first data group, and the second job identified by the table dependency analysis unit 152 accesses a second data group.

The prediction model learning program 170 uses a predetermined trained model to predict the future processing time and usage fee of the batch processing program 120. Specifically, the prediction model learning program 170 includes a prediction model creation unit 171, a prediction execution unit 172, and a learning unit 173.

The prediction model creation unit 171 acquires each processing time of the program to be processed (batch processing program 120) by executing in parallel jobs that can be executed in parallel, identified by the management program 125, under multiple execution environments, and creates information (processing time prediction model) that stores the relationship between the processing time of the batch processing program 120 and the execution environment of the batch processing program 120 based on each acquired processing time.

The processing time prediction model is information in the form of, for example, a program, a conditional expression, a database, etc., and the format is not particularly limited.

In this embodiment, the execution environment refers to, for example, the execution timing of the batch processing program 120 (specified by year, month, day, etc.), the specifications of the virtual machine, the content or amount of data (transaction data, etc.) input to the batch processing program 120, etc.

The prediction execution unit 172 inputs the execution environment specified by the user or the like into the above-mentioned information (processing time prediction model), calculates the processing time of the program to be processed (batch processing program 120) in that execution environment, and calculates an execution cost having a predetermined relationship with the calculated processing time.

The learning unit 173 updates the processing time prediction model created by the prediction model creation unit 171 by inputting new training data into the processing time prediction model.

The information presentation program 180 displays the estimation results based on the processing time prediction model created by the prediction model creation unit 171 on a specified screen. Specifically, the information presentation program 180 includes an estimate presentation unit 181.

The estimate presentation unit 181 displays the cost information calculated by the prediction execution unit 172.

In addition, the estimate presentation unit 181 displays information about the execution environment specified by the prediction execution unit 172 along with information about the cost calculated by the prediction execution unit 172.

The above programs are read and executed by the CPU 102 of the AP server 100 from the memory 101 or storage 103. Each program may be recorded on a recording medium that is readable by the AP server 100.

Next, we will explain the processing performed by the job execution support system 1. The job execution support system 1 executes a job execution support process, which is a process that analyzes the batch processing program 120, rewrites it as necessary, creates a processing time prediction model, and predicts the processing time and usage fee of the batch processing program 120 on the cloud system 10.

<Job Execution Support Processing>
3 is a flowchart illustrating an example of a job execution support process. The job execution support process is started, for example, at a predetermined timing (for example, at a predetermined time interval or at a predetermined time) after the AP server 100 is started, or when the AP server 100 receives a predetermined input from the user system 50.

First, the analysis program 150 of the AP server 100 executes an analysis process of the batch processing program 120 (step 201). Then, the conversion program 160 of the AP server 100 executes a job parallelization process to modify the batch processing program 120 based on the results of the analysis process so that jobs that can be executed in parallel among the jobs in the batch processing program 120 are processed in parallel (step 202).

Details about the analysis process and job parallelization process will be described later.

The management program 125 of the AP server 100 executes an operation test process to execute the batch processing program 120 modified by the job parallelization process (step 203). This determines the specifications (performance) of the virtual machine. The details of the operation test process will be described later.

The prediction model learning program 170 of the AP server 100 executes an initial learning process to create a new processing time prediction model based on the results of the operation test process (step 204). The details of the initial learning process will be described later.

The information presentation program 180 of the AP server 100 predicts the future processing time of the batch processing program 120 based on the processing time prediction model created by the initial learning process or the re-learning process described below, and also executes information presentation processing to predict the usage fees for the batch processing program 120 and the service program 104 (step 205). At this time, the Web server 300 or the user system 50 displays the results of the information presentation processing on a specified screen.

After executing the initial learning process, the AP server 100 also executes a business operation process that executes the batch processing program 120 at any time, at a timing set according to the user's business situation (e.g., a specified time interval, a specified time, or a timing specified by user input) (step 206).

The prediction model learning program 170 of the AP server 100 executes a re-learning process to update the processing time prediction model depending on the degree of deviation between the predicted value of the processing time of the batch processing program 120 before execution of the business operation processing and the actual processing time in the business operation processing (step 207).

Details of the business operation process and the learning process will be described later. After that, the processes from step 205 onwards are repeated. Note that the execution of the processes of steps 204, 205 and 207 may be omitted.

Next, we will explain the details of each process in the job execution support process.

<Analysis processing and job parallel processing>
FIG. 4 is a flowchart illustrating the analysis process and the job parallelization process in detail.

First, the analysis program 150 loads the batch processing program 120 (step 301).

Then, the intermediate file dependency analysis unit 151 executes an intermediate file dependency analysis process to analyze the dependencies of each job based on the intermediate files input and output by the batch processing program 120 (step 302).

(Intermediate file analysis table)
5 is a diagram for explaining an example of an intermediate file analysis table 500, which is data created by the intermediate file dependency analysis process. The intermediate file analysis table 500 is data that stores intermediate files output by each job or intermediate files input to each job. For example, a job "JOB A" of the batch processing program 120 outputs an intermediate file "FILE f1"("OUT"), and "JOB B" reads the intermediate file ("IN").
On the other hand, the job "JOB E" reads the intermediate files "FILE f2" and "FILE f3"("IN").

Furthermore, as shown in step 303 of FIG. 4, the structure analysis unit 153 executes a structure analysis process to analyze the structure of the batch processing program 120. The results of the analysis are stored in a structure analysis table.

(Structural analysis table)
6 is a diagram illustrating an example of a structure analysis table 550 and an execution order and processing unit of each job.
5, and a job 556 to be executed immediately after each job 552. In the example shown in FIG. 5, the batch processing program 120 is made up of one processing unit 557 (which is divided into multiple groups in the subsequent processing) that executes each job in the order of "JOB A", "JOB B", "JOB C", ..., and "JOB G".

Also, as shown in step 304 of FIG. 4, the access destination dependency analysis unit 152 executes an access destination dependency analysis process to identify the tables accessed by the SQL part of each job, and stores the results in the access destination analysis table.

(Access destination analysis table)
7 is a diagram showing an example of an access destination analysis table 600, which is data created by the access destination dependency analysis process. The access destination analysis table 600 is a table consisting of one or more records having each of the following items: a program item 602 in which information specifying a program (hereinafter referred to as a job program) that realizes a job in the batch processing program 120 is set, a job item 604 in which an identifier of a job that constitutes a job related to the program item 602 is set, a table item 606 in which an identifier of a table accessed by a job related to the job item 604 is set, a column item 608 in which a specific data portion of the table related to the table item 606 (e.g., a column, record, or other specific area in a table) is set, and an access item 610 in which information specifying an access form (e.g., data registration, update, reference, etc.) to the data portion related to the column item 608 by the job related to the job item 604 is set.

In the example of the figure, the batch processing program 120 "job-c-3.xxx" that realizes "JOB C" registers ("INSERT") each piece of data into columns "t21", "t22", and "t23" in table "t2". On the other hand, the batch processing program 120 "job-d-1.xxx" that realizes "JOB D"
It references the data in columns "t11", "t12", "t13", "t14", and "t15" in table "t1"("SELECT").

FIG. 8 shows an example of a job program. In this example, the job program registers a specific value in a part of a column in a specific table.

Next, as shown in step 305 of FIG. 4, after executing the intermediate file dependency identification process and the structure analysis process, the intermediate file dependency analysis unit 151 executes a job group creation process to create job groups by grouping each job based on the results of these processes (intermediate file analysis table 500 and structure analysis table 550). The job group creation process will be described in detail later (step 305).

The program reconfiguration unit 161 of the conversion program 160 also uses the results of the table dependency determination process (access destination analysis table 600) to execute a program reconfiguration process to reconfigure the batch processing program 120 and each table it accesses based on the table being accessed by the job group (step 306). The program reconfiguration process will be described in detail later.

The program reconfiguration unit 161 creates a job definition for executing parallel processing by the batch processing program 120 reconfigured in step 306 (step 307). This completes the analysis process and job parallelization process.

<Job group creation process>
9 is a flowchart for explaining the details of the job group creation process. The intermediate file dependency analysis unit 151 acquires the processing result of the intermediate file dependency analysis process (step 401). Specifically, for example, the intermediate file dependency analysis unit 151 reads the intermediate file analysis table 500.

Based on the processing results read in step 401, the intermediate file dependency analysis unit 151 identifies all jobs that do not have input files (starting job set) and stores each identified job as a starting job in each job group (hereinafter referred to as a starting job) (step 402).

Specifically, for example, the intermediate file dependency analysis unit 151 identifies all records of jobs in which "IN" is not set in the intermediate file analysis table 500. In the example of the intermediate file analysis table 500 in FIG. 5, the records "JOB A", "JOB C", "JOB D", and "JOB F" are identified.

For each origin job identified in step 402, the intermediate file dependency analysis unit 151 identifies all other jobs that have dependencies on the input and output of intermediate files in relation to the origin job, and executes a subsequent job identification process to identify each of the identified jobs and the origin job as one job group (step 403). The subsequent job identification process will be described in detail later.

The intermediate file dependency analysis unit 151 identifies all jobs that belong to multiple job groups based on each job group identified in the subsequent job identification process, and for each identified job, combines the multiple job groups to which the job belongs into one job group (step 404).

The intermediate file dependency analysis unit 151 identifies the execution order of each job group based on the execution order of each job identified by the structural analysis process (step 405).

Specifically, for example, the intermediate file dependency analysis unit 151 identifies the order of execution of each job group by referring to the structural analysis table 550 and identifying the precedence/sequence relationship between the first job (start job) and the last job of each job group.

Here we explain the details of the subsequent job identification process.

<Subsequent job specific processing>
10 is a flowchart for explaining the details of the succeeding job identification process. First, the intermediate file dependency relationship analysis unit 151 judges whether or not the starting job has been identified in step 402 (step 501).

If the starting job can be identified, the intermediate file dependency analysis unit 151 executes step 502 described below, and if the starting job cannot be identified, the subsequent job identification process ends.

In the example of the intermediate file analysis table 500 in FIG. 5, the set SJS of source jobs includes {JOB A
, JOB C, JOB D, and JOB F}, step 502 is executed.

In step 502, the intermediate file dependency analysis unit 151 selects one origin job from the set of origin jobs obtained in step 402, and creates a new job group that includes the selected origin job.

In the example of the intermediate file analysis table 500 in FIG. 5, if the selected starting job is "JOB A", "JG1=[JOB A]" is created as a new job group.

In the job group (hereinafter referred to as the target group) created in step 502, the intermediate file dependency analysis unit 151 creates a successor job set NJS, which is data that stores information about jobs (successor jobs) that are executed after the execution of the starting job, and a successor file set NFS, which is data that stores information about intermediate files accessed by the successor jobs, as empty sets (step 503).

The intermediate file dependency analysis unit 151 determines whether the origin job selected in step 502 (hereinafter referred to as the selected job) outputs an intermediate file (step 504). Specifically, the intermediate file dependency analysis unit 151 determines whether "OUT" is registered in the record of the selected job in the structure analysis table 550.

If the selected job outputs an intermediate file (step 504: YES), the intermediate file dependency analysis unit 151 executes step S505 described below. If the selected job does not output an intermediate file (step 504: NO), the intermediate file dependency analysis unit 151 executes step S506 described below.

In the example of intermediate file analysis table 500 in Figure 5, "JOB A" outputs intermediate file "FILE f1", so step 505 is executed.

In step 505, the intermediate file dependency analysis unit 151 adds all intermediate files output by the selected job to the elements of the successor file set NFS.

In the example of the intermediate file analysis table 500 in FIG. 5, the successor file set NFS = {FILE f1}.

The intermediate file dependency analysis unit 151 acquires one intermediate file from the successor file set NFS obtained in step 505 (step 507). The intermediate file dependency analysis unit 151 deletes the information about the acquired intermediate file from the successor file set NFS (as it has already been taken into account).

In the example of the intermediate file analysis table 500 in FIG. 5, the intermediate file "FILE f1" is acquired, resulting in "successor file set NFS={} (empty set)."

The intermediate file dependency analysis unit 151 adds all jobs that input the intermediate file obtained in step 507 to the elements of the successor job set NJS (step 508).

In the example of the intermediate file analysis table 500 in FIG. 5, since there exists "JOB B" which has input the previous intermediate file "FILE f1", "successor job set NJS={JOB B}" is obtained.

The intermediate file dependency analysis unit 151 acquires one job from the subsequent job set NJS acquired in step 508 (step 510). The intermediate file dependency analysis unit 151 deletes the acquired job from the subsequent job set NJS (as it has already been considered).

5, as a result of "JOB B" being selected, "successor job set NJS={} (empty set)" is obtained (step 511). After that, the process of step 504 is repeated for the added job.

The intermediate file dependency analysis unit 151 adds the job obtained in step 510 to the job group created in step 502.

In the example of the intermediate file analysis table 500 in FIG. 5, “Job group “JG1”=[JOB
After that, the process from step 504 onwards is repeated.

In step 506, the intermediate file dependency analysis unit 151 checks whether an intermediate file exists in the subsequent file set NFS previously acquired in step 505. If an intermediate file exists in the subsequent file set NFS (step 506: YES), the intermediate file dependency analysis unit 151 executes the processing from step 507 onwards, and if an intermediate file does not exist in the subsequent file set NFS (step 506: NO), the intermediate file dependency analysis unit 151 executes step 509 described below.

In step 509, the intermediate file dependency analysis unit 151 checks whether the element of the succeeding job set NJS created in step 508 includes a job. If the element of the succeeding job set NJS includes a job (step 509: YES), the intermediate file dependency analysis unit 151 executes step 511 for that job, and if the element of the succeeding job set NJS does not include a job (step 509: NO), the intermediate file dependency analysis unit 151 repeats the processing from step 501 onwards.

In the example of the intermediate file analysis table 500 in FIG. 5, the succeeding job identification process results in the following: "JG1 = [JOB A, JOB B, JOB E], JG2 = [JOB C], JG3 = [JOB D, JOB E], JG4 = [JOB
Then, in step 404, job groups "JG1" and "JOB3" are combined, and finally three job groups "JG1 = [JOB A, JOB B, JOB D, JOB E], JG2 = [JOB C], JG4 = [JOB F, JOB G]" are extracted.

11 is a diagram showing an example of a job group in the batch processing program 120 specified by the job group creation process, and an example of the corresponding structure analysis table 550. As shown in the diagram, the batch processing program 120 is made up of a plurality of job groups 558, specifically, a job group 558a in which jobs are executed in the order of "JOB A", "JOB B", "JOB D", and "JOB E", a job group 558b in which "JOB C" is executed, and a job group 558c in which jobs are executed in the order of "JOB F" and "JOB G".

In this way, the job group creation process identifies job groups that are independent of each other in terms of processing and can be processed in parallel based on dependencies related to the input and output of intermediate files, making it possible to achieve parallel processing at a finer granularity than when simply considering the timing of job execution while maintaining the consistency of the processing results of the batch processing program 120. This makes it possible to further shorten the processing time of the batch processing program 120.

<Program Reconstruction Processing>
FIG. 12 is a flowchart illustrating the program reconfiguration process in detail.

First, for each job group determined in the job group creation process, the program reconfiguration unit 161 identifies all pairs of job groups that have a common table to be accessed (step 601). Note that "access" here refers to registering, updating, or deleting data in a table, and does not include referencing data.

If there is no job group pair with a common access table (step 60
If there is a pair of job groups that share a common access table (step 601: YES), the program reconfiguration unit 161 executes step 602 described below.

In the example of the access destination analysis table 600 in FIG. 7 and the structure analysis table 550 in FIG. 11, there are three job groups "JG1 = [JOB A, JOB B, JOB D, JOB E]" and "JG2 = [JOB C]
" and "JOB3 = [JOB F, JOB G]" are present, the pair of "JG1" and "JG2" that accesses table "t2" is identified.

In step 602, the program reconfiguration unit 161 determines, for each of the sets of job groups identified in step 601 (hereinafter referred to as competing job groups), whether the competing job groups are accessing a common column in a table (i.e., whether the accessed columns are in conflict) (step 602).

If there are no competing job groups accessing the common column (step 602: NO), the program reconfiguration unit 161 executes the processing of steps 603 to 605 described below. On the other hand, if there are competing job groups accessing the common column (step 602: YES), the program reconfiguration unit 161 executes the processing of step 604 described below.

7, as a result of referring to the table item 606 and the access item 610, it is determined that "job-c-3.xxx", which is a job program in a job group having "JOB C", has registered data in the table "t2", and "job-e-5.xxx", which is a job program in another job group having "JOB E", has also registered data in the table "t2" (step 601), so that the process of step 602 is performed. In step 602, as a result of referring to the column item 608 and the access item 610, it is determined that the columns in which the job programs "job-c-3.xxx" and "job-e-5.xxx" in each competing job group have registered data do not overlap ("t21", "t22", "t23" and "t24", "t25"), so that the process of step 603 is performed.

In step 603, for all competing job groups whose access destination tables are determined to be in conflict in step 602, the program reconfiguration unit 161 divides the common access destination table of the competing job groups into multiple tables.

Specifically, the program reconfiguration unit 161 performs vertical partitioning of the table. For example, the program reconfiguration unit 161 identifies the portion of the column that is accessed by each competing job group in the table to be accessed, and separates the portion of the column from the table to create a new table.

In the example of the access destination analysis table 600 in FIG. 7, the access destination table "t2" is divided into a table "t2A" that includes columns "t21,""t22," and "t23" that are accessed by a job program "job-c-3.xxx" in a competing job group, and a table "t2B" that includes columns "t24" and "t25" that are accessed by a job program "job-e-5.xxx" in another competing job group.

The program reconfiguration unit 161 modifies, i.e., rewrites, the batch processing program 120 so that each competing job group can access the new table created by division in step 603 (step 605).

In the example of the access destination analysis table 600 in Fig. 7, the batch processing program 120 is rewritten so that "job-c-3.xxx", which has been accessing columns "t21", "t22", and "t23", accesses table "t2A" instead of table "t2", and "job-e-5.xxx", which has been accessing columns "t24" and "t25", references table "t2B" instead of table "t2". The program reconfiguration process then ends (step 606).

On the other hand, in step 604, the program reconfiguration unit 161 creates a new job group by combining competing job groups whose accessed tables and columns conflict with each other.

This process is performed because job groups in which both tables and columns are in conflict have dependencies on their execution order, making parallel processing impossible. The program reconfiguration process then ends (step 606).

In this way, by dividing the tables accessed by each job group based on their competitive relationships, the waiting time for accessing each table (DB
This is expected to have a significant effect in shortening the processing time of the batch processing program 120.

<Operation test processing>
13 is a flowchart illustrating an example of an operation test process. Note that this process may be started by the cloud system 10 receiving a predetermined input from a user after the job parallelization process is completed, or may be automatically executed by the cloud system 10 at a predetermined timing (e.g., at a predetermined time interval or at a predetermined time).

First, the management program 125 creates a system environment on the cloud that is equivalent to the environment of the user system 50 that was operated in an on-premise environment (step 701).

Specifically, for example, the management program 125 creates a virtual machine in the AP server 100 that has the same specifications as the virtual machine that is configured in the user system 50 and is used to execute the batch processing program 120.

The management program 125 executes the batch processing program 120 created by the job parallelization process on the virtual machine created in step 701. The management program 125 then obtains the time (processing time) required for the execution of this batch processing program 120 and calculates the usage fee (step 702).

Specifically, first, the management program 125 executes the batch processing program 120 by inputting specified business data (e.g., transaction data from a specified past month) into the batch processing program 120, and obtains the processing time. The management program 125 also calculates the usage fee corresponding to the processing time by referring to a fee table (database) that stores the relationship between the usage fee (cost) of the cloud system 10 and the processing time of the program, which has been stored in advance. The management program 125 may also accept input of the processing time or usage fee from the user.

The management program 125 changes the performance of the virtual machine created in step 701, and then executes the batch processing program 120 again, as in step 702. The management program 125 calculates the processing time required to execute this batch processing program 120, and the usage fee for it (step 703).

Specifically, for example, the management program 125 increases the performance of the virtual machine created in step 701 by one level (for example, by increasing the memory by a predetermined value, increasing the CPU performance by a predetermined value, etc.). Note that the performance may also be decreased. In this case, the management program 125 may automatically change the performance, or the management program 125 may receive performance information input from the user.

The management program 125 determines whether the usage fee calculated in step 703 is less than the usage fee calculated in step 702 and whether the processing time calculated in step 703 is shorter than the processing time calculated in step 702 (step 704).

If the usage fee has been reduced and the processing time has been shortened (step 704: YES), the management program 125 changes the virtual machine to a different performance (further increasing the performance by one step) and executes step 703. On the other hand, if the usage fee has not been reduced or the processing time has not been shortened (step 704: NO), the management program 125 executes step 705, which will be described next.

In step 705, the management program 125 sets the performance immediately prior to the performance last set in step 703 as the final performance of the virtual machine. This ends the operation test process.

<Initial learning process>
14 is a flowchart for explaining an example of the initial learning process. First, the management program 125 executes the batch processing program 120 created in the job parallelization process by inputting business data from a predetermined period in the past (e.g., transaction data from a predetermined month in the past) into the batch processing program 120 created in the job parallelization process in a virtual machine with the performance set in the operation test process. Then, the management program 125 calculates the processing time and usage fee of this batch processing program 120 in the same manner as the operation test process (step 801). The management program 125 stores the calculated processing time and usage fee in the analysis result storage area 203 and the prediction model storage area 204.

In addition, the above-mentioned operation test process may use the same business data as in this process. In addition, the management program 125 may accept input of the processing time or usage fee from the user.

Then, the prediction model creation unit 171 of the prediction model learning program 170 learns the relationship between the execution environment of the batch processing program 120 (e.g., the execution time of the batch processing program 120, the content or amount of business data used in step 801, the specifications of the virtual machine) and the processing time calculated in step 801, and stores the relationship as a processing time prediction model (step 802).

Specifically, for example, the prediction model creation unit 171 creates a program, database, or conditional expression that specifies the relationship between the two.

The prediction execution unit 172 calculates the prediction accuracy of the processing time prediction model created in step 802 (step 803).

Specifically, for example, the prediction execution unit 172 inputs the business data for a predetermined period in the past used in step 801 into the processing time prediction model, thereby calculating the processing time of the batch processing program 120 for that period. The prediction execution unit 172 also calculates the usage fee (cost) corresponding to the calculated processing time by using the fee table. The prediction execution unit 172 then compares the calculated processing time and usage fee with the actual processing time of the batch processing program 120 stored in advance as actual values and the usage fee actually charged, and calculates the difference value (the difference between the calculation result using the processing time prediction model and the actual value) for the processing time and usage fee. This ends the initial learning process (step 804).

<Information presentation processing>
15 is a flowchart for explaining the details of the information presentation process. First, the estimate presentation unit 181 of the information presentation program 180 stores the performance set in the operation test process as the performance (resource proposal) of the virtual machine to be proposed to the user (step 901).

The estimate presentation unit 181 uses the processing time prediction model to calculate the usage fee for the execution of the batch processing program 120 for a specified future period (step 902).

Specifically, for example, the estimate presenting unit 181 calculates the usage fee by inputting information on a predetermined future period to be predicted (e.g., information on the next three years) into the processing time prediction model. Note that the estimate presenting unit 181 may calculate the usage fee by inputting business data to be used in a predetermined future period.

The estimate presentation unit 181 calculates a parameter (reliability) indicating the reliability of the usage fee calculated in step 902 (step 903).

Specifically, for example, the estimate presenting unit 181 calculates the prediction accuracy (prediction accuracy calculated in the initial learning process) of the processing time prediction model for a past period (e.g., the same past month) corresponding to the period specified in step 902 as the reliability.

The quotation presentation unit 181 creates quotation presentation screen data 302 for displaying information such as the batch processing program 120 information, the virtual machine performance (resource plan) stored in step 901, the predicted usage fee calculated in step 902, the actual usage fee value, and the reliability calculated in step 903, and displays the created quotation presentation screen data 302 on a specified screen of the AP server 100, the Web server 300, or the user system 50, etc.

(Quote presentation screen)
16 is a diagram showing an example of a quote presentation screen. The quote presentation screen 1000 has a job information item 1010 including information 1012 on the current execution status of the batch processing program 120 and information 1014 on the last execution time, a machine information item 1020 including information on the performance 1022 of a virtual machine, information 1024 on the usage fee related to the use of the virtual machine (e.g., unit price per unit time), and calculated usage fee 1026 for this month (comparison information with the previous month may also be included), a cost transition item 1030 in which fluctuations 1032 of the actual value of the usage fee in the past and fluctuations 1034 of the predicted value of the usage fee are displayed in graph form, and a processing time transition item 1040 in which the actual value and calculation result of the processing time are displayed similarly to the cost transition item 1030.

By referring to this quotation presentation screen 1000, the user can learn the details of the prediction regarding the extent to which usage fees and processing time can be optimized by using the batch processing program 120 based on the performance of the virtual machine, as well as the accuracy of the prediction.

<Business operation processing and re-learning processing>
FIG. 17 is a flowchart illustrating the details of the business operation process and the relearning process.

First, the management program 125 executes the batch processing program 120 as needed in accordance with business operations (step 1001). Specifically, for example, the management program 125 executes the batch processing program 120 based on the input of specified transaction data at a timing specified by the user or at a preset timing (for example, at a specified time interval or at a specified time).

Then, the management program 125 calculates the processing time and usage fee of the batch processing program 120, similar to the initial learning process (step 1002). The management program 125 stores the calculated processing time and usage fee in the analysis result storage area 203 and the prediction model storage area 204. The management program 125 may also accept input of the processing time or usage fee from the user.

The learning unit 173 of the prediction model learning program 170 learns the processing time prediction model in the same manner as the initial learning process by inputting the results of steps 1001 and 1002 into the processing time prediction model as additional training data (step 1003). In addition, the prediction execution unit 172 calculates the prediction accuracy of the processing time prediction model in the same manner as the initial learning process.

Based on the prediction accuracy calculated in step 1003 and the prediction accuracy calculated so far, the prediction execution unit 172 determines whether the prediction accuracy of the processing time prediction model is stable (step 1004). Specifically, the prediction execution unit 172 analyzes the time series fluctuation of the prediction accuracy calculated so far, and determines whether the prediction accuracy has converged to a certain value.

If the prediction accuracy of the processing time prediction model is not stable (step 1004: NO), the processing from step 1001 onwards is repeated, and if the prediction accuracy of the processing time prediction model is stable (step 1004: YES), the prediction execution unit 172 does not further train the processing time prediction model for the batch processing program 120 (step 1005). Specifically, for example, the prediction execution unit 172 sets a predetermined re-learning flag to "FALSE". This ends the business operation processing and re-learning processing.

As described above, the job execution support system 1 of this embodiment identifies intermediate files for each job in a batch processing program 120 that has multiple jobs, identifies processing dependencies between each job based on the intermediate files, and based on the identified dependencies, identifies jobs in the batch processing program 120 that are to be executed in parallel with each other, and executes the identified jobs in parallel.

In this way, by focusing on intermediate files in the batch processing program to identify dependencies between jobs and configuring parallel processing based on this, the execution time of the batch processing program 120 can be reduced. This makes it possible to optimize job execution.

The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. For example, it is possible to add, delete, or replace part of the configuration of the embodiment with other configurations.

For example, in this embodiment, the program reconfiguration process vertically divides the table based on columns, but other division methods may be used. For example, horizontal division based on records, division based on specific sections within the table, etc. may be used.

In addition, in this embodiment, the table is a database, but the data format is not particularly limited, and it is sufficient if the table constitutes a predetermined data group.

In addition, in this embodiment, the batch processing program 120 or the service program 104 is executed by a virtual machine, but it may also be executed by a physical computer.

In addition, in this embodiment, the flow of the succeeding job identification process is shown as a method for creating a job group based on an intermediate file, but the grouping method is not limited to the flow shown here, and any method can be used as long as the dependencies between jobs based on intermediate files can be identified.

In addition, in this embodiment, in the operation test process, a virtual machine similar to that used during on-premise operation is created on the cloud to perform an operation test of the batch processing program 120, but a virtual machine may also be created locally (on the user system 50) to perform an operation test of the batch processing program 120. This can reduce the usage fee for the cloud system 10 related to the operation test.

In addition, in this embodiment, transaction data for each month is used as the business data to be used in the processing time prediction model, but other data, such as data showing the relationship between transaction data for each month and the month before and after, and data such as the fee plan for usage fees set by the user, may also be used.

In addition, in this embodiment, the execution cost may be set as an output parameter of the processing time prediction model in addition to or instead of the processing time (i.e., a fee table is not used).

The above description of the present specification makes at least the following clear. That is, in the job execution support system 1 of this embodiment, at least one of the jobs accesses a portion of data in a predetermined data group, and when the arithmetic device analyzes the access destinations of the jobs executed in parallel to each other for the data group, and identifies a combination of a first job and a second job executed in parallel to each other in which the data group of the access destination is the same but the data portions of the data group of the access destination do not overlap, the arithmetic device may execute an access destination dependency analysis process that divides the data group into a first data group including a data portion accessed only by the first job and a second data group including a data portion accessed only by the second job.

In this way, if there is a set of jobs (job groups) that conflict over the data groups (tables) they access but do not conflict over the specific access destinations (columns, etc.) within those data groups, you can split the tables so that each job accesses a different table, eliminating bottlenecks related to accessing the tables and effectively reducing the processing time of the batch processing program.

In addition, in the job execution support system 1 of this embodiment, the arithmetic device may execute a program reconfiguration process that changes the target program so that the identified first job accesses the first data group and the identified second job accesses the second data group.

In this way, by rewriting the description of the access destination for each job in the batch processing program 120, the access time to the access destination for each job can be reduced.

Furthermore, in the job execution support system 1 of this embodiment, the arithmetic device may execute in parallel jobs under a plurality of execution environments to acquire each processing time of the target program, and based on each acquired processing time, execute a prediction model creation process to create information storing a relationship between the processing time of the target program and the execution environment of the target program, a prediction execution process to input a specified execution environment into the information to calculate the processing time of the target program under that execution environment, and calculate an execution cost having a predetermined relationship with the calculated processing time, and an estimate presentation process to display information of the calculated cost.

In this way, by creating a prediction model for the processing time of the batch processing program 120 and its execution environment, it is possible to calculate the processing time of the batch processing program 120 in an execution environment specified by a user, etc. This also allows the user to know the execution cost of the batch processing program 120 in the specified execution environment.

In addition, in the job execution support system of this embodiment, the computing device may display information about the specified execution environment together with the cost information during the estimate presentation process.

This allows the user to know the appropriate execution environment for the batch processing program 120.

As described above, according to the job execution support system 1 of this embodiment, by performing parallelization of jobs, etc., it is expected that the processing time of batch processing for the batch processing program 120 can be shortened on the cloud (cloud system 10) compared to when it is operated on-premise (user system 50), and the usage fee for the cloud system 10 (usage fee for virtual machine) can be reduced.

In addition, by analyzing dependencies between conflicting tables and specific access destinations, you can maximize the effect of reducing processing time by parallelizing jobs.

In addition, by using the processing time prediction model, the operating costs (usage fees) of the cloud system 10 can be predicted, making it easier for companies to secure budgets.

1 Job execution support system, 10 Cloud system, 100 AP server, 200 DB server, 300 Web server

Claims (6)

a storage device storing a database having predetermined data items; and
an intermediate file dependency analysis process for identifying data output by or input to each job in a process target program that executes a plurality of jobs that execute one or a plurality of processing instructions, thereby identifying dependency relationships between the jobs based on the data, and grouping the jobs in the process target program based on the identified dependency relationships to create job groups that can be executed in parallel with each other;
an access dependency analysis process for identifying the data items of the database accessed by each job and whether the access form to the data related to the data items is registration, update, deletion, or reference;
an access destination dependency relationship analysis process for, when a combination of the first job group and the second job group executed in parallel with each other is identified in which the databases to which the access destinations are the same but data items to which the access destinations are accessed by registering, updating, or deleting data in the databases do not overlap, dividing the databases into a first database including data items accessed only by the first job group by registering, updating, or deleting data, and a second database including data items accessed only by the second job group by registering, updating, or deleting data;
a job execution process for executing in parallel the job groups that can be executed in parallel with each other based on the program to be processed;
A job execution support system comprising a computing device that executes the above. The computing device,
executing a program reconfiguration process for modifying the processing target program so that the specified first job group accesses the first database and the specified second job group accesses the second database ;
The job execution support system according to claim 1 . The computing device,
a prediction model creation process for acquiring each processing time of the target program by executing the job groups that can be executed in parallel in a plurality of execution environments in parallel, and creating a model that learns the relationship between the processing time of the target program and the execution environment of the target program based on each acquired processing time;
a prediction execution process for inputting a specified execution environment into the model to calculate a processing time of the target program in the execution environment, and calculating an execution cost having a predetermined relationship with the calculated processing time;
and executing an estimate presentation process for displaying information on the calculated cost.
The job execution support system according to claim 1 . the computing device displays information on the specified execution environment together with the cost information in the estimate presentation process.
The job execution support system according to claim 3 . The computing device,
performing a re-learning process for re-learning the model by inputting an execution environment and a processing time of the program to be processed as additional training data every time the program to be processed is executed;
The job execution support system according to claim 3 . An information processing device having a storage device storing a database having predetermined data items ,
an intermediate file dependency analysis process for identifying data output by or input to each job in a process target program that executes a plurality of jobs that execute one or a plurality of processing instructions, thereby identifying dependency relationships between the jobs based on the data, and grouping the jobs in the process target program based on the identified dependency relationships to create job groups that can be executed in parallel with each other;
an access dependency analysis process for identifying the data items of the database accessed by each job and whether the access form to the data related to the data items is registration, update, deletion, or reference;
an access destination dependency relationship analysis process for, when a combination of the first job group and the second job group executed in parallel with each other is identified in which the databases to which the access destinations are the same but data items to which the access destinations are accessed by registering, updating, or deleting data in the databases do not overlap, dividing the databases into a first database including data items accessed only by the first job group by registering, updating, or deleting data, and a second database including data items accessed only by the second job group by registering, updating, or deleting data;
a job execution process for executing in parallel the job groups that are executable in parallel with each other based on the program to be processed; and a job execution support method for executing the job execution process.

Images