JP2014215764A - Set value generation device, distribution processor, set value generation method and set value generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To apply an appropriate set value to each job in response to request processing to a distribution processor.SOLUTION: In a distribution processor 10 including: a distribution processing base part 130 for executing a job by using a plurality of task execution parts 132; and an interface presentation part 110 for converting an input query into the instruction of a job which can be processed by the distribution processing base part 130, a set value generation part 120 for generating the set value of each job includes: an execution time information acquisition part 121 for acquiring the execution time information of the job in the case of starting the job; and a set value generation processing part 123 for generating a set value to be used for the execution of the job by referring to the acquired execution time information of the job and a set value generation model 122, and for outputting it to the distribution processing base part 130. The distribution processing base part 130 executes the job by the distribution processing base part 130 by using the output set value of the job.

Description

  The present invention relates to a setting value generation device, a distributed processing device, a setting value generation method, and a setting value generation program.

  In order to analyze a large amount of various data in a short time, a distributed processing system having a scale-out type feature is attracting attention. In this technique, a plurality of computers (nodes) are coordinated, and data processing is shared among individual computers, thereby enabling large-scale data processing as a whole. Since a distributed processing infrastructure (such as MapReduce) is responsible for managing to which computer the processing is entrusted, the user only has to describe the processing contents to be realized as instructions to the distributed processing infrastructure. Here, since the description of the instruction is complicated, interface tools (Hive, Pig, etc.) for easily instructing the contents of the process requested by the user (request process) have appeared.

  When this interface tool receives an input of a request process described in a format with a high level of abstraction that is easy for the user to understand, the interface tool converts the request process into an instruction format of an execution unit that can be processed on the distributed processing platform.

J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. In OSDI’04, 2004, pp.137-149. A. Thusoo, JSSarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive-a petabyte scale data warehouse using Hadoop. In ICDE'10 , IEEE, 2010, pp. 996-1005. Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar, and Andrew Tomkins.Pig latin: a not-so-foreign language for data processing.In SIGMOD'08, ACM, 2008

  In general, the processing (request processing) content at a higher concept level specified by the user is converted into a process composed of a plurality of execution units (jobs) and executed on the distributed processing platform. However, the setting value designated by the user cannot be changed for each job constituting the request process, and the appropriateness of the setting value is unknown until the job is executed. Therefore, the optimization of the setting values of each job for the request processing in the distributed processing system becomes insufficient, and there is a problem that the distributed processing system cannot exhibit sufficient performance.

  For example, among the plurality of jobs generated from request processing (query, etc.) in the distributed processing system, the latter half of the job may take over the output data of the first half of the job. An appropriate value could not be set for each job, such as changing the setting value of the latter half job according to the amount.

  In view of the above, an object of the present invention is to solve the above-described problems and to sufficiently exhibit the performance of the distributed processing system by giving an appropriate setting value to each job for the request processing to the distributed processing system.

  In order to solve the above-described problem, the present invention provides a distributed processing base unit that executes one or more jobs using one or more nodes, and receives the request processing input to the distributed processing base unit. An interface providing unit that converts processing contents into a job command that can be processed by the distributed processing platform, and executes the one or more jobs using the distributed processing platform based on the converted command; A setting value generation device that generates setting values for each of the jobs executed in the distributed processing device, and at the time of executing execution of the job, the execution time information of the job is acquired from the distributed processing infrastructure unit Based on the information acquisition unit, the execution time information of the job, a storage unit that stores a setting value generation model for obtaining the setting value of the job, and the execution time information of the job, The serial set value generation model, generates a setting value used for execution of the job, and the set value generating device, characterized in that it comprises a set value generation processing unit for outputting to the distributed processing foundation unit.

  According to the present invention, since an appropriate setting value is given to each job for the request processing to the distributed processing system, the performance of the distributed processing system can be sufficiently exhibited.

FIG. 1 is a diagram illustrating a configuration of a distributed processing apparatus. FIG. 2 is a diagram for explaining the function f used in the set value generation model. FIG. 3 is a diagram for explaining the optimum value of the division size M of the input data amount D. FIG. 4 is a flowchart showing the processing procedure of the distributed processing apparatus. FIG. 5 is a graph showing the division size M of the input data amount D and the measured value of the execution time. FIG. 6 is a diagram illustrating a computer that executes a distributed processing program.

[Constitution]
Hereinafter, embodiments of a distributed processing apparatus (distributed processing system) according to the present invention will be described with reference to the drawings. As shown in FIG. 1, when receiving an input of a request process (query or the like) requested by the present system from an external apparatus or the like, the distributed processing apparatus 10 interprets the content of the request process and executes data processing. Data processing is executed by one or more task execution units 132 included in the distributed processing apparatus 10. The task execution unit (node) 132 is realized by a computer (not shown) included in the distributed processing apparatus 10.

  The distributed processing apparatus 10 includes an interface providing unit 110, a setting value generating unit 120, and a distributed processing base unit 130.

  When the interface providing unit 110 receives an input of a request process (query or the like) to the distributed processing apparatus 10, the interface providing unit 110 interprets the content of the request process and can execute one or more execution units (jobs) that can be processed by the distributed processing base unit 130. ) Command. Then, the interface providing unit 110 executes each job using the distributed processing base unit 130 based on the converted command. That is, the interface providing unit 110 instructs the distributed processing base unit 130 to execute each job, and receives the job execution result from the distributed processing base unit 130. It is realized by the described Hive and the Pig described in Non-Patent Document 3.

  The interface providing unit 110 includes a conversion unit 111 and an execution unit 112. When receiving the input of the request process, the conversion unit 111 converts the content of the request process into one or more job instructions that can be processed by the distributed processing base unit 130. The execution unit 112 executes a job using the distributed processing platform unit 130 based on the instruction converted by the conversion unit 111. Here, based on the converted instruction, the execution unit 112 uses the distributed processing platform unit 130 according to any dependency between jobs (for example, the output result of job 1 becomes the input of job 2). Execute the job and output the execution result to an external device. The execution unit 112 uses the setting value of the job output from the setting value generation unit 120 when the distributed processing base unit 130 executes the job.

  The setting value generation unit 120 generates setting values used in the job using job execution time information (for example, input data amount D to the job), and outputs the generated setting values to the distributed processing platform unit 130. . The setting value used in the job is, for example, the maximum data amount M for each task constituting the job, which requires that the resource is used to the maximum to minimize the job execution time. The setting value generation unit 120 includes a runtime information acquisition unit 121, a setting value generation model 122, and a setting value generation processing unit 123. The model update unit 124 may or may not be equipped, and will be described later.

  The runtime information acquisition unit 121 acquires job runtime information from the distributed processing infrastructure unit 130. This execution time information is information obtained at the start of execution of each job, and depends on the execution result of a job executed before that job. This execution time information is, for example, the input data amount D input to the job after the job preceding the job ends, the number N of task execution units 132, and the like.

The setting value generation model 122 is a model for obtaining an optimum setting value of the job based on the execution time information of the job, and is given by, for example, a function f as shown in FIG. In this function f, (x ₁ ,..., X _n ) is an argument given to the model and is a value of execution time information of the job (for example, input data amount D to the job). Further, (w ₁ ,..., W _l ) in the function f is a constant value appearing in the model, for example, the number of task processes w of each task execution unit 132 in the job. Then, (c ₁ ,..., C _m ) obtained from the function f is an output of the model, which is an optimal setting value for the job. The set value generation model 122 is stored in a predetermined area of a storage unit (not shown) included in the distributed processing apparatus 10 and is referred to by the set value generation processing unit 123. It should be noted that the function f in the set value generation model 122, the types of parameters used in the function f, constants, and the like are obtained in advance through experiments or the like.

  Here, the optimum setting value of the job is, for example, the value of the maximum data amount per task (that is, the division size of the input data amount D) M that shortens the execution time of the job as much as possible. The optimum value of the division size M of the input data amount D is obtained by, for example, the set value generation model 122 shown in the following equation (1).

  In Equation (1), w is a constant given in advance.

  The set value generation processing unit 123 uses the set value generation model 122 shown in Expression (1), so that the division size M of the input data amount D that can shorten the execution time of the job as much as possible can be obtained. The reason for this will be described with reference to FIG.

  For example, the number of task execution units 132 that execute jobs is two (task execution unit 132a and task execution unit 132b), and w = 3. Here, as shown in (a), when the input data amount D is divided into five, that is, when the maximum data amount M per task is set to D / 5, the task execution unit 132b has a waiting time. Will occur. That is, the job execution time of the entire task execution unit 132 group is the time until the task execution unit 132a finishes the fifth task.

  On the other hand, as shown in (b), if the input data amount D is six (that is, the number of task processes w (3) × the number N of task execution units 132 (2)), the waiting time of the task execution unit 132b The task execution units 132a and 132b can finish the job almost simultaneously. Thereby, the job execution time of the task execution unit 132 group as a whole can be reduced.

  As described above, the set value generation processing unit 123 uses the above-described equation (1) as the set value generation model 122, thereby reducing the job execution time of the entire task execution unit 132 group so as to reduce the input data amount D. Can be obtained.

  The setting value generation processing unit 123 refers to the execution time information of the job and the setting value generation model 122 to generate a setting value used in the job. For example, when the set value generation processing unit 123 acquires the input data amount D to the job 2 as the execution time information of the job 2, the set value generation processing unit 123 uses the formula (1) that is the set value generation model 122 to input data in the job 2 The division size M of the quantity D is determined. Then, the set value generation processing unit 123 outputs the set value of the division size M of the input data amount D in the determined job 2 to the distributed processing base unit 130. The distributed processing infrastructure unit 130 that has received this setting value executes job 2 using the setting value.

  Based on the job execution instruction output from the interface providing unit 110 and the setting value output from the setting value generation unit 120, the distributed processing base unit 130 sends to each task execution unit 132 under the management of the distributed processing base unit 130. Share job execution processing. Then, the distributed processing infrastructure unit 130 returns data that summarizes the execution results of the task execution units 132 that have assigned the job to the interface providing unit 110. The distributed processing base unit 130 is realized by, for example, MapReduce described in Non-Patent Document 1. The distributed processing base unit 130 includes a job execution management unit 131 and one or more task execution units 132.

  The job execution management unit 131 manages how the job execution unit 132 executes the job as a task based on the job execution instruction and the set value of the job. For example, the job execution management unit 131 arranges tasks so that the task execution unit 132 processes data held in the local disk as much as possible. Then, the job execution management unit 131 returns to the interface providing unit 110 data that summarizes the execution results of the task execution units 132 that have assigned the job.

  When the task execution unit 132 receives a task or setting value from the job execution management unit 131, the task execution unit 132 reads the corresponding input data by local disk or network transfer, and performs a task execution process based on the setting value.

[Processing procedure]
Next, the processing procedure of the distributed processing apparatus 10 will be described with reference to FIG. First, when receiving an input of request processing to the distributed processing apparatus 10 (S1), the interface providing unit 110 interprets the content of the request processing by the conversion unit 111, and converts the content of the request processing into the distributed processing base unit. 130 converts the job command into a processable command (S2). For example, as shown in FIG.

  The runtime information acquisition unit 121 acquires job runtime information from the distributed processing infrastructure unit 130 (S3). For example, the runtime information acquisition unit 121 acquires the input data amount D from the distributed processing infrastructure unit 130 to the job 1 and the value of the number N of task execution units 132 that execute the job 1. Then, the set value generation processing unit 123 generates a set value for the job (for example, job 1) with reference to the execution time information and the set value generation model 122 (S4). Then, the set value generation processing unit 123 outputs the generated set value to the distributed processing base unit 130.

  The execution unit 112 executes the job by the distributed processing base unit 130 based on the instruction converted by the conversion unit 111 (S5). For example, the execution unit 112 causes the distributed processing base unit 130 to execute job 1 based on the setting value of job 1.

  If there is an unexecuted job after S5 (Yes in S6), the process returns to S3. For example, in the instruction converted by the conversion unit 111, if execution of job 2 is instructed following job 1, the process returns to S3. The execution time information acquisition unit 121 then distributes the input data amount D to the job 2 and the number N of task execution units 132 that execute the job 2 when the execution of the job 2 starts after the execution of the job 1. Obtained from the unit 130 (S3). Then, the setting value generation processing unit 123 refers to the input data amount D to the job 2, the number N of task execution units 132 that execute the job 2, and the setting value generation model 122 to determine the setting value of the job 2. Generate (S4). Thereafter, the execution unit 112 executes the job 2 by the distributed processing infrastructure unit 130 using the generated setting value of the job 2 (S5).

  After S5, if there is no unexecuted job in S6 (No in S6), the execution unit 112 outputs the job execution result (S7) and ends the process.

  By doing so, the distributed processing device 10 can give optimum setting values for each job to the request processing, and thus the performance of the distributed processing device 10 can be sufficiently exerted. Further, the user of the distributed processing apparatus 10 does not need to adjust the setting value for each job.

[Experimental result]
FIG. 5 shows the relationship between the division size M of the input data amount D to the job in the distributed processing apparatus 10 and the measured value of the execution time of the job. Here, the number N of task execution units 132 used for job execution is set to 10, and the amount of input data D is set to 18 (GB). As shown in FIG. 5, when the division size M of the input data amount D is D / N and when it is D / 2N, the job execution time is the shortest. I understand. From this, it can be seen that the above-described expression (1) is an expression suitable for calculating the optimum value of the division size M of the input data amount D.

  Actual input data to the distributed processing apparatus 10 is distributed and held in a plurality of computers (task execution unit 132). This affects when determining the amount of data to be processed by one task. When an instruction is given to process data larger than the amount of data held by one computer in one task, it is necessary to receive data from another computer, which adds to the cost of transfer. In order to avoid the cost, it is rare that the data is regarded as one data across the computers. Actually, in the implementation of the prior art, the user cannot specify the amount of data to be processed by one task for the distributed processing system, but can specify only the maximum amount (referred to as the division size M). The job execution management unit 131 divides data (that is, the number of tasks increases) for data that exceeds the division size. Here, in order to avoid the transfer cost, the remaining data less than the division size M constitutes one piece of divided data and is executed as a task. That is, the division size M and the amount of data processed by the actual task do not necessarily match. For example, when the division size M is “6” and a computer holds a data amount of “10”, a task for processing the division data “6” and a task for processing the division data “4” can be performed. . As described above, since strict equal allocation cannot be performed, it is necessary to search for a setting value that minimizes the execution time in an experiment. As a result of the experiment, it is understood that if the set value in the case where the input data amount D is divided by the division size M obtained by the above equation (1) is adopted, it works well in the environment of this experiment. 1) is adopted as a model.

[Other embodiments]
(Example of set value generation model 1)
Note that the set value generation processing unit 123 may generate an allocated memory amount for the job as the set value of the job. For example, the set value generation processing unit 123 may generate an optimal memory amount used when processing (for example, sorting) the intermediate output in the job, an optimal memory amount used when transferring the intermediate output, or the like. Good. The optimum memory amount here is, for example, a memory amount that makes the execution time of the job as short as possible.

In this case, as the set value generation model 122, a function f for obtaining an optimum memory amount used when processing the intermediate output of the job or an optimum memory amount used when transferring the intermediate output is used. The parameters of the function f include, for example, the amount of input data D to the job and network transfer per unit time in a network connecting each task execution unit 132 (that is, a computer that performs data processing by the task execution unit 132). This is the speed v _t or the maximum memory amount that can be used by each task execution unit 132. Note that the value acquired by the runtime information acquisition unit 121 is used as the input data amount D to the job. In addition, the network transfer rate v _t and the value of the maximum memory amount that can be used by each task execution unit 132 are assumed to be acquired in advance from the distributed processing platform unit 130 or the like.

  As described above, the distributed processing apparatus 10 gives the optimum memory amount used when processing the intermediate output in the job, the optimum memory amount used when transferring the intermediate output, and the like as the setting value of the job. As a result, it is avoided that the amount of memory for processing the intermediate output in the job is too large, so that the amount of memory for transferring the intermediate output after processing becomes insufficient and the execution time of the job becomes long. be able to. Further, since the amount of memory for processing the intermediate output in the job is too small, it can be avoided that the processing itself takes time and the execution time of the job becomes long.

(Example of setting value generation model 2)
Further, when transferring the intermediate output data of the job, the set value generation processing unit 123 generates a value indicating whether to transfer the intermediate output data after compressing the intermediate output data. Also good.

  In this case, the setting value generation model 122 used by the setting value generation processing unit 123 calculates, for example, a transfer time when the intermediate output data of the job is compressed (including the decompression time) and a transfer time when the compression is not performed. However, this is a model that employs the shorter transfer time (whether or not to compress the intermediate output data). That is, the setting value generation model 122 determines that the intermediate output data is compressed when the transfer time when the intermediate output data of the job is compressed is shorter, but the transfer time when the compression is not performed is shorter. The intermediate output data is determined not to be compressed in the job.

  In this case, the set value generation processing unit 123 uses the input data amount D to the job and the user function f used in the job as execution time information. The set value generation model 122 used by the set value generation processing unit 123 is a data transfer time of intermediate output data that is not compressed when transferring intermediate output data obtained by executing the process by the user function f for the input data amount D. The length of a certain uncompressed transfer time and the compression transfer time that is the sum of the time required for compression and decompression of the intermediate output data when the intermediate output data is compressed and the intermediate output data transfer time after compression. In comparison, it is a model that determines that compression is not performed when the non-compression transfer time is short with respect to the compression transfer time, and compression is performed when the non-compression transfer time is long. Further, the setting value of the job generated by the setting value generation processing unit 123 is a value determined by the setting value generation model 122 and indicating whether or not intermediate output data is compressed in the job.

  The transfer time when not compressed is estimated by, for example, the following equation (2).

In Expression (2), v _t is a network transfer rate, and g (D, f) is an output data amount (that is, an intermediate output data amount) when the input data amount D is processed by the user function f.

  The transfer time in the case of compression is estimated by the following formula (3), for example.

As described above, the input data amount D and the user function f to the job in Expression (2) and Expression (3) are acquired by the runtime information acquisition unit 121. The network transfer rate v _t , the data compression rate v _c , and the data decompression rate v _d are values acquired in advance from the distributed processing platform 130 and the like.

  In this way, the distributed processing apparatus 10 sets whether or not to compress the intermediate output in the job as the job setting value, so that the execution time of the job can be shortened as much as possible.

(Model update part)
Note that the set value generation unit 120 may acquire job statistical information based on the generated set value and correct (update) the set value generation model 122 based on the statistical information. Such a set value generation unit 120 further includes a model update unit 124 indicated by a broken line in FIG. The runtime information acquisition unit 121 acquires statistical information of the job as the runtime information of the job. The statistical information acquired here is the execution time of the job.

  The model update unit 124 acquires statistical information of each job by the runtime information acquisition unit 121 and the like, and acquires setting values set for the job from the setting value generation processing unit 123 and the like. Then, the model update unit 124 corrects the function f of the setting value generation model 122 using the acquired statistical information of the job and the setting value. Thereafter, the setting value generation processing unit 123 generates a setting value for the job using the corrected function f of the setting value generation model 122 and outputs the job setting value to the distributed processing platform unit 130. Then, the model update unit 124 further modifies the function f of the setting value generation model 122 using the setting value based on the corrected setting value generation model 122 and the job statistical information based on the setting value. By repeating such processing, the model update unit 124 can make the setting value generation model 122 a model that calculates an appropriate setting value.

(Set value generator)
In the above-described embodiment, the setting value generation unit 120 is configured in the distributed processing apparatus 10, but is not limited thereto. For example, the setting value generation unit 120 is realized as a device separate from the distributed processing device 10, and the setting value generation unit 120 acquires execution time information of each job from the distributed processing device 10 via the network, and the distributed processing device The setting value of each job may be output to 10.

(Example of job settings)
The setting value of the job in the above-described embodiment is not limited to the maximum data amount M per task constituting the job, which requires that the execution time of the job is minimized by making maximum use of resources. The following setting values may be used. For example, when a plurality of request processes (queries or the like) are input in the distributed processing device 10, the task execution unit 132 may execute a plurality of jobs at the same time. In such a case, as the setting value of the job, each request process completes the process within the time set by the user, and from the start of the process based on the request process to the completion of the execution of all the request processes In order to shorten the time as much as possible, it is preferable to minimize consumption resources such as a CPU and a memory amount used by each task execution unit 132. Therefore, as the setting value of the job in such a case, the number of task execution units 132 that execute a task related to the execution of the job, the amount of memory allocated to the task, the number of tasks executed by the task execution unit 132, and the like are considered. It is done. Furthermore, as a setting value of a job when it is required to minimize the total power consumption of each task execution unit 132, for example, the number of task execution units 132 that are calculation resources can be considered.

(program)
The processing executed by the distributed processing apparatus 10 can also be realized by a program (distributed processing program) described in a language that can be executed by a computer. In this case, when the computer executes the program, the same effects as those of the above-described embodiment can be obtained. Further, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer and executed to execute the same processing as in the above-described embodiment. Hereinafter, an example of a computer that executes a program that realizes the same function as the distributed processing apparatus 10 will be described.

  FIG. 6 is a diagram illustrating a computer 1000 that executes a program. As illustrated in FIG. 6, the computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, and a network interface 1070, and these units are connected by a bus 1080. The

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 as illustrated in FIG. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031 as illustrated in FIG. The disk drive interface 1040 is connected to the disk drive 1041 as illustrated in FIG. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041.

  Here, as illustrated in FIG. 6, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above program is stored in, for example, the hard disk drive 1031 as a program module in which a command to be executed by the computer 1000 is described.

  The various data described in the above embodiment is stored as program data, for example, in the memory 1010 or the hard disk drive 1031. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1031 to the RAM 1012 as necessary, and the interface providing unit 110, the distributed processing base unit 130, and the runtime information acquisition unit 121 described above. The functions of the set value generation processing unit 123 and the model update unit 124 are realized.

  Note that the program module 1093 and program data 1094 related to the program are not limited to being stored in the hard disk drive 1031, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1041 or the like. Good. Alternatively, the program module 1093 and the program data 1094 related to the program may be stored in another computer connected via a network and read by the CPU 1020 via the network interface 1070.

DESCRIPTION OF SYMBOLS 10 Distributed processing apparatus 40 Node 110 Interface provision part 111 Conversion part 112 Execution part 120 Setting value generation part 121 Runtime information acquisition part 122 Setting value generation model 123 Setting value generation processing part 124 Model update part 130 Distributed processing base part 132 Task execution Part

Claims (8)

A distributed processing base unit that executes one or more jobs using one or more nodes, and when the request processing input to the distributed processing base unit is received, the content of the request processing is processed by the distributed processing base unit Each of the jobs to be executed in a distributed processing device that includes an interface providing unit that converts the command into a possible job command and executes the one or more jobs using the distributed processing platform unit based on the converted command A setting value generation device for generating a setting value of
An execution time information acquisition unit for acquiring execution time information of the job from the distributed processing infrastructure unit at the start of execution of the job;
A storage unit that stores a setting value generation model for obtaining a setting value of the job based on the execution time information of the job;
A setting value generation processing unit configured to generate a setting value used for execution of the job by the setting value generation model using the execution time information of the job and to output the setting value to the distributed processing base unit; Setting value generator.   The set value generation apparatus according to claim 1, wherein a model that optimizes the set value is used as one of the set value generation models as a requirement for shortening the execution time of the job. The runtime information is the amount of input data to the job,
The set value generation model is a model for obtaining, as a set value, a division size that is a maximum data amount processed by one task with respect to the input data amount in all task processes for executing the job,
The setting value generation apparatus according to claim 1, wherein the setting value of the job to be generated is the division size obtained by the setting value generation model. The runtime information is an input data amount to the job and a user function used in the job,
The set value generation model includes a non-compressed transfer time that is a data transfer time of the intermediate output data that is not compressed when transferring intermediate output data obtained by executing processing by the user function for the input data amount, and the intermediate The compression transfer time is compared between the time required for compression and decompression processing of the intermediate output data when the output data is compressed and the transfer time during compression that is the total value of the transfer time of the intermediate output data after compression. On the other hand, when the uncompressed transfer time is short, the model is determined not to compress, and when the uncompressed transfer time is long, the model is determined to compress.
The setting value of the job to be generated is a value determined by the model and indicating whether or not intermediate output data is to be compressed in the job. Setting value generator. The runtime information acquisition unit further includes:
Obtain statistical information of the job executed by the generated setting value,
The set value generation device further includes:
5. The model update unit according to claim 1, further comprising a model update unit configured to update the set value generation model with reference to the generated set value and statistical information of the job. Setting value generator. A distributed processing platform that executes one or more jobs using one or more nodes;
When receiving an input of request processing, the content of the request processing is converted into a job command that can be processed by the distributed processing platform, and based on the converted command, the distributed processing platform is used, An interface providing unit for executing one or more jobs;
An execution time information acquisition unit for acquiring execution time information of the job from the distributed processing infrastructure unit at the start of execution of the job;
A storage unit that stores a setting value generation model for obtaining a setting value of the job based on the execution time information of the job;
With reference to the acquired job execution time information and the setting value generation model, a setting value used for execution of the job is generated, and a setting value generation processing unit that outputs the setting value to the distributed processing infrastructure unit,
The distributed processing platform is
The distributed processing apparatus, wherein the job is executed using the output setting value of the job. A distributed processing base unit that executes one or more jobs using one or more nodes, and when the request processing input to the distributed processing base unit is received, the content of the request processing is processed by the distributed processing base unit Each of the jobs to be executed in a distributed processing device that includes an interface providing unit that converts the command into a possible job command and executes the one or more jobs using the distributed processing platform unit based on the converted command A setting value generation method for generating a setting value of
Based on the execution time information of the job, a computer that stores a setting value generation model for obtaining the setting value of the job,
Obtaining execution time information of the job from the distributed processing platform when starting execution of the job;
Referring to the acquired execution time information of the job and the setting value generation model, a setting value used for execution of the job is generated, and a setting value generation processing step is output to the distributed processing platform A setting value generation method characterized by the above. A distributed processing base unit that executes one or more jobs using one or more nodes, and when the request processing input to the distributed processing base unit is received, the content of the request processing is processed by the distributed processing base unit Each of the jobs to be executed in a distributed processing device that includes an interface providing unit that converts the command into a possible job command and executes the one or more jobs using the distributed processing platform unit based on the converted command A setting value generation program for generating a setting value of
Based on the execution time information of the job, a computer that stores a setting value generation model for obtaining the setting value of the job,
Obtaining execution time information of the job from the distributed processing platform when starting execution of the job;
Referring to the execution time information of the acquired job and the setting value generation model, a setting value used for execution of the job is generated, and a setting value generation processing step that is output to the distributed processing platform is executed. Setting value generation program.

Images