JP2022006909A - Control device, information processing device, information processing control method, and computer program - Google Patents

Abstract

To allow a stream process to be conducted in considering a delay factor and to achieve improvement of resource usage efficiency.SOLUTION: A control device includes an intra-group scheduler which schedules a job of each application on the basis of a delay requirement of each application for each group constituted with one or a plurality of applications, and an inter-group scheduler which controls a resource quantity to be used for execution of the job for each of the groups on the basis of observance of the delay requirement being a level indicating whether the delay requirement of the application is satisfied in the group.SELECTED DRAWING: Figure 1

Description

The present invention relates to a control device, an information processing device, an information processing control method, and a computer program.

Conventionally, stream processing is known in which data processing is sequentially executed by a predetermined application when data is generated. According to the stream processing, it is possible to perform processing such as aggregation and analysis in real time on the sequentially generated data. As an information processing technique for executing stream processing, for example, the techniques described in Patent Document 1 and Non-Patent Document 1 are known. In the prior art described in Patent Document 1, the switch executes the primary processing of data and transfers the execution result to the stream cluster. The switch performs stream processing based on the descriptor received from the server.
Non-Patent Document 1 describes a distributed stream processing system including a "Worker Node" that executes stream processing by an application, a "Driver Program" that schedules jobs of an application, and a "Cluster Manager" that manages resources. Has been done.

Japanese Patent No. 5802215

Apache Spark, “Cluster Mode Overview”, Internet <URL: https://spark.apache.org/docs/latest/cluster-overview.html>

However, in the prior art described in Patent Document 1 described above, stream processing cannot be executed in consideration of the delay requirement required by the application. In the prior art described in Non-Patent Document 2, resources are fixedly allocated to the "Worker Node". Therefore, when the amount of data to be streamed per unit time fluctuates depending on the time zone, per unit time. It is necessary to fixedly allocate the amount of resources according to the maximum amount of generation. Therefore, surplus resources are generated in the time zone when the amount of data to be streamed per unit time is small, and the resource utilization efficiency is lowered.

The present invention has been made in consideration of such circumstances, and an object of the present invention is to execute stream processing in consideration of a delay requirement for stream processing and to improve resource utilization efficiency.

(1) One aspect of the present invention is an in-group scheduler that schedules a job of each application based on the delay requirement of each application for each group composed of one or a plurality of applications, and an application delay in each of the above groups. It is a control device including an intergroup scheduler that controls the amount of resources of each of the above groups that can be used to execute a job based on the degree of compliance with the delay requirement of the application, which is a degree indicating whether or not the requirement is satisfied. ..
(2) In one aspect of the present invention, the scheduler in the group indicates whether or not the job for each job is proceeding as planned based on the delay requirement of each application. Is the control device of the above (1), which calculates the priority of each job based on the calculated progress schedule achievement degree and the required resource amount of each job.
(3) In one aspect of the present invention, the scheduler in the group determines the priority of the job by weighting the achievement level of the progress schedule of the job and the required resource amount of the job. It is a device.
(4) In one aspect of the present invention, the in-group scheduler controls each weighting coefficient for the degree of progress schedule achievement of the application and the required resource amount so that the degree of compliance with the delay requirement of the application is improved. This is the control device of (3) above.
(5) In one aspect of the present invention, the degree of compliance with the delay requirement of the application is calculated based on the difference between the sum of the job execution time of the application and the record read delay time and the processing delay time required by the application. The control device according to any one of (1) to (4) above.

(6) One aspect of the present invention comprises an execution unit provided for each group composed of one or a plurality of applications to execute a job, and a control device according to any one of (1) to (5) above. The execution unit is an information processing device that executes a job of an application of its own group according to a schedule by the control device.

(7) One aspect of the present invention is an intra-group scheduling step in which an intra-group scheduler schedules jobs for each application based on the delay requirements of each application for each group composed of one or a plurality of applications, and a group. The inter-scheduler controls the amount of resources in each of the groups available to run the job, based on the degree of compliance with the application's delay requirements, which is the degree to which the application's delay requirements are met in each of the groups. It is an information processing control method including an intergroup scheduling step.

(8) One aspect of the present invention includes an intra-group scheduling step for scheduling a job of each application based on the delay requirement of each application for each group composed of one or a plurality of applications on a computer, and each of the above-mentioned groups. An intergroup scheduling step that controls the amount of resources available to each said group to execute a job, based on the degree of compliance with the application's delay requirements, which is the degree to which the application's delay requirements are met. It is a computer program to be executed.

According to the present invention, it is possible to obtain the effect that the stream processing can be executed in consideration of the delay requirement for the stream processing and the resource utilization efficiency can be improved.

It is a figure which shows the structural example of the information processing apparatus which concerns on one Embodiment. It is a flowchart which shows the example of the whole procedure of the information processing control method which concerns on one Embodiment. It is a flowchart which shows the example of the procedure of the group scheduling which concerns on one Embodiment. It is a flowchart which shows the example of the procedure of the inter-group scheduling which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a configuration example of an information processing apparatus according to an embodiment. In FIG. 1, the information processing device 1 includes a control device 10 and processing nodes wnd1 and wnd2. The control device 10 controls information processing for the processing nodes wnd1 and wnd2 that execute stream processing by the application.

The processing nodes wnd1 and wnd2 include execution units excc1 and excc2 that execute jobs issued by the application. Each execution unit excc1 and excc2 are provided for each group of applications. A group of applications consists of one or more applications. The execution unit exc1 executes a job issued by the applications ap1, ap2, and app3 belonging to the group A. The execution unit exc2 executes a job issued by the application application 4 belonging to the group B. The upper limit of the information processing resource that can be used by each execution unit excc1 and exc2 is controlled by the control device 10 for each processing node wnd1 and wnd2. The execution units excc1 and excc2 are constructed using, for example, a container-type virtual execution environment.

Hereinafter, information processing resources may be simply referred to as resources. Further, when the processing nodes wnd1 and wnd2 are not particularly distinguished, they are referred to as processing nodes wnd. Further, when the execution units exc1 and exc2 are not particularly distinguished, they are referred to as an execution unit exc. Further, when groups A and B are not particularly distinguished, they are referred to as groups. Further, when the applications app1, app2, app3, and app4 are not particularly distinguished, they are referred to as application app.

In this embodiment, each application application issues a job so that only one job is executed at a time. Therefore, in each application application, the next job is not issued until the previous job is completed. As a result, the execution unit exc executes only one job at a time for one application application.

The control device 10 includes an intra-group scheduler (Intra-group Scheduler) 11, an inter-group scheduler (Inter-group Scheduler) 12, an application management unit (Application Manager) 13, and a metric collection unit (Metrics Collector) 14. ..

The function of the control device 10 is realized by the control device 10 including a computer hardware such as a CPU (Central Processing Unit) and a memory, and the CPU executes a computer program stored in the memory. The control device 10 may be configured by using a general-purpose computer device, or may be configured as a dedicated hardware device. For example, the control device 10 may be configured by using a server computer connected to a communication network such as the Internet. Further, each function of the control device 10 may be realized by cloud computing.

The in-group scheduler 11 schedules jobs for each application application in the group for each processing node wnd. The in-group scheduler 11 schedules a job for each application application for each group based on the delay requirement of each application application. The execution unit exc executes the job of each application application according to the scheduling of the scheduler 11 in the group. The delay requirement (Service Level Objective: SLO) of each application application is set in the control device 10 in advance.

The inter-group scheduler 12 allocates resources between groups for each processing node wnd. As resources, there are CPU, memory, storage, network, etc. used by the processing node wnd. The inter-group scheduler 12 controls the amount of resources in each group that can be used to execute the job, based on the degree of compliance with the delay requirement of the application application in each group. The degree of compliance is a degree indicating whether or not the delay requirement of the application application is satisfied.

The inter-group scheduler 12 constantly monitors whether or not the delay requirement of the application application is satisfied in each group. Based on the monitoring result, the inter-group scheduler 12 reduces the amount of resources that can be used by a group that sufficiently meets the delay requirement of the application application, while it can be used by a group that does not meet the delay requirement of the application application. Periodically execute control to increase the amount of resources.

The application management unit 13 has an interface function for executing or stopping a job issued by an application application and checking the status.

The metric collection unit 14 monitors the status of the application application. The metric collection unit 14 collects information from each processing node wnd for determining whether or not the delay requirement of each application application is satisfied.

The overall procedure of the information processing control method according to the present embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing an example of the overall procedure of the information processing control method according to the present embodiment. Here, the overall procedure of the information processing control method will be described focusing on one processing node wnd, but the same procedure is performed for each of a plurality of processing nodes wnd.

(Step S1) The inter-group scheduler 12 performs the first inter-group schedule. In the initial inter-group schedule, resources are allocated to each group by default. For this initial setting, a default value or an arbitrary setting value by the user may be used, or a value based on the actual allocation of resources between past groups may be used. The initial inter-group schedule sets the initial value of the upper limit of the amount of resources that can be used for the execution unit exc corresponding to each group.

(Step S2) The intra-group scheduler 11 performs intra-group scheduling. In intra-group scheduling, the intra-group scheduler 11 schedules jobs for each application application for each group based on the delay requirements of each application application. The details of intra-group scheduling will be described later.

(Step S3) The inter-group scheduler 12 calculates the degree of compliance with the delay requirement of the application application in each group. The method of calculating the degree of compliance will be described later.

(Step S4) The inter-group scheduler 12 determines whether or not rescheduling between groups is necessary based on the degree of compliance with the delay requirement of the application application in each group. As a result of this determination, if reschedule between groups is necessary (step S4, YES), the process proceeds to step S5, and if not (step S4, NO), the process returns to step S2.

(Step S5) The inter-group scheduler 12 performs inter-group scheduling. In this intergroup scheduling, the intergroup scheduler 12 redistributes resources between groups based on the degree of compliance with the delay requirements of the application application in each group. The details of inter-group scheduling will be described later.

(Step S6) When the execution of the stream processing at the processing node wnd is completed (step S6, YES), the processing of FIG. 2 is terminated. On the other hand, when the execution of the stream processing in the processing node wnd is continued (step S6, NO), the process returns to step S2.

Next, the procedure of intra-group scheduling according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing an example of the in-group scheduling procedure according to the present embodiment.

(Step S11) The in-group scheduler 11 calculates the required resource amount rJ of the job of each application application for each group. The required resource amount rJ is calculated for each application application and for each job.

Although the intra-group scheduling (step S2 in FIG. 2) is executed periodically, the required resource amount rJ of the job is used until the initially calculated value is completed. Therefore, the in-group scheduler 11 holds the required resource amount rJ initially calculated for each job, and uses the held required resource amount rJ without changing until the job is completed.

(Calculation method for required resource amount)
Here, a method for calculating the required resource amount according to the present embodiment will be described. In order to satisfy the delay requirement (SLO) of the application application, it is necessary to satisfy the following equation.
SLO ≧ J + Rd
However, SLO is the processing delay time required by the application application. J is the time required to execute the job (job execution time). Rd is the time required to read the record required for the job (record read delay time).

As an example of the record read delay time Rd measurement method, the time stamp of the storage system in which the record is written is used, the difference between the time when the record is written and the time when the record is read is calculated, and the difference is recorded. Used for read delay time Rd. As another example of the record read delay time Rd measurement method, a probe (dummy data) is written to the storage system in which the record is written, the time until the probe is read is measured, and the time of the measurement result is the record read delay. Used for time Rd. The in-group scheduler 11 uses the record read delay time Rd measured by the method for measuring the record read delay time Rd.

The in-group scheduler 11 uses the known record read delay time Rd, and determines the required resource amount rJ of the job so as to satisfy "SLO-Rd ≧ J". Here, the required resource amount rJ is represented by a ratio (the following equation) to the total resource amount of the group (the upper limit of the resources that can be used by the execution unit exc).
rJ = (Amount of resources used by the job) ÷ (Amount of resources in the group)

When a plurality of types (CPU, memory, storage, network, etc.) exist as resources, the required resource amount rJ is represented by an N-dimensional vector corresponding to the resource types (N types). In the following description, a case where the required resource amount rJ is a one-dimensional value will be described.

There is a relationship expressed by the following relational expression between the job execution time J, the required resource amount rJ, and the job input data amount IJ.
rJ = f (J, IJ)
However, the job input data amount IJ is the amount of data used by the job.

As for the method of acquiring the job input data amount IJ, when the job input data amount IJ can be measured, the scheduler 11 in the group uses the measured value of the job input data amount IJ. On the other hand, when it is difficult to measure the job input data amount IJ, the in-group scheduler 11 estimates the job input data amount IJ from the latest data amount in the past for the data used by the job. The in-group scheduler 11 uses the job input data amount IJ (measured value or estimated value) acquired by the method for acquiring the job input data amount IJ.

The derivation method of f (J, IJ) is based on the information of the job execution time J, the required resource amount rJ, and the job input data amount IJ collected by the metric collection unit 14, for example, by linear regression analysis f (J, IJ). Ask for. Generally, there is a proportional relationship between the job input data amount IJ and the job execution time J. Further, there is an inverse proportional relationship between the job execution time J and the required resource amount rJ. From this, linear regression analysis can be mentioned as a method for deriving f (J, IJ). In addition, f (J, IJ) may be obtained by using a machine learning algorithm such as Support Vector Machine (SVM) or Deep Learning. The in-group scheduler 11 uses f (J, IJ) obtained by the method of deriving f (J, IJ).

The in-group scheduler 11 uses the known record read delay time Rd, job input data amount IJ, and the relational expression “rJ = f (J, IJ)” to satisfy “SLO-Rd ≧ J”. The required resource amount rJ of is determined.

(Step S12) The in-group scheduler 11 calculates the progress schedule achievement degree “Pr / Ptr” of each application application job for each group. Progress Schedule Achievement degree "Pr / Ptr" is a degree indicating whether or not the job is proceeding as planned. The progress schedule achievement degree "Pr / Ptr" is calculated by the following equation.
"Pr / Ptr" = "Job progress Pr" ÷ "Elapsed time ratio Ptr"
Elapsed time ratio Ptr = (t-Ts) ÷ (Tdj-Ts)
However, t is the current time. Ts is the time when the job was issued. Tdj is the scheduled end time (deadline) of the job.

The job progress Pr is measured by the application management unit 13. “0 ≦ Pr ≦ 1”. The time Ts when the job is issued is acquired by the application management unit 13. The scheduled end time Tdj of the job is calculated by the following equation based on the time Ts at which the job was issued, the processing delay time SLO requested by the application application, and the record read delay time Rd.
Tdj = Ts + SLO-Rd

When the progress schedule achievement level "Pr / Ptr" is 1 or more, the job is proceeding as scheduled or faster than planned. On the other hand, when the progress schedule achievement degree "Pr / Ptr" is less than 1, the job is behind schedule.

(Step S13) The in-group scheduler 11 calculates the job priority P of each application application for each group. The priority P is calculated by the following equation.
P = α (1 ÷ (Pr / Ptr)) + (1-α) × rJ
However, α is a weighting coefficient. “0 ≦ α ≦ 1”. The weighting coefficient α of each group can be arbitrarily set.

The priority P is calculated based on the job progress schedule achievement degree "Pr / Ptr" and the required resource amount rJ (when 0 <α <1). The priority P becomes higher as the job progress schedule achievement degree "Pr / Ptr" is smaller (when 0 <α). The priority P becomes higher as the required resource amount rJ is larger (when α <1).

(Method of determining the weighting coefficient α)
A static determination method and a dynamic determination method will be described for determining the weighting coefficient α.

(1) Static determination method The weighting coefficient α of each group can be set arbitrarily, but the weighting coefficient α of each group is statically set by the static determination method of Example 1-1 and Example 1-2 shown below. It may be decided.

(Example 1-1) When it is known that the required resource amount rJ of all application applications in the group has almost the same value, the weight (α) of the progress schedule achievement degree “Pr / Ptr” is increased.

(Example 1-2) If one of the progress schedule achievement degree "Pr / Ptr" and the required resource amount rJ is inaccurate, the weight of the other is increased.
For example, when the job execution time J is a long time exceeding several minutes, there is a high possibility that the required resource amount rJ is incorrect, so the weight (α) of the progress schedule achievement degree “Pr / Ptr” is increased.
For example, when there is a job composed of a plurality of task sets (TaskSet) in the jobs of the application application in the group (for example, in the case of a multi-stage configuration in which the calculation result by the first task set is used in the second task set). ), Since the progress degree Pr of the job is not proportional to the elapsed time ratio Ptr, the progress schedule achievement degree "Pr / Ptr" is not accurate. In this case, the weight (1-α) of the required resource amount rJ is increased.

(2) Dynamic determination method The weighting coefficient α of each group can be arbitrarily set, but the weighting coefficient α of each group may be dynamically determined by the dynamic determination method shown below.

In the dynamic determination method, the weighting coefficient α suitable for each group is dynamically determined for each group. An example of a dynamic determination method will be described. In advance, the weighting coefficient α is defined as a discrete value, for example, “α ∈ [0.0, 0.1, 0.2, ..., 0.9, 1.0]”. Next, in the actual scheduling in the group, when each weighting coefficient α is used, the degree of compliance Ga of the delay requirement of the application application in the group is calculated. As a result, the degree of compliance Ga at each weighting coefficient α can be obtained for each group. The initial value of the weighting coefficient α of each group is preset by the default value or the static determination method described above.

The method of calculating the degree of compliance Ga of the delay requirement of the application application in the group is to calculate the average value of "J + Rd-SLO" for each application application using the sliding window, and calculate the "J + Rd-SLO" of each application application. The average value obtained by averaging the average value within the group is set as the degree of compliance Ga. In this compliance example, the larger the positive value of the compliance Ga, the less the application delay requirement is satisfied. In step S3 of FIG. 2 described above, the compliance degree Ga of the delay requirement of the application application is calculated for each group by the calculation method of the compliance degree Ga.

When the compliance degree Ga of the group is a positive value for a certain weighting coefficient α, the group is in a state where the delay requirement of the application cannot be satisfied by the weighting coefficient α. Therefore, by searching for the weighting coefficient α in which the value of the compliance degree Ga of the group tends to be small (the degree of compliance is likely to be improved), the weighting coefficient α most suitable for the group is searched for. For example, a Markov decision process can be used for the algorithm for determining the optimum weighting coefficient α.

(Step S14) The in-group scheduler 11 schedules each job based on the job priority P of each application application for each group. In this job scheduling, the group scheduler 11 calculates the scheduling weight W of each scheduling target job for each group. The scheduled job is a job waiting to be executed that is registered in the execution waiting queue. The scheduling weight W is calculated by the following equation.
W = P_w ÷ (ΣP)
However, P_w is the priority P of the scheduling target job for which the scheduling weight is calculated. ΣP is the sum of the priority Ps of all the pending jobs registered in the pending queue.

The in-group scheduler 11 selects jobs to be executed in order from the scheduling target job having the largest scheduling weight W based on the scheduling weight W of each scheduling target job for each group. As a result, among all the jobs waiting to be executed in the group, the jobs having the larger scheduling weight W are executed in order.

In the present embodiment, the intra-group scheduling (step S2 in FIG. 2) is periodically executed. Therefore, the scheduling weight W of the job waiting to be executed in the group is updated every time the scheduling in the group (step S2 in FIG. 2) is executed. This makes it possible to schedule jobs according to the latest job execution status in the group. For example, the required resource amount rJ of a job is fixed until the job is completed, but even if the required resource amount rJ of the job is properly calculated, the job input data amount IJ suddenly becomes excessive. May make it impossible to meet the delay requirements of the application application. According to the present embodiment, by periodically executing the scheduling within the group, the job scheduling is performed so as to satisfy the delay requirement of the application application even for such an unexpected change in the job execution environment. It can be carried out.

In the above-mentioned procedure of intra-group scheduling, intra-group scheduling is performed for each job, but when the application application job is divided into multiple tasks (Tasks) and processed, intra-group scheduling is the task of the same processing. It may be performed in units of a task set (TaskSet) that summarizes the above.

Further, in the above description, the case where the target resource of the required resource amount rJ is one type and the required resource amount rJ is a one-dimensional value has been described, but the case where the target resource of the required resource amount rJ is a plurality of types is described below. Explain to. Here, a method 1 of treating a plurality of types of resource amounts as a multidimensional amount and a method 2 of treating a plurality of types of resource amounts independently will be described.

(Method 1: A method of treating multiple types of resource quantities as multidimensional quantities)
In method 1, the required resource amount rJ is represented by an N-dimensional vector (N is an integer of 2 or more). Required resource amount Each of the N elements of the vector possessed by the required resource amount rJ (N-dimensional vector rJ) is the required resource amount of each target resource of N types. For example, when the target resource of the required resource amount rJ is two types of CPU and memory, the required resource amount rJ is expressed as a two-dimensional vector “rJ = (CPU_rJ, Memory_rJ)”. CPU_rJ is the required resource amount of the CPU. Memory_rJ is the required resource amount of the memory. In addition, the required resource amount of each target resource is shown as a ratio to the total amount of each target resource of the group. For example, the required resource amount CPU_rJ of the CPU is expressed by the following equation.
CPU_rJ = (Amount of CPU resources used by the job) ÷ (Amount of CPU resources in the group)

Then, when calculating the priority P, the required resource amount rJ (N-dimensional vector rJ) is mapped to one dimension. In this mapping, if the mapping result is "rJ1", then "rJ1 = | rJ |. | RJ | is the magnitude of the N-dimensional vector rJ. Here," rJ1 "is the magnitude of the N-dimensional vector rJ. The reason is that the required resource amount rJ (N-dimensional vector rJ) has N elements of the vector (required resource amount of each target resource of N types) is shown as a ratio to the total amount of each target resource of the group. This is because they have the same scale regardless of the type of target resource.

As another method of mapping the required resource amount rJ (N-dimensional vector rJ) to one dimension, the mapping may be weighted for each of N types of target resources. For example, when "rJ = (a, b, c)", a function that returns "rJ1 = √ (2a + 3b + 1c) ^ 2" may be used.

The priority P is calculated by applying the result "rJ1" obtained by mapping the required resource amount rJ (N-dimensional vector rJ) to "rJ" in the above-mentioned calculation formula of the priority P.
P = α (1 ÷ (Pr / Ptr)) + (1-α) × rJ1
Further, the scheduling weight W is calculated by the same formula as described above.

(Method 2: A method of handling multiple types of resources independently)
In the method 2, the required resource amount rJ is calculated separately for each of N types of target resources. Then, for each of the N types of target resources, the priority P and the scheduling weight W are calculated using the required resource amount of each target resource.

For example, when the target resources of the required resource amount rJ are two types, the CPU and the memory, the CPU required resource amount CPU_rJ and the memory required resource amount Memory_rJ are calculated separately by the following equations, respectively.
CPU_rJ = (Amount of CPU resources used by the job) ÷ (Amount of CPU resources in the group)
Memory_rJ = (Amount of memory resources used by the job) ÷ (Amount of memory resources in the group)
Then, the priority P_CPU of the CPU and the priority P_Memory of the memory are calculated separately by the following equations.
P_CPU = α (1 ÷ (Pr / Ptr)) + (1-α) × CPU_rJ
P_Memory = α (1 ÷ (Pr / Ptr)) + (1-α) × Memory_rJ

Then, the scheduling weight W is calculated separately for the CPU and the memory by the same calculation formula as described above. As a result, the scheduling weight W_CPU when scheduling the resource amount of the CPU and the scheduling weight W_Memory when scheduling the resource amount of the memory can be obtained. The amount of CPU resources is allocated to each job based on the scheduling weight W_CPU. The amount of memory resources is allocated to each job based on the scheduling weight W_Memory.

Next, the procedure of intergroup scheduling according to the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the procedure of intergroup scheduling according to the present embodiment.

The inter-group scheduler 12 calculates the degree of compliance Ga of the delay requirement of the application application in each group in step S3 of FIG. 2 described above. As a result, the degree of compliance Ga can be obtained for each group. The degree of compliance Ga of a group is a degree indicating whether or not the group meets the delay requirement of the application application.

(Step S21) The inter-group scheduler 12 determines whether or not the compliance level Ga is good or bad for each group based on the compliance level Ga of each group.

When the compliance degree Ga of the group is a positive value (“Ga> 0”), the group does not meet the delay requirement of the application. Therefore, the inter-group scheduler 12 sets the group in which the compliance degree Ga is a positive value as the resource allocation target group.

On the other hand, when the compliance degree Ga of the group is "Ga <r (r is a predetermined negative value)", the delay requirement of the application is sufficiently satisfied and a sufficient amount of resources has already been allocated to the group. It is thought that there is. Therefore, the inter-group scheduler 12 sets a group in which the compliance degree Ga is "Ga <r (r is a predetermined negative value)" as the resource release target group.

(Step S22) The inter-group scheduler 12 redistributes resources for each group. In this resource reallocation, the resource release target group releases a certain amount of resources. Then, the resources released by the resource release target group are distributed to the resource allocation target group. This resource allocation method allocates by the ratio of "compliance degree Ga x allocation priority Pg" of the resource allocation target group. The allocation priority Pg is preset for each group.

If there is no group in a state where the delay requirement of the application is not satisfied (a group in which the degree of compliance Ga is a positive value), the above-mentioned step S22 is not executed. In this case, there is no need to reschedule between the groups, which corresponds to the above-mentioned determination result “NO” in step S4 of FIG. On the other hand, if there is a group in a state where the delay requirement of the application is not satisfied (a group in which the degree of compliance Ga is a positive value), reschedule between the groups is necessary, and the determination in step S4 of FIG. 2 described above is performed. The result corresponds to "YES".

Further, when there is a group in which the application delay requirement is not satisfied (a group in which the compliance degree Ga is a positive value), a group in which the application delay requirement is sufficiently satisfied (compliance degree Ga is "Ga <". When r (a group in which r is a predetermined negative value) ”does not exist, rescheduling between the groups is required. However, in step S22 described above, since the resource release target group does not exist, resource allocation is required. Since the resources for allocating to the target group are not released, resource reallocation to the resource allocation target group is not performed.

According to the above-described embodiment, the scheduler 11 in the group schedules the job of each application application based on the delay requirement of each application application for each group, so that the stream processing in consideration of the delay requirement for the stream processing by the application application is taken into consideration. Can be executed. Further, the inter-group scheduler 12 can improve the resource utilization efficiency by controlling the amount of resources of each group that can be used for job execution based on the degree of compliance Ga of the delay requirement of the application application in each group. can.

In addition, grouping of application applications can adjust the trade-off between independence and efficiency of execution of application applications. By grouping application applications, for example, even if the execution part exc of one group fails and the application application becomes infeasible, it does not affect the execution of the application application of other groups. , Independence of execution of application application between groups is obtained. Further, by dividing into a plurality of groups and scheduling for each group, even if the number of applications in a certain group increases, it is possible to efficiently deal with the scheduling in the group. Further, by dividing into a plurality of groups and reducing the number of applications in one group, it is possible to suppress interference between application applications in the group and perform efficient scheduling.

This section describes how to group applications. The grouping of applications can be arbitrarily set, but the grouping of applications may be determined by the grouping method shown below.

(1) Grouping method considering failures All applications belonging to the same group may become unexecutable at the same time if a failure occurs in the execution part of the group. Here, consider implementing a service that provides value by using multiple applications. If the same group is configured from multiple applications for providing multiple services unrelated to each other, if the execution part of the group fails, the multiple services may go down at the same time. Therefore, applications for providing the same service are grouped together. As a result, the influence of the failure of the execution unit on the service can be limited to a specific service. When grouping applications for providing the same service, the number of groups is equal to the number of services.

(2) Grouping method considering performance When the execution cycle of inter-group scheduling is sufficiently longer than the execution cycle of intra-group scheduling, the amount of job input data IJ suddenly increases and the amount of resources in the group becomes insufficient. However, it takes time for resources to be replenished by the next intergroup scheduling. This is not desirable from the point of view of the delay requirements of the application application.

In order to deal with this problem, applications whose job input data amount IJ repeatedly increases and decreases in a short cycle and whose timing tendency of increase and decrease of job input data amount IJ is the same are not included in the same group. As a result, even if applications whose job input data amount IJ repeatedly increases and decreases exist in the same group, the timing of increase and decrease of the job input data amount IJ is different, so that resource shortage is suppressed for the group as a whole. To.

It is also preferable to create groups so that the number of applications belonging to each group is as equal as possible. This is because when the number of applications belonging to a group is small, the margin for schedule adjustment between jobs generally decreases in scheduling within the group, so a group with a small number of applications is particularly vulnerable to fluctuations in the amount of job input data IJ. Because it will end up.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes design changes and the like within a range that does not deviate from the gist of the present invention.

Further, a computer program for realizing the functions of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The "computer system" here may include hardware such as an OS and peripheral devices.
The "computer-readable recording medium" is a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disc), and a built-in computer system. A storage device such as a hard disk.

Further, the "computer-readable recording medium" is a volatile memory inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line (for example, DRAM (Dynamic)). It also includes those that hold the program for a certain period of time, such as Random Access Memory)).
Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned function in combination with a program already recorded in the computer system.

1 ... Information processing device, 10 ... Control device, 11 ... In-group scheduler, 12 ... Intergroup scheduler, 13 ... Application management unit, 14 ... Metric collection unit, app1, app2, app3, app4 ... Application, excc1, excc2 ... Execution Part, wnd1, wnd2 ... Processing node

Claims (8)

An in-group scheduler that schedules jobs for each application based on the delay requirements of each application for each group consisting of one or more applications.
With an intergroup scheduler that controls the amount of resources available to each said group to run jobs based on the degree of compliance with the application's delay requirements, which is the degree to which each said group meets the application's delay requirements. ,
A control device equipped with. The in-group scheduler calculates the progress schedule achievement level and the required resource amount, which are the degrees indicating whether or not the job for each job is progressing as planned based on the delay requirement of each application, and the calculated progress of each job. Priority is given to each job based on the degree of achievement of the schedule and the amount of required resources.
The control device according to claim 1. The in-group scheduler determines the priority of a job by weighting the achievement level of the progress schedule of the job and weighting the required resource amount of the job.
The control device according to claim 2. The in-group scheduler controls each weighting factor for the job progress schedule achievement and required resource amount of the application so as to improve the compliance with the delay requirement of the application.
The control device according to claim 3. The degree of compliance with the delay requirement of the application is calculated based on the difference between the sum of the job execution time of the application and the record read delay time and the processing delay time required by the application.
The control device according to any one of claims 1 to 4. An execution unit that is provided for each group consisting of one or more applications and executes a job,
The control device according to any one of claims 1 to 5 is provided.
The execution unit executes the job of the application of its own group according to the scheduling by the control device.
Information processing equipment. An intra-group scheduling step in which the intra-group scheduler schedules jobs for each application based on the delay requirements of each application for each group consisting of one or more applications.
The intergroup scheduler controls the amount of resources in each of the groups available to run the job, based on the degree to which the application's delay requirements are met, which is the degree to which the intergroup scheduler meets the application's delay requirements in each of the groups. Intergroup scheduling steps and
Information processing control method including. On the computer
In-group scheduling steps that schedule jobs for each application based on the delay requirements of each application for each group consisting of one or more applications.
An intergroup scheduling step that controls the amount of resources available to each said group to run jobs based on the degree of compliance with the application's delay requirements, which is the degree to which each said group meets the application's delay requirements. When,
A computer program to run.

Images