JP2004192400A - Job scheduling method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and efficiently operate a multiprocessor system by dynamically assigning processors again. <P>SOLUTION: A management process 2 issues a request for job cluster ID to a job cluster ID generator 11 and obtains a new job cluster ID. Next, the management process 2 informs a job scheduler 1 of the job cluster ID and requests the number of processors for use. The processor request may be permitted or not be permitted, and the management process 2 is informed of processors for assignment when it is permitted. The management process 2 receives a list of processors for assignment from the job scheduler 1, and an execution management device 6 assigns jobs to processors to make the processors execute jobs. As a means for execution management, the management process 2 may function as a parallel task execution program, or a program or hardware having a task distribution function may be provided. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for scheduling a job instructed to be executed by a user in a multiprocessor device, and particularly relates to a plurality of jobs instructed by a user, and schedules them collectively. Thus, the present invention relates to a job scheduling method and apparatus that enable an administrator to easily manage jobs.
[0002]
[Prior art]
Conventionally, for example, a job scheduling method as shown in FIG. 6 is known (see, for example, Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3).
6 includes a batch job management system 101 that manages the execution status of jobs and processors, and performs job scheduling. An administrator 102 prepares a plurality of queues in advance and each queue uses them. The number of processors to be processed, the execution time allowed for one job, and the like are set, and an instruction is given to the batch processing system 101.
The user 103 selects an appropriate queue and submits a job.
The batch processing system 101 manages jobs submitted to the queue and the execution status of the processors of the multiprocessor device 104. For a running job that has already been assigned a processor, the limit set by the administrator, such as the execution time, is monitored. If the limit is exceeded, the execution of the job is forcibly terminated.
It also monitors whether the job has finished executing within the limit. If a processor that does not execute a job is found, an unexecuted job is assigned to it and executed. If there are no unexecuted jobs, the processor is idle until the next job comes.
If the queue settings are appropriate, the number of idle processors can be reduced.
For example, it is possible to perform settings such as assigning several processes for a job that ends in a short time and assigning all the remaining processes for a job that takes a long time.
Non-Patent Document 1 describes the features of the function of the job scheduling method as described above, Non-Patent Document 2 describes its outline, and Non-Patent Document 3 describes a job scheduler.
[0003]
A method for automatically setting the queue setting has also been proposed. For example, in Patent Document 1, it is possible to dynamically reset the correspondence between a job class and a job allocation unit according to the status of a job that has been submitted, and the resource can be effectively managed by appropriate job scheduling according to the system usage form. A “job scheduling method” that improves use and system performance is disclosed.
The device described in Japanese Patent Laid-Open No. 2004-228561 can dynamically change the processor to be assigned to the queue according to the number of jobs in each queue, the number of accumulated jobs, and the like.
[0004]
Furthermore, as a case where setting only in the queue is not sufficient, there is a case where the priority of a job is determined for each user or each department, and jobs with equal priority are desired to be executed regardless of the order of job submission. There is a scheduler as a tool for setting in such a case.
Examples of conventional schedulers include those described in Non-Patent Documents 4 and 5 (hereinafter referred to as SunGrid / EE) and those described in Non-Patent Documents 6, 7, 8, and 9 (hereinafter referred to as maui scheduler).
[0005]
Each scheduler has various functions, including a fair share function. By using this function, jobs can be selected and executed equally on the basis of each user, each group, or the like.
The SunGrid / EE is outlined in Non-Patent Document 4 and detailed in Non-Patent Document 5.
According to this, SunGrid / EE performs scheduling using a virtual computing resource usage right called ticket and a ticket distribution method set by an administrator.
The ticket distribution method determines the ratio of ticket allocation for each department, group, user, etc. The ticket is distributed only to the department, group, and user who wants to use the computer, that is, the job is submitted.
For example, if department A, department B, and department C are assigned at a ratio of 1: 3: 6, and all users are assigned equally within each department, then user X of department A, department C When user Y and user Z submit a job, a ticket (total amount is set to 700) is first allocated to department A and department C that submit the job. As a result, department A receives 100 tickets and department B receives 600 tickets. User X is assigned all tickets for department A, and users Y and Z are assigned tickets for department B evenly, so that users X, Y, and Z receive tickets for 100, 300, and 300, respectively. Tickets are further allocated to jobs submitted by each user, and are executed in order from jobs with more tickets per job.
In this way, jobs can be selected and executed equally on the basis of each user, each group, or the like.
[0006]
As for the Maui scheduler, there is a summary in Non-Patent Document 6, a user manual in Non-Patent Document 7, and a management manual in Non-Patent Documents 8 and 9 (particularly, Chapter 6.3 in Non-Patent Document 8). (See FairShare (PP.67-73), Non-Patent Document 9, Chapter 3.3 Scheduling Interation and Job Flow, PP.24-28). According to this, the Maui scheduler calculates the usage amount of the past calculation resources, and if the calculation amount is larger than the calculation amount that should be allocated to the user, group, etc., the job priority that remains in the queue and should be executed later Decrease the degree.
Unlike SunGrid / EE, which distributes tickets by hierarchically configuring users, groups, departments, etc., the calculation amount for each user, group, etc. is aggregated independently. The priority is determined by comparing the amount of calculation to be used.
Thereby, management can be performed succinctly when the distribution ratio can be simply determined in terms of user units or group units.
[0007]
[Patent Document 1]
JP-A-9-101902
[Non-Patent Document 1]
"Product Features", [Search 2002/11/05], Internet <URL: http://www.openpbs.com/features.html>
[Non-Patent Document 2]
"PBS Technical Overview" (searched 05/11/2002), Internet <URL: http://www.openpbs.com/overview.html>
[Non-Patent Document 3]
"The Job Scheduler" [searched 05/11/2002], Internet <URL: http://www.openpbs.com/scheduler.html>
[Non-Patent Document 4]
"Sun [tm] ONE Grid Engine, Enterprise Edition", 2002, Sun Microsystems [searched on September 30, 2002], Internet <URL: http://www.sun.com/software/gridware/sgeee53>
[Non-Patent Document 5]
"Sun [tm] ONE Grid Engine, Enterprise Edition White Paper How Sun [tm] Grid Engine, Enterprise Edition 5.3 Works", 2002, Sun Microsystems [Search 2002/9/30], Internet <URL: http: // www.sun.com/software/gridware/sgeee53/wp-sgeee/wp-sgeee.pdf>
[Non-Patent Document 6]
"Maui Overview", 2002, Supercluster Research and Development Group, [Search 11/05/2002], Internet <URL: http://www.supercluster.org/documentation/maui/mauioverview.html>
[Non-Patent Document 7]
"Maui Users Manual", 2002, Supercluster Research and Development Group, [Search 11/05/2002], Internet <URL: http://www.supercluster.org/documentation/maui/mauiusers.html>
[Non-Patent Document 8]
"6.3 FairShare (pp.67-73)" 2002, Supercluster Research and Development Group, [Search 2002/11/05], Internet <URL: http://www.supercluster.org/documentation/maui/mauiusers.html>
[Non-patent document 9]
"3.3 Scheduling Iterations and Job Flow (pp.24-28)" 2002, Supercluster Research and Development Group, [Search 2002/11/05], Internet <URL: http://www.supercluster.org/documentation/maui /mauiusers.html>
[0008]
[Problems to be solved by the invention]
However, in the conventional batch job management system, the queue is set manually by the administrator, and it is difficult for the administrator to predict in advance the number of processors to be assigned to each queue. Even if it can be set once, it is necessary to change the setting according to the job submission status of the user.
If the number of processors allocated to the queue is inappropriate, an idle processor is generated even though there are jobs in other queues, resulting in inefficient operation.
In the “job scheduling method” described in Patent Document 1, the processor to be assigned to a queue can be dynamically changed according to the number of jobs in each queue, the cumulative number of jobs, and the like. It is also necessary for the administrator to set in advance restrictions such as necessity and execution time to be imposed thereon.
[0009]
Further, in the conventional batch job management system, since the calculation amount required at the time of job submission has been found, the required number of processors can be calculated and given in advance by the batch job management system. However, when the required amount of processing changes dynamically due to an external request, such as when a WWW server is operated on such a multiprocessor system, the conventional batch job management system uses There was also a problem that it was impossible to cope with such a situation.
Furthermore, in a simple batch job management system, it was not particularly distinguished which user submitted the jobs in the queue, so if one user submitted a very large number of jobs, the jobs of other users could be Despite being, it was possible to occupy almost all processors.
[0010]
This problem can be partially addressed by setting a scheduler. In Sun-Grid / EE, the priority of a job is determined by dividing the right to use a computational resource called ticket by the number of jobs. In contrast to a simple batch job management system, it is used when the number of jobs is extremely large. There is a problem that jobs are not processed even though there are many rights, and it is necessary to perform complicated operation to alleviate this problem by using ticket allocation based on past calculation usage similar to Maui scheduler is there.
In addition, since the Maui scheduler calculates the calculation amount for each user and each group and calculates the weighted sum together with other priority parameters, very complicated priority management is required, and management is difficult. Furthermore, there is a problem that it is not possible to describe a priority that allocates a calculation amount to a user within a department.
[0011]
The present invention has been made in view of the above circumstances. The first object of the present invention is to enable easy and efficient operation of a multiprocessor system by dynamically reassigning processors without using a queue. Objective.
A second object of the present invention is to make it possible to change the allocation of the number of processors within a range set by an administrator when the user wants to dynamically change the number of processors to be used.
In addition, the present invention does not hinder the execution of other users' jobs regardless of how many jobs are submitted by the user, and allows the administrator to easily set the usage rate in units of users, groups, etc. This is the third purpose.
[0012]
[Means for Solving the Problems]
FIG. 1 shows the basic configuration of the system of the present invention. As shown in the figure, in the present invention, a job cluster ID generator 11 and a processor allocator 12 are provided in the job scheduler 1.
The job cluster ID generator 11 generates and returns a new ID every time an ID request is received. The processor allocator 12 obtains an available processor so as to satisfy the processor request as much as possible based on the management policy determined in advance by the administrator 3 and the available processor designated in advance. Note that the management policy and available processors can be changed according to the situation even while the job scheduler 1 is being used.
In FIG. 1, a user 4 creates one management process 2 when using the job scheduling of the present invention.
The management process 2 first issues a job cluster ID request to the job cluster ID generator 11 to obtain a new job cluster ID.
Here, the job cluster ID is an ID generated for a job group or server group designated by the user. For example, by requesting an ID for each job group that the user considers to be strongly related, A job cluster ID is assigned to each job group. A plurality of job cluster IDs can be issued to a single user, and a job cluster ID can be shared by a plurality of users.
[0013]
Next, the management process 2 transmits the job cluster ID to the job scheduler 1 and requests the number of processors to be used. The possible request method may be in any form as long as the processor allocator 12 can solve. For example, the request may be made with a specific number of processes, and may be performed with an amount that may be paid when using the processor. It can be done as much as possible, simply within the available range.
The processor request is permitted or not permitted. If permitted, the allocation processor is notified to the management process 2 at some point. The allocation processor may be notified several times depending on the situation. There may be no allocation processor in some situations. Also, depending on the situation, a request that has been granted once may be rejected.
The management process 2 receives a list of assigned processors from the job scheduler 1 and executes a program that the user wants to execute on the multiprocessor device 5 based on the list.
[0014]
The following various methods are conceivable as an execution management apparatus for executing a program.
(i) As shown in FIG. 1, an execution management device 6 similar to the batch job management system shown in FIG. 6 is used.
As the execution management apparatus, a batch job management system similar to that shown in FIG. 6 is used. As shown in FIG. 1, an available node (processor) is assigned to the job queue 61, and the management process 2 queues the job. The method of throwing in can be considered.
In this case, the processor allocation result sent from the processor allocator 12 of the job scheduler 1 at any time can be directly received by the execution management apparatus in addition to being transferred to the execution management apparatus 6 after being received by the management process 2.
(ii) The management process also serves as a parallel task execution program.
As another example, as shown in FIG. 2 to be described later, the management process 2 also serves as a parallel task execution program, starts a task processing program on an assigned processor, and distributes the tasks to those task processing programs. It is also possible to execute it. In this case, it is considered that the execution management apparatus is a virtual existence, and the management process 2 also serves as its role.
(iii) Provide another program or hardware that distributes tasks to those task processing programs.
As another example, as shown in FIG. 3 to be described later, a task processing program is started on an assigned processor, but another program or hardware (hereinafter referred to as task allocation) that allocates tasks to those task processing programs. It is also conceivable that the management process 2 is executed to notify the task distribution means of the distribution destination. In this case, it is considered that the execution management apparatus is a virtual existence, and the management process also serves as its role.
In this case, the task distribution means can also accept an external task distribution request.
[0015]
As described above, in the present invention, a job cluster ID is generated in response to a request from the management process, a job cluster ID is designated from the management process, and when there is a request for the number of processors to be used, processor allocation The means allocates a processor from the available processors in units of the job cluster ID based on a predetermined management policy and an available processor designated in advance.
In addition, the processor allocation means can use the job cluster ID as a unit in response to a request from a processor by specifying a job cluster ID, the number of processors, a desired allocation time, the number of exclusive hours, etc. Processors can be assigned from within the processor. It is also possible to permit / deny a plurality of allocation requests.
Further, the same method as the SUNGrid / EE ticket calculation method may be used, and the processor allocation unit may be configured to allocate processors equally for each department, group, and user.
[0016]
As a result, there is no need for the administrator to set queues in advance and assign processors to each queue. In addition, since no queue is used, the administrator does not need to set in advance what kind of queue is necessary.
Conventionally, the queue is set in order to solve the problem that the average waiting time becomes long when a short-time job and a long-time job are arranged in the same queue. In the present invention, since a queue exists for each management process (job cluster ID), there is no problem that the average waiting time becomes long.
For example, it is not necessary to reserve a processor for a queue for a short time job, so that when there is no short time job, it can be easily and efficiently operated using all the processors.
Therefore, by dynamically reallocating processors without using a queue, the multiprocessor system can be operated easily and efficiently, and the first problem can be achieved.
In the present invention, it may be permitted to issue a processor request a plurality of times. For this reason, even when the required processing amount is dynamically increased or decreased by a request from the outside like a WWW server, the number of processors to be requested can be increased as the processing amount increases. The processor allocation means reallocates the processor every time a request is received, and the processing amount of the processor allocated to the job cluster ID having a lower priority (or no problem even if calculated at another time) is processed. It can be reassigned to the increased job cluster ID. As a result, it is possible to cope with the case where the processing amount dynamically changes.
Therefore, the assignment of the number of processors can be changed within the range set by the administrator, and the second problem can be achieved.
Furthermore, in the present invention, the administrator can instruct to allocate the number of processors evenly for each user or group by describing the management policy.
In addition, the same calculation as that of the SunGrid / EE ticket is performed, and a processor is assigned to a department, a group, a user, a job cluster ID, etc. according to a predetermined ticket assignment. Allocation methods that were difficult with the Maui scheduler, such as allocation, can be easily performed.
Furthermore, the problem of ticket allocation to jobs that has been a problem in the SunGrid / EE is eliminated, so the same number of processors can be used regardless of whether the number of jobs is extremely large or small.
Therefore, even if a large number of jobs are input or a job cluster is created, the number of processors that can be used should be equal to the number of processors that can be used by other users / groups, or the ratio specified by the administrator. I can do it.
Therefore, no matter how many jobs a user submits, the execution of other users' jobs is not hindered, and the administrator can easily set the usage rate in units of users, groups, etc. The third problem can be achieved.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a diagram showing the configuration of a system according to a first embodiment of the present invention, and FIG. 1 shows a configuration when a program is executed by an execution management apparatus 6 similar to the batch job management system shown in FIG. Is shown.
In the figure, reference numeral 1 denotes a job scheduler, a job cluster ID generator 11 that generates a job cluster ID in response to a request from a management process, and a processor for each job cluster ID based on a management policy set by the administrator 3. Is provided. Further, the job scheduler 1 is provided with a processor allocation DB 13 that manages information regarding each job cluster ID.
Reference numeral 2 denotes a management process, which makes a job cluster ID request to the job cluster ID generator 11, transmits a job cluster ID to the job scheduler 1, requests the number of processors to be used, and is allowed to be assigned. The assignment result is transferred to the execution management device 6. The management process 2 has an area for storing an environment variable JCID such as a job cluster ID.
Reference numeral 3 denotes an administrator who determines the management policy described above, 4 denotes a user, and 5 denotes a multiprocessor device including a plurality of processors.
Reference numeral 6 denotes an execution management apparatus similar to the conventional batch job management system, and as described above, manages and executes the jobs input to the job queue 61 and the execution status of the processors of the multiprocessor apparatus 5. Assign an unexecuted job to a non-processor and execute it.
The job scheduler 1 and the execution management device 6 constitute a job scheduling device of the present invention.
[0018]
The processor allocator 12 shown in FIG. 1 performs processor allocation as follows.
The processor allocator 12 is called whenever processor allocation is required. More specifically, it is called when:
-The management policy has been changed.
・ Available processors increased / decreased.
-A processor request has occurred from one of the management processes. Or the request was changed.
-There is no management process. That is, the required processor has decreased.
The unit for assignment is a job cluster ID unit, and a set of processors is assigned to one job cluster ID.
When assignment is performed, the assignment result is notified to each job cluster ID. The output destination of the notification is the management process 2 that transmitted the job cluster ID and made the request, or the process or file designated by the management process 2.
[0019]
The processor allocator 12 changes the operation according to the management policy described by the administrator 3. Various management policies can be considered. Specifically, for example, the following policies can be considered.
(i) Whether a request that occupies a certain number of processors for a certain time from a certain job cluster ID should be permitted.
The determination is made based on, for example, the user name, the group to which the user belongs, the user's payment account, and the like. The determination is also made on the basis of the number of processors desiring exclusive use and the number of hours desiring exclusive use.
(ii) Whether a request to occupy a certain number of processors from a certain job cluster ID for a certain time from a certain time should be permitted.
The determination is made based on, for example, the user name, the group to which the user belongs, the user's payment account, and the like. The determination is also made based on the number of processors that are desired to be occupied, the number of hours that are desired to be owned, and the time that is desired.
(iii) Whether or not a plurality of processor requests should be permitted when a request is made by increasing / decreasing a plurality of processor requests from a certain job cluster ID.
The determination is made based on, for example, the user name, the group to which the user belongs, the user's payment account, and the like. Even if it is permitted to make a plurality of requests, the determination for each request is also made based on the upper limit of the number of processors at the time of each processor request, the change frequency, and the number of requests.
[0020]
(iv) When requesting as many processors as possible from a certain job cluster ID, the same number of processors should be allocated equally to the job cluster IDs that make the same request, or more preferentially Should be assigned to?
The determination is made based on, for example, the user name, the group to which the user belongs, the user's payment account, and the like.
Also, what should be done with the unit for even allocation? For example, all job clusters should be treated equally, or the number of processors assigned to all job clusters issued by one user and the number of processors assigned to all job clusters issued by other users are equal. Should it be equal, or should be equal for each group to which each user belongs?
In the case of equal allocation, the processor allocation ratio can be calculated based on the allocation unit and ratio in the same manner as the SUNGrid / EE ticket calculation method, and the remaining number of processors can be proportionally distributed.
For example, if tickets are assigned to departments, groups, users, and job cluster IDs, and there is an equal allocation request, according to ticket assignments predetermined for departments, groups, users, and job cluster IDs, as in SUNGrid / EE Assign processors.
(v) Which job cluster ID request should be rejected when the number of processors is insufficient.
The determination is made based on, for example, the user name, the group to which the user belongs, the user's payment account, and the like. The determination is also made based on how the request was made (dedicated processor number / time designation, exclusive number of processors / time / time designation, request that can be requested multiple times, equal request).
[0021]
With these management policies set, the processor assigner 12 assigns processors as follows.
(i) Exclude unauthorized requests. If a request is made a plurality of times from a management process having a certain job cluster ID and the request is not permitted, the request contents made last time with the same job cluster ID are used.
In other words, when there is a request from a job cluster ID that does not permit the request multiple times, the request is not permitted as in the previous case.
(ii) Allocate a request for specifying the number of processors / time / time.
(iii) Allocate requests for number of processors / time.
(iv) Allocate a job cluster ID request that can be requested multiple times.
(v) Allocate the remaining processors equally. The unit of allocation follows the description of the management policy.
(vi) If the total number of requested processors is greater than the number of available processors at any time, the permission of an appropriate job cluster ID is cancelled. Which job cluster ID should be canceled depends on the description of the management policy.
[0022]
FIG. 2 is a diagram showing the configuration of a system according to the second embodiment of the present invention. FIG. 2 shows a case where the management process 2 described above also serves as a parallel task execution program. Parallel task execution means 21 for performing the above is provided. In this case, as described above, the execution management apparatus is a virtual existence.
The parallel task execution means 21 starts a task processing program on the assigned processor, distributes the tasks to those task processing programs, and executes them.
Other configurations are the same as those shown in FIG. 1, and the management process 2 issues a job cluster ID request to the job cluster ID generator 11 to obtain a new job cluster ID. Next, the management process 2 transmits the job cluster ID to the job scheduler 1 and requests the number of processors to be used.
When the processor request is permitted, the management process 2 receives the list of assigned processors from the job scheduler 1, and the parallel task execution means 21 starts a task processing program on the assigned processor, and the user uses the multiprocessor device 5. Run the program you want to run.
[0023]
FIG. 3 is a diagram showing the configuration of a system according to the third embodiment of the present invention. FIG. 3 shows a case where another program or hardware for distributing tasks is provided, as shown in FIG. Are provided with task distribution means 7. Also in this case, as described above, the execution management apparatus is a virtual existence.
When the task distribution unit 7 is notified of the task distribution destination from the management process 2, the task distribution unit 7 starts a task processing program on the allocated processor. Further, the task distribution means 7 can also accept requests from outside.
Other configurations are the same as those shown in FIG. 1, and the management process 2 issues a job cluster ID request to the job cluster ID generator 11 to obtain a new job cluster ID. Next, the management process 2 transmits the job cluster ID to the job scheduler 1 and requests the number of processors to be used.
When the processor request is permitted, the management process 2 receives a list of assigned processors from the job scheduler 1 and notifies the task distribution means 7 of it. The task distribution means 7 starts a task processing program on the assigned processor.
[0024]
Hereinafter, a case where job scheduling of this embodiment is performed on UNIX will be described.
(1) Information transmission means between job scheduler 1 and management process 2
Information transmission from the management process 2 to the job scheduler 1 is acquisition of a job cluster ID, processor request, and end of the job cluster. Information transmission from the job scheduler 1 to the management process 2 is a request rejection notification or a processor allocation result. The latter communicates asynchronously.
Therefore, the following three types of commands are prepared.
-Jobcluster command
% jobcluster management process command string
The jobcluster command first acquires the job cluster ID, puts it in the environment variable JCID, and then starts the management process using the exec system call according to the management process command character string. Since the exec system call is used, the jobcluster process becomes a management process as it is.
[0025]
-Jcrequest command
% jcrequest [-e notification command | -f notification file | -s notification process]
Request file
The jcrequest command refers to the environment variable JCID to obtain the job cluster ID. Then, the processor request written in the request file is interpreted. If the processor request is rejected because it violates the management policy, an error message is output and the end status is other than zero.
If the policy is not violated, the result is notified by means of an option. Notification is made each time the allocation processor is changed.
If the -e option is specified, a list of processor names is input to the standard input of the notification command for that argument.
If the -f option is specified, a list of processor names is written to the argument notification file.
If the -s option is specified, a list of processor names and blank lines are written to the standard output, and then a SIGHUP signal is sent to the argument notification process. In this case, the jcrequest command does not end until a jcend command described later is input from another process.
If no option is specified, only a list of processor names as a result of processor allocation resulting from the request is written to the standard output, and the process is terminated. No further notification is performed with this command.
[0026]
Jcend command
% jcend
The jcend command refers to the environment variable JCID to obtain the job cluster ID. Then, all processors assigned to the job cluster ID are released and reassigned to other job cluster IDs. If necessary, terminate the jcrequest command normally.
[0027]
(2) Starting the management process
The management process 2 must first obtain a job cluster ID. Therefore, the management process 2 is started after obtaining the job cluster ID using the above-described jobcluster command.
The management process 2 requests a processor request with a jcrequest command as necessary. Then, every time there is a notification, the allocation processor can perform appropriate execution.
If the management process ends normally, execute the jcend command. Since the detailed work contents differ depending on the use, the management processes are listed.
[0028]
(3) Job cluster ID generation
Exclusive control is performed to create an ID. In order to be used as a batch queue name, the ID takes the form of alphabet + numerical value. The ID is not reused, and the ID is generated by monotonically increasing the numerical value.
[0029]
(3) Processor allocation
As described above, the jcrequest command performs management policy check and processor assignment.
If the processor request is not rejected by the management policy check, the request content is stored in the processor allocation DB, and then a jcresched command for performing processor allocation internally, which will be described below, is called. When saving, a lock is used to exclusively operate the DB.
The jcresched command is called every time processor allocation is required. More specifically, it is called at the following times.
(i) The management policy has been changed.
(ii) Available processors increased / decreased.
(iii) A processor request originated from one of the management processes. Or the request was changed.
(iv) There is no management process. That is, the required processor has decreased.
[0030]
Jcresched command
% jcresched
The jcresched command performs processor allocation with reference to the processor allocation DB 13, management policy, and available processor list (file). There are no arguments. The jcresched command uses a lock to exclusively operate the processor allocation DB 13. This prevents two or more jcresched commands from being executed at the same time.
As described above, the processor allocation DB 13 is provided in the job scheduler 1 and includes at least the following four items, and manages information regarding each job cluster ID.
-Job cluster ID
Processor request
-Previous assignment result
-Notification means (may be more than one)
[0031]
Then, as described above, the processor processor assigner 12 of the job scheduler 1 assigns processors as follows.
(i) Allocate a request for specifying the number of processors / time / time.
(ii) Allocate a request for specifying the number of processors / time.
(iii) Allocate a job cluster ID request that can be requested multiple times.
(iv) Allocate the remaining processors evenly. The allocation unit follows the policy description.
(v) If the total number of requested processors is smaller than the number of available processors at any time, the permission of an appropriate job cluster ID is cancelled. If the job cluster ID is assigned evenly, no processor is assigned to the assigned processor. Which job cluster ID should be canceled depends on the description of the management policy.
Then, the assignment result regarding each job cluster ID is notified to the notification means of the job cluster. The notification method is as described for the jcrequest command.
[0032]
(4) Work of the management process that submits jobs
This management process is required when the conventional batch job input operation is used in the present invention. This management process works in cooperation with the execution management apparatus 6 shown in FIG.
FIG. 4 shows the flow of processing when a job is submitted.
First, the job cluster ID is acquired by the jobcluster command, put in the environment variable JCID, then the management process 2 is started according to the management process command character string, and a queue having the same name as the job cluster ID is created on the execution management device 6 ((1) in FIG. 4). The environment variable JCID is used as the job cluster ID.
Next, a command “qresched $ JCID” for changing the processor assigned to the queue is registered in the job scheduler 1 using the jcrequest -e command. The qresched command interprets the standard input as an assigned processor using the argument JCID as the queue name, and changes the processor assigned to the queue. The jcresched command allocates processors by referring to the processor allocation DB, management policy, and available processor list (file). ((2) in FIG. 4).
Thereafter, a normal job is submitted ((3) in FIG. 4). A job cluster ID is used as a queue name when a job is submitted. As a result, the execution management apparatus 6 takes out the queue, assigns the job to the execution node group, and executes it ((4) in FIG. 4).
When the processor assignment is changed asynchronously, a command “qresched $ JCID” for changing the processor assigned to the queue is registered in the job scheduler 1 using the jcrequest -e command as described above. The qresched command changes the processor assigned to the queue by interpreting the standard input as the assigned processor, using the argument JCID as the queue name. The jcresched command assigns processors ((5) in Fig. 4). At that time, the allocation result is also notified to all the JCIDs.
Then, it waits whether all the input jobs have been completed, and after all the jobs are completed, the jcend command is input ((6) in FIG. 4). The queue is discarded and the user is notified of the end.
The creation of the queue and the change of the processor assigned to the queue can be realized by linking with the execution management apparatus 6 in FIG. 1, and for example, PBS can be realized by calling the qmgr command. The job can be submitted using the qsub command, and the job can be monitored using the qstat command.
(5) Work of the management process that dynamically changes the number of processors
When executing a program that processes a task from an external client such as a WEB server or a search server, the following management process is useful.
FIG. 5 shows the flow of processing when the number of processors is dynamically changed.
First, the server administrator notifies the management command name as an activation argument, and acquires the job cluster ID by the jobcluster command ((1) in FIG. 5).
Next, a command “resched $ JCID” for changing the processor assigned to the queue is registered in the job scheduler using the jcrequest -e command. The resched command changes the processor that starts the server by interpreting the standard input as an assigned processor, using the argument JCID as the queue name. The jcresched command performs processor allocation. ((2) in FIG. 5).
Then, if a processor of a server that is already running is not included in the assigned processor, the server is terminated. If a processor that does not yet run the server is included in the assigned processor, the server is started by the processor ((3) in FIG. 5).
After that, tasks are allocated (assigned). The task is allocated to one of the allocation processors, and the obtained result is returned to the client that has entered the task ((4) in FIG. 5). If another task distribution mechanism or distribution device is used, the resched command performs the task of notifying the mechanism or device of the distribution destination in addition to the server activation / termination described above.
When there is a processor request due to high load detection or abandonment of a processor due to low load detection, the command “resched $ JCID that changes the processor assigned to the queue using the jcrequest -e command as described above. "Is registered in the job scheduler, and the processor for starting the server is changed ((5) in FIG. 5).
Then, as described above, the server is started and tasks are assigned ((6) in FIG. 5).
In addition, such a management process is not normally terminated by itself, and therefore is terminated only when the user instructs the termination of the management process from the outside. When ending, the jcend command is executed as described above ((7) in FIG. 5).
(6) Work of management process to reserve execution date and time
The management process can be executed in exactly the same management process as that shown in FIG.
The only difference is that the processor is requested by specifying time / private time / number of dedicated processors as the request processor request contents in the jcrequest command.
[0033]
(Appendix 1) A job scheduling method in a multiprocessor system,
In response to a request from the management process, generate an ID
When there is a request for the number of processors to use by specifying the ID above from the management process,
Based on a predetermined management policy and a pre-designated available processor, a processor is assigned in units of the ID.
A job scheduling method.
(Appendix 2) A job scheduling apparatus in a multiprocessor system,
ID generation means for generating an ID in response to a request from the management process;
When there is a request for the number of processors to be used by specifying the ID from the management process, a processor management means for allocating a processor in units of the ID based on a predetermined management policy and an available processor specified in advance. Prepared
A job scheduling apparatus.
(Supplementary note 3) A job scheduling apparatus in a multiprocessor system,
ID generation means for generating an ID in response to a request from the management process;
A processor allocating means for allocating a processor in units of the ID based on a predetermined management policy and a pre-designated available processor when there is a request for the number of processors to be used by designating the ID from the management process;
Equipped with an execution management means to manage the execution of jobs submitted from the management process
A job scheduling apparatus characterized by that.
(Supplementary Note 4) When a job is submitted from the management process, the execution management means assigns a processor of the multiprocessor system to a job queue,
Management process submits job to the queue
The job scheduling apparatus according to supplementary note 3, wherein
(Supplementary Note 5) When a job is submitted from the management process, the execution management unit starts a tax processing program on the assigned processor, and distributes the task to the tax processing program for execution.
The job scheduling apparatus according to supplementary note 3, wherein
(Additional remark 6) It has the distribution means which distributes a task to a task processing program, The said execution management means starts a tax processing program on the allocated processor,
The management process notifies the distribution means of the distribution destination.
The job scheduling apparatus according to supplementary note 3, wherein
(Supplementary note 7) The management process outputs a request to the processor allocation means by specifying the ID, the number of processors, the desired allocation time, and the number of exclusive hours.
In response to the request, the processor assigning means assigns a processor in units of the ID based on a predetermined management policy and an available processor designated in advance.
The job scheduling apparatus according to appendix 2, 3, 4, 5, or appendix 6.
(Supplementary Note 8) The management process specifies a job cluster ID, the number of processors, and the number of exclusive hours, and outputs a request to the processor allocation means.
In response to the request, the processor assigning means assigns a processor in units of the ID based on a predetermined management policy and an available processor designated in advance.
The job scheduling apparatus according to appendix 2, 3, 4, 5, or appendix 6.
(Supplementary Note 9) The processor allocation means includes means for permitting or not permitting whether or not to accept a plurality of allocations.
The job scheduling apparatus according to Supplementary Note 2, 3, 4, 5, 6, 7 or Supplementary Note 8, wherein
(Supplementary Note 10) The processor assigning means assigns processors equally for each department, each group, and each user.
The job scheduling apparatus according to Supplementary Note 2, 3, 4, 5, 6, 7, 8 or Supplementary Note 9, wherein
(Supplementary Note 11) A job scheduling program in a multiprocessor system,
In response to a request from the management process, the program generates a job cluster ID and assigns a job cluster ID;
When there is a request for the number of processors to be used by specifying a job cluster ID from the management process, a process for assigning a processor in units of job clusters to the computer based on a predetermined management policy and a pre-specified available processor Execute
A job scheduling program.
[00034]
【The invention's effect】
As described above, according to the present invention, the following effects can be obtained.
(1) Since the processor assigning means assigns processors from the available processors in units of job cluster IDs, it is not necessary for the administrator to set queues in advance and assign processors to each queue.
For example, there is no need to reserve a processor for a queue for a short-time job, so that when there is no short-time job, all processors can be used efficiently.
(2) It may be permitted to issue a processor request multiple times. For this reason, even when the required processing amount is dynamically increased or decreased by a request from the outside like a WWW server, the number of processors to be requested can be increased as the processing amount increases. The processor allocator reassigns the processor each time a request is received, and the processor assigned to the job cluster ID with a lower priority (or no problem at another time) is processed. It can be reassigned to the increased job cluster ID. Thereby, it is possible to cope with a case where the processing amount dynamically changes.
(3) By describing the management policy, the administrator can give an instruction to allocate the number of processors equally for each user or group.
For this reason, even if a large number of jobs are input or a job cluster is created, the number of available processors may be equal to the number of available processors of other users or the ratio specified by the administrator. I can do it.
[Brief description of the drawings]
FIG. 1 is a diagram showing the configuration of a system according to a first embodiment of the present invention.
FIG. 2 is a diagram showing a system configuration of a second exemplary embodiment of the present invention.
FIG. 3 is a diagram showing a system configuration of a third exemplary embodiment of the present invention.
FIG. 4 is a sequence diagram illustrating an operation of a management process for submitting a job according to the embodiment of this invention.
FIG. 5 is a sequence diagram illustrating an operation of a management process for dynamically changing the number of processors according to the embodiment of this invention.
FIG. 6 is a diagram for explaining scheduling by a conventional batch job management system.
[Explanation of symbols]
1 Job scheduler
11 Job cluster ID generator
12 processor allocator
13 Processor allocation DB
2 Management process
21 Parallel task execution means
3 managers
4 users
5 Multiprocessor device
6 execution management device
7 Task distribution means

Claims (5)

A job scheduling method in a multiprocessor system,
In response to a request from the management process, generate an ID
When there is a request for the number of processors to use by specifying the ID above from the management process,
A job scheduling method, wherein a processor is assigned in units of the ID based on a predetermined management policy and an available processor designated in advance. A job scheduling apparatus in a multiprocessor system,
ID generation means for generating an ID in response to a request from the management process;
When there is a request for the number of processors to be used by specifying the ID from the management process, a processor management means for allocating a processor in units of the ID based on a predetermined management policy and an available processor specified in advance. A job scheduling apparatus comprising: A job scheduling apparatus in a multiprocessor system,
ID generation means for generating an ID in response to a request from the management process;
A processor allocating means for allocating a processor in units of the ID based on a predetermined management policy and a pre-designated available processor when there is a request for the number of processors to be used by designating the ID from the management process;
A job scheduling apparatus comprising execution management means for managing execution of a job input from a management process. 4. The job scheduling apparatus according to claim 2, wherein said processor allocation means includes means for permitting or not permitting whether or not to accept a plurality of allocations. A job scheduling program in a multiprocessor system,
In response to a request from the management process, the program generates a job cluster ID and assigns a job cluster ID;
When there is a request for the number of processors to be used by specifying a job cluster ID from the management process, a process for assigning a processor in units of job clusters to the computer based on a predetermined management policy and a pre-specified available processor A job scheduling program that is executed.

Images