KR101092359B1 - Scheduling apparatus and method in grid - Google Patents

Abstract

The grid job scheduling apparatus according to the present invention receives and analyzes the computing resource and the network resource from the information collecting unit and the information collecting unit for collecting information of the computing resource and the network resource of the agent processing the assigned job, the analysis By including a task manager to set the number of tasks to be assigned to the agent according to the result, it is possible to improve the processing speed of the grid task.

Grid, Scheduling, Agent, Task Manager, Information Collector

Description

SCHEDULING APPARATUS AND METHOD IN GRID

The present invention relates to an apparatus and method for scheduling a grid task, and more particularly, to an apparatus and method for scheduling a grid task that can improve grid task processing efficiency by dynamically setting the number of tasks to be allocated to an agent that processes the task. It is about.

Grid is an internet network structure that behaves like a neural tissue, unlike a web that can be connected to only one place at a time. A grid is an environment that enables geographically distributed interoperability of high-performance computing resources, such as hundreds of millions of computers, databases, and advanced equipment, scattered around the world, across networks. . In this way, it is possible to integrate heterogeneous grid resources in the grid as if using one high-performance resource.

In order to use distributed grid resources as a resource in a system with such computing power, it is necessary to collect information about heterogeneous resources and find a grid resource suitable for a task to be executed based on the information.

Finding suitable grid resources for tasks and allocating tasks appropriately to found grid resources is one of the main concerns of the grid.

The present invention is to provide a grid job scheduling apparatus and method that can improve the grid job processing efficiency by dynamically setting the number of jobs to be allocated to the agent for processing the job.

Technical problems to be achieved by the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

In order to achieve the above object, a grid job scheduling apparatus according to the present invention includes an information collection unit for collecting information of a calculation resource and a network resource of an agent for processing an assigned job, and the calculation resource and network resource from the information collection unit. It may include a task manager to receive and analyze, and set the number of tasks to be assigned to the agent according to the analysis result.

In this case, the data manager may further include a data manager configured to store information on a local disk, a pilot job for collecting information of a compute resource and a network resource of the agent, on the local disk, and the task manager may perform the pilot job for collecting the information. The information collecting unit may collect information of the computational resource and the network resource of the agent by using the processing result of the information gathering pilot task.

Also, the pilot task for collecting information may be part of a task to be assigned to the agent.

In addition, the pilot task for collecting information may be different for each agent.

In addition, at least a portion of the task to be allocated except for the pilot task for collecting information may be stored in a grid storage resource.

On the other hand, the grid job scheduling method according to the present invention stores a job requested by the user in a grid storage resource, and storing a part of the job on a local disk as a pilot job for information collection for collecting information of the calculation resource and network resources And assigning the information gathering pilot task to an agent and receiving the processing result of the agent to collect information of the computing resource and the network resource of the agent, analyzing the collected computing resource and the network resource, and analyzing the analysis result. And setting the number of tasks to be allocated to the agent.

In this case, the pilot task for collecting information may be an initial stage task among the tasks to be allocated.

The job stored in the grid storage resource may be a job except for the pilot job for collecting information.

As described above, the grid job scheduling method according to the present invention sets the number of jobs to be allocated to the agent by using information of the computing resource and the network resource of the agent to process the job requested by the user, thereby processing efficiency of the grid job. Can improve.

In other words, it is possible to improve the efficiency of grid-based high-volume processing tasks and to provide scheduling that can be suitably applied to the dynamic characteristics of the grid (dynamic changes such as configuration and performance).

In addition, it is possible to simultaneously support a variety of tasks, that is, various applications can be utilized as a service means for effectively providing grid resources.

Hereinafter, an apparatus and method for scheduling a grid job according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a grid job scheduling apparatus according to the present invention.

The grid job scheduling apparatus 100 illustrated in FIG. 1 includes an information collecting unit 110 collecting information of a computing resource and a network resource of an agent processing an assigned job, and the computing resource and the network resource from the information collecting unit. It receives and analyzes, and includes a task manager 130 for setting the number of tasks to be assigned to the agent according to the analysis result.

Grid is an infrastructure that can mutually share / use information and communication resources such as geographically dispersed high-performance computers, large-scale databases, and high-tech equipment by interworking with high-speed networks. The characteristics of the grid are that Internet users are connected horizontally, high-speed computation and large data processing are possible, and many can access and work at the same time. Grid middleware is needed to implement this grid.

Grid middleware is a software that interoperates with various devices connected to a network. It is a technology that makes use of geographically dispersed resources as if it were a single computer. Currently used middleware includes Globus, Legion, and Condor. There are problems such as information compatibility problem, difficulty of resource management, difficulty of application development, and difficulty of integration in system development due to the use of various heterogeneous computing resources. Currently, standardization of grid middleware is in progress to solve this problem.

Grid middleware differs greatly in size and technology compared to traditional distributed systems that have been researched and developed so far. This is because parametric studies and data analysis of applications in the grid require processing thousands of processes, tens of thousands of data files, and petabytes of data. To solve the huge problems in the grid environment, resource retrieval service that finds idle resources connected to the grid, scheduling service that determines and distributes the processing order of assigned tasks, grid security service for system stability, and computing resources A user account service for the cost processing is required.

For example, the element that performs the resource information service in the Globus toolkit is called MDS. MDS is similar to the Internet's DNS service as an element for sharing information of resources in the grid and providing them to users. The part responsible for resource management services in the Globus toolkit is called GRAM. GRAM is the central element of the Globus toolkit, allowing remote resources to be used, using distributed resources simultaneously, and handling the administrative differences that exist between them. The Globus Toolkit provides GASS, GridFTP, and RC functions for data management services. In the Globus toolkit, security is called GSI, and grid security is an important problem that naturally occurs as sharing distributed resources.

In grid middleware other than Globus, there are element technologies with similar functions but with different names.

1 illustrates a system for scheduling services among the aforementioned resource search service, scheduling service, grid security service, and user account service. Although elements used for other services are included, they are also indicated as elements to be used for scheduling described in the present invention.

The agent manager (AMGR) 190 manages maintenance and schedule for agents participating in the grid resource 230. Specifically, the agent will install and manage agents to participate in grid resources. An agent is an element that properly allocates a task requested by a user and performs processing.

The information collecting unit 110 performs a function of an information service. Specifically, it is to efficiently manage and service necessary metadata so that other grid services (resource management service or data management service) can be performed smoothly. This can be thought of as a specific virtual machine when comparing a grid computing environment to a local computing environment, and to take advantage of it, it is like installing grid middleware such as Globus first. In addition, any operating system needs the necessary configuration and information (in Linux's / etc, / proc directory) to operate, and the information collector can be thought of as maintaining the information needed for the grid computing environment. In this embodiment, the information of the computing resource and the network resource of the agent is collected. For example, the computational resource of the agent corresponds to a speed, time, etc. of processing an assigned task, and the network resource corresponds to a communication speed of a network utilized by the agent in a grid resource. Collecting information of the computational resources and network resources of the agent as described above is to allow the task manager to set the number of tasks to be assigned to each agent.

The task manager 130 assigns and starts a task to the agent. The task manager is also responsible for monitoring and controlling the status of the task during task execution. The existing task manager takes the form of assigning a certain number of tasks to each agent according to the number of agents installed in the grid resource. In other words, the number of tasks assigned to each agent is the same. The number of tasks assigned to each agent is generally one.

2 is a flow chart of the grid task as seen from the user's point of view.

First, a user prepares input data using a user interface (S 510), and executes a task (S 520). Thereafter, the user monitors the work status and other information and performs management tasks such as stopping and resuming the work (S 530). When the work is completed, the user accesses the storage resources to obtain a work result (S540). Among these, the task manager 130 is involved in the management task, but is not displayed to the user, but the task is divided and assigned one by one according to the number of agents. Due to the difference in computational resources of each agent in this reality, even if the task assigned to a certain agent is completed, there is a problem of waiting for the agent that completed the task relatively late. That is, the result received from each agent is collected and the final result is transmitted to the user or transmitted to the storage resource 235. A bottleneck occurs due to the speed difference of each agent.

In order to solve this problem, the job manager 130 of the present embodiment receives and analyzes the calculation resource and the network resource 233 of the agent 231 from the information collecting unit 110 as shown in FIG. Accordingly, the number of tasks to be allocated to the agent is set. As a result, a larger number of tasks can be allocated to agents with faster processing speeds and communication speeds, and the overall processing speed is improved. As a result, processing results of each agent are output at similar time points, and bottlenecks are eliminated. For this purpose, the number of tasks to be allocated to the agent is preferably set so that the output (considering network resources) of each agent that is allocated the tasks requested / executed by the user is made at a similar time.

In order to collect the information of the computational resources and network resources of the agent, it is necessary to assign a schedule task to each agent and to sense the speed of processing the task, the data transmission rate, and the like. In this way, a job used to collect agent-related information is called a pilot job for collecting information.

Since the pilot job for collecting information is preferably managed by the grid job scheduling apparatus according to the present invention, the grid job scheduling apparatus may further include a local disk 170 which is separated from the storage resources included in the grid resources.

The pilot work for collecting information is stored in the local disk, and the storage is performed by the data manager 150. In this state, the task manager assigns and processes information gathering pilot tasks to the agent, and the information gathering unit collects information of the computational and network resources of the agent using the processing result of the information gathering pilot task.

Meanwhile, the pilot task for collecting information may be a task used only for collecting agent-related information. In this case, the agent is in a situation of processing a task not requested by the user. In addition, since the information of the network resource is a change part, it is desirable to collect the information immediately after the user requests a job. In such a situation, immediately after the user requests a task, if the task for collecting agent-related information is preceded instead of the task requested by the user, the task requested by the user may be processed late and the load of each agent may increase. Therefore, the pilot task for collecting information is preferably part of the task to be assigned to the agent. That is, the pilot task for collecting information may be part of a task requested by the user, and may be different for each agent. This is to prevent duplicate processing of the same task by multiple agents when all agents request the same information gathering pilot task even for some of the requested tasks. Even though the information gathering pilot task assigned to each agent is different, each information gathering pilot task corresponds to the task requested by the user, so there is no difference in the processing result. For example, if the task requested by the user uses the same processing routine and the input value is tens of thousands to hundreds of thousands, each agent should use the same processing routine and instead distribute the input properly. Will not. Therefore, even if the pilot task for collecting information allocated to each agent is different, it does not cause any error in the task manager's assigned task count setting.

 According to this, the processing speed is not impeded by executing a part of the work requested by the user and collecting agent-related information (calculation resource information, network resource information) according to the processing result. In this case, since agent-related information is not collected, the number of pilot tasks for collecting information to be allocated to each agent is considered. In general, it is preferable to allocate one pilot task for collecting information, and if the previously collected agent-related information exists in the information collecting unit, set the number of pilot tasks for collecting information according to the collected agent-related information. Can be assigned. After that, the processing result of the pilot task for collecting information is collected to newly set the number of tasks to be allocated to each agent.

On the other hand, it is preferable that only a pilot job for collecting information is stored on the local disk. Looking at it from the other direction, it means that the data / task entered by the user in the storage resource may not be saved as a whole. That is, the data / job entered by the user may be stored on the local disk through the data manager and then stored in the storage resource, which is a grid resource. In this case, the data manager does not store all the data in the storage resource, but performs a pilot job for collecting information. Can be stored on local disk

In this case, the entire job requested by the user may be stored in the local disk. However, in this case, the load of network resources and the grid job scheduling apparatus may increase because each agent uses the local disk. Therefore, at least a portion except for the pilot task for collecting information is preferably stored in a grid storage resource. That is, the stored resource may store the entire work requested by the user or only a part of the work requested by the user except for the pilot work for collecting information. The number of pilot tasks for collecting information may be set uniformly, but may be set flexibly in consideration of the characteristics of each task and agent-related information previously stored in the information collecting unit.

4 is a flowchart illustrating a grid job scheduling method according to the present invention.

The grid job scheduling method shown in FIG. 4 may be described as an operation of the grid job scheduling apparatus shown in FIG. 1.

First, a job requested by a user is stored in a grid storage resource, and a part of the job is stored on a local disk as a pilot job for collecting information on information of a calculation resource and a network resource (S610). The pilot task for collecting information, which is performed by the data manager, may be an early stage task among tasks to be assigned to an agent. Of course, depending on the type of work, there may be cases where it is not necessary to consider the after relationship, but when it is a job that uses previously processed data, it is necessary to first process the preceding data. It is advisable to set the task of processing nature as a pilot task for collecting information. In addition, a job stored in a grid storage resource may be a job except for the pilot job for collecting information.

The pilot task for collecting the information is allocated to an agent, and the calculation result and the network resource of the agent are collected by receiving the processing result of the agent (S620). When assigning the pilot task for collecting information is performed by the task manager, the processing result of the assigned pilot task for collecting information is collected in the information collecting unit.

The collected calculation resource and the network resource are analyzed and the number of tasks to be allocated to the agent is set according to the analysis result (S 630).

The following is a general overview of the tasks processed in the grid.

(1) The agent manager calculates the maximum amount of computing resources and the amount of work to determine the number of agents to be installed. The initial value is max (min (maximum allocation value, number of work / variable 1), constant 1). Variable 1 is initially set to 100, and then changed according to the processing speed per task and the system situation. The constant 1 depends on the situation of the acquired system. According to the constant 1, the number of agents periodically checks the number of jobs to be processed to determine and manage the number of agents according to the speed and degree of progress. (2) The job manager assigns a user's work to agents installed on grid resources. The assignment of tasks is determined by the task scheduler included in the task manager. (3) Each agent installed in the grid resource gets and executes the job according to the task scheduler's assignment. (4) As the agent completes one unit task, the agent periodically transmits information such as task execution time and network status to the information system. According to this embodiment, the task scheduler determines the number of tasks to be assigned to each agent based on the task status currently assigned to the task manager and the existing grid task information stored in the information collecting unit. By default, 1 is set, and the basic processing task is adjusted according to the processing status of each agent and the network status of storage resources.

In other words, in processing the work requested by the user, the dynamic job scheduler determines the number of jobs allocated to the agent according to the time of the previous job processing of the agent and the network situation with the storage resource. The assignment of the number of work is determined by the following equation.

Variable 2 = number of work / agent

Number of Jobs = (Constant2 * Job Processing Time (Seconds)) + (Constant3 * Network Latency / Network Transfer Duration) + (Constant4 * Jobs / Agents)

The sum of the constants 2, 3, and 4 is determined by the amount of data stored for each task, and the size of each constant is determined by the situation of the system.

On the other hand, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features. Therefore, the above-described embodiments are to be understood as illustrative in all respects and not as restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Applicable to grid systems.

Specifically, the present invention can be applied to a case where the capability difference between the computing resource of each agent forming the grid resource and the network resource between the agent and the storage resource is remarkable.

Also, application is advantageous when the load of the work allocated to each agent is similar.

1 is a block diagram showing a grid job scheduling apparatus according to the present invention.

2 is a flow chart of the grid work from the user's point of view.

Figure 3 is a schematic diagram showing the process of setting the number of jobs to be assigned to the agent in the grid job scheduling apparatus according to the present invention.

4 is a flowchart illustrating a grid job scheduling method according to the present invention.

Description of the Related Art

110 ... Information Collector 130 ... Task Manager

150 ... Data Manager 170 ... Local Disk

Agent Manager 210 User Interface

230 Grid Resource 231 Agent

233 Network Resources 135 Storage Resources

Claims (8)

delete An information collector configured to collect information of a computing resource and a network resource of an agent for processing the assigned task; A task manager receiving and analyzing the calculation resource and the network resource from the information collecting unit, and setting a number of tasks to be allocated to the agent according to the analysis result; Local disk; And A data manager for storing an information collection pilot job for collecting information of the compute resource and the network resource of the agent on the local disk; , ≪ / RTI & The task manager assigns and processes the information gathering pilot task to the agent, And the information collecting unit collects information of the computing resource and the network resource of the agent by using the processing result of the information collecting pilot job. The method of claim 2, And the pilot task for collecting information is part of a task to be assigned to the agent. The method of claim 3, wherein The grid job scheduling apparatus, wherein the pilot job for collecting information is different for each agent. The method according to claim 3 or 4, At least a portion of the task to be allocated except for the pilot task for collecting information is stored in a grid storage resource. Storing a job requested by a user in a grid storage resource, and storing a part of the job on a local disk as a pilot job for gathering information of a computing resource and a network resource; Assigning the pilot task for collecting the information to an agent and receiving the processing result of the agent to collect information of the computational resource and the network resource of the agent; Analyzing the collected computing resources and network resources and setting the number of tasks to be allocated to the agent according to the analysis result; Grid job scheduling method comprising a. The method of claim 6, And the pilot task for collecting information is an initial stage task of the task to be allocated. The method of claim 6, The job stored in the grid storage resource is a job other than the pilot job for collecting information grid scheduling method.

Images