JP7385156B2 - Scheduling method, scheduler, GPU cluster system and program - Google Patents

Description

The present invention relates to a scheduling method, a scheduler, a GPU cluster system, and a program.

GPU (Graphics Processing Unit) is hardware that performs the calculation processing necessary for rendering high-definition images and videos. In recent years, GPUs have been used as computing units for machine learning and other applications. GPU clusters, which cluster multiple GPUs, are also being developed. Kubernetes exists as open source software that manages container-type GPU clusters (Non-Patent Document 1).

Kubernetes, [online], Internet <URL: https://github.com/kubernetes/kubernetes> Kubernetes, [online], Internet <URL: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/>

In conventional GPU clusters, machine learning processing is performed while reading data such as learning targets that have been uploaded to storage. GPU processing speed is fast, but storage processing speed is slow compared to GPU processing speed. As a result, idle time occurs on the GPU secured by the job as it waits to read data.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a scheduling method, scheduler, GPU cluster system, and program that reduce GPU idle time and improve GPU utilization rate. It is in.

In order to achieve the above object, one aspect of the present invention is a scheduling method performed by a GPU cluster system, in which the scheduler stores the submitted job in a first stage queue that stores jobs waiting to start fetching. and fetching a job from the first stage queue, registering it in a fetching job list, and instructing the cache cluster to start fetching the data of the job, and fetching a job whose fetched data amount exceeds a predetermined threshold. The cache cluster performs the steps of extracting the job from the job list and storing it in a second stage queue that stores jobs waiting to be deployed, and extracting the job from the second stage queue and instructing the deployment of the job. The GPU cluster performs a step of fetching the data of the job registered in the processing job list from the storage where the data is stored and storing it in the cache cluster, and the GPU cluster accesses the data of the cache cluster to process the job. Perform the steps you want to perform.

One aspect of the present invention is a scheduler for a GPU cluster system, which includes a first queue selector that stores submitted jobs in a first stage queue that stores jobs waiting to start fetching, and a first queue selector that stores jobs in the first stage queue. A first job selector that causes the cache cluster to fetch data of the job that is retrieved and registered in a fetching job list and stored in the storage, and a job whose fetched data amount exceeds a predetermined threshold; A second queue selector that extracts a job from a job list and stores it in a second stage queue that stores jobs waiting to be deployed, and a second job selector that extracts a job from the second stage queue and instructs to deploy the job. , the job deployment instruction specifies the cache cluster as a storage location for data of the job, and the GPU cluster accesses the cache cluster and executes the job.

One aspect of the present invention is a GPU cluster system including a scheduler, a cache cluster, and a GPU cluster, wherein the cache cluster stores data of a job registered in a fetching job list. The data is fetched from storage and stored in the cache cluster, and the GPU cluster accesses the data in the cache cluster to execute the job.

One aspect of the present invention is a program that causes a computer to function as the scheduler.

According to the present invention, it is possible to provide a scheduling method, a scheduler, a GPU cluster system, and a program that reduce idle time of GPUs and improve utilization rates of GPUs.

FIG. 1 is a configuration diagram of a basic GPU cluster system. FIG. 2 is a configuration diagram of the GPU cluster system of FIG. 1 that accesses user storage. FIG. 1 is a configuration diagram of a GPU cluster system according to the present embodiment. FIG. 2 is a configuration diagram of a cache cluster. FIG. 2 is a configuration diagram of a scheduler. It is a flowchart which shows the process of a 1st queue selector. 3 is a flowchart showing processing of a first job selector. It is a flowchart which shows the process of a 2nd queue selector. 7 is a flowchart showing the processing of the second job selector. FIG. 2 is a configuration diagram of a GPU cluster according to the first embodiment. FIG. 2 is a configuration diagram of a GPU cluster according to a second embodiment. FIG. 3 is a configuration diagram of a GPU cluster according to a third embodiment. FIG. 2 is a schematic diagram showing a closed area connection of method 1. FIG. 2 is a schematic diagram showing a closed area connection of method 2. FIG. 3 is a schematic diagram showing a closed area connection of method 3. FIG. 4 is a schematic diagram showing closed area connection of method 4. FIG. 7 is a schematic diagram showing a closed area connection of method 5. FIG. 7 is a schematic diagram showing a closed area connection of method 6. FIG. 7 is a schematic diagram showing a closed area connection of method 7. FIG. 2 is a sequence diagram showing the operation of a basic GPU cluster system. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 3 is a sequence diagram showing the operation of the GPU cluster according to the present embodiment. FIG. 7 is a sequence diagram showing “closed connection establishment processing” of method 2; FIG. 7 is a sequence diagram showing “closed connection release processing” of method 2; FIG. 7 is a sequence diagram showing “closed connection establishment processing” of method 7; FIG. 7 is a sequence diagram showing “closed connection release processing” of method 7; FIG. 2 is a sequence diagram showing "cluster storage processing of learning target data". FIG. 2 is a sequence diagram showing "cluster storage processing of learning target data". FIG. 3 is a sequence diagram illustrating "data access processing to cache clusters in learning processing." FIG. FIG. 2 is a sequence diagram showing "job checkpoint processing." FIG. FIG. 2 is a sequence diagram showing "job checkpoint processing." FIG. FIG. 2 is a sequence diagram showing "job checkpoint processing." FIG. FIG. 3 is a sequence diagram illustrating "job restoration processing." FIG. FIG. 3 is a sequence diagram illustrating "job restoration processing." FIG. FIG. 3 is a sequence diagram illustrating "job restoration processing." FIG. FIG. 2 is a hardware configuration diagram.

Embodiments of the present invention will be described below with reference to the drawings.

(Basic configuration of GPU cluster system)
FIG. 1 is a configuration diagram showing a schematic configuration of a basic GPU cluster system. The illustrated GPU cluster system is a GPU learning cluster system for executing learning processing using GPU.

A cluster provider (hereinafter referred to as a "company") provides users with equipment that performs learning processing on their behalf using GPU clusters. Users do not own expensive GPUs, but pay the operator a pay-as-you-go amount based on the usage time of the GPU cluster. Since learning processes such as machine learning only need to be executed once, it is cheaper for users to pay a pay-per-use amount than to purchase an expensive GPU.

On the other hand, increasing the utilization rate of GPUs is the key to maximizing profits for operators. Therefore, GPU cluster systems are required to be able to execute a variety of jobs (that is, job virtualization) and to deploy jobs at high speed.

An overview of the basic GPU cluster operation will be explained with reference to FIG. Here, we use a container-type cluster that allocates GPU resources for each job execution. In response to a user's instruction, the user terminal 5 stores data and the like to be a learning target in the cluster shared storage 4A, which is instructed by the GPU cluster provider (S1A). The user terminal 5 registers a desired learning process job to the scheduler 1A according to the user's instruction (S2A). The scheduler 1A schedules jobs received from multiple user terminals 5 based on priorities, estimated processing time, etc., and instructs the master 2A to execute the jobs as soon as GPU resources are secured (S3A).

The master 2A deploys the job to the node, attaches the GPU, and causes the GPU to execute learning processing (S4A). That is, the master 2A generates a virtual environment for executing a learning/inference program for each job, and attaches a GPU to the virtual environment. The master 2A releases the GPU when the job is completed. The GPU performs learning processing while reading the learning target data uploaded in advance to the cluster shared storage 4A, and stores the results of the learning processing in the cluster shared storage 4A (S5A). When the user finishes executing his or her job, the user can obtain the execution results of the learning process by accessing the cluster shared storage 4A.

In the case of the basic GPU cluster system shown in Figure 1, it is difficult to deal with the following assumed situations and constraints.

(1) If the storage speed (data transfer speed) is slower than the processing speed of the learning program, idle time occurs in the GPU secured by the job due to insufficient storage speed in the process of S5A. Big data is stored unprocessed or almost unprocessed in the cluster shared storage 4, which is a large-capacity distributed storage using Ceph or the like. Distributed storage is characterized by the fact that it does not slow down even when the capacity increases due to the effect of distributed parallelization, but the speed does not increase dramatically, and the performance is only a few hundred MB/s at most.

High-speed storage that matches the processing speed of GPUs is extremely expensive, so expensive storage with the capacity to store all of the big data is not available. On the other hand, all of big data is not needed at the same time.

(2) There are cases where it is not possible to upload all the learning target data to the cluster shared storage 4, or the learning target data is so large that it is not practical to upload all the learning target data.

In such a case, as shown in FIG. 2, the user terminal 5 stores the learning target data in the user storage 6A at the user base (S1A'), and the job of the node 3A connects to the user storage 6A in a closed area and Directly access storage 6A (S5A'). However, since there is a communication section from the user storage 6A to the node 3A, and the speed of the user storage 6A is slow, the data transfer speed becomes slower than the processing speed of the learning program, resulting in GPU idle time.

(3) In order to operate the GPU efficiently, there are cases where data accumulation and learning processing need to be executed in parallel.

The GPU cluster system of this embodiment that can cope with such a situation will be described below.

(GPU cluster system of this embodiment)
FIG. 3 is a configuration diagram showing a schematic configuration of the GPU cluster system of this embodiment. The GPU cluster system of this embodiment is a GPU learning cluster system for executing learning processing using GPU. Learning processing is reading the learning target data and performing processing such as machine learning. At this time, rather than reading all of the learning target data at once, the learning process is performed while dividing it into blocks or files and reading them one after another.

The illustrated GPU cluster system includes a scheduler 1, a master 2, a node 3, a cluster shared storage 4, and a cache cluster 7. Here, we use a container-type GPU cluster that allocates GPU resources for each job execution. The user base may include a user storage 6 in which the user stores learning target data.

The GPU cluster system of this embodiment includes a high-cost and high-speed cache cluster 7 (cache), and the scheduler 1 schedules the cache cluster 7 and the GPU simultaneously. Large-capacity data is normally stored in low-cost, low-speed storage (cluster shared storage 4, user storage 6), and is stored in cache cluster 7 when a job is executed. As a result, in this embodiment, since the GPU reads data from the high-speed cache cluster 7, it is possible to avoid a situation where the GPU becomes idle while waiting to read data.

The scheduler 1 receives jobs submitted from the user terminal 5. The scheduler 1 monitors the free status of GPU resources in the GPU cluster, and if there is free space, instructs the master 2 to deploy the job (deploy it to the execution environment). That is, the scheduler 1 instructs the master 2 to execute the job.

Master 2 (Master) manages node 3 (Node) and deploys jobs. When instructed to deploy a job by the scheduler 1, the master 2 constructs a virtual environment such as a container defined in the job on the node 3, and causes the program defined in the job to be executed in the virtual environment. Master 2 deletes the virtual environment when the program defined in the job is completed.

A plurality of GPUs are pooled in the node 3 (Node). The GPU executes the job when attached to master 2. A job defines a program (for example, a learning or inference program) that the user wants to execute and an execution environment for the program. Specifically, a job includes one or more programs to be executed and their order. Further, a job includes an environment for executing a program (virtual environment, runtime, OS, distribution, library, etc.). For example, a job includes the container image file name, VM (Virtual Machine) image file name, etc. as the environment. Further, if necessary, the job may include a procedure for automatically constructing the above environment, and the job may automatically generate an image of the execution environment. The job of this embodiment includes a main container (Main Container), and may also include containers other than the main container. The main container is a container of a virtual environment in which the learning program of this embodiment is executed. Note that in this embodiment, a container is used as a form of implementation of the virtual environment, but a VM may also be used.

The cluster shared storage 4 (Cluster Shared Storage) is a storage system that stores data. For example, the cluster shared storage 4 stores learning target data and execution results. The cluster shared storage 4 can be accessed from the job's virtual environment. A user can directly or indirectly store learning target data read by a job in the cluster shared storage 4 by some means. In the cluster shared storage 4, storage technologies such as Ceph, GlusterFS, Swift, and RAID are assumed to be used to store a large amount of learning target data. Ceph (https://ceph.io/) and GlusterFS (https://www.gluster.org/) are open source distributed storage software.

The cache cluster 7 (Cluster Shared Storage) will be described later.

Next, an overview of the operation of the GPU cluster system of this embodiment will be explained with reference to FIG. Here, we use a container-type cluster that allocates GPU resources for each job execution.

In this embodiment, the cache cluster 7 fetches data from the cluster shared storage 4 or the user storage 6. Note that "fetch" for reading data will also be referred to as "cache" hereinafter.

In response to a user's instruction, the user terminal 5 stores learning target data and the like in the cluster shared storage 4 or the user storage 6 instructed by the provider providing the GPU cluster system (S1). The user terminal 5 registers a desired learning process job to the scheduler 1 according to the user's instruction (S2). The scheduler 1 instructs the cache cluster 7 to cache the data (S3). The cache cluster 7 starts fetching the learning target data from the cluster shared storage 4 or the user storage 6 (S4). The scheduler 1 schedules jobs received from multiple user terminals 5 (users) based on registration order, priority, required resource amount (number of GPUs, number of CPUs, etc.), expected processing time, etc., and secures GPU resources. Then, the master 2 is instructed to execute the job (S5). The required resource amount may be included in the metadata of the job in advance by the user and notified to the scheduler 1, or the scheduler 1 may estimate it from the contents of the job.

The master 2 deploys the job to the node, attaches the GPU, mounts the cache area of the cache cluster 7, and causes the GPU to execute the learning process of the job (S6). That is, the master 2 generates a virtual environment for executing a learning/inference program for each job, and attaches a GPU to the virtual environment. Master 2 releases the GPU when the job is completed. The GPU performs learning processing while reading out the learning target data cached in the cache area, and stores the results of the learning processing in the cache cluster 7 or the cluster shared storage 4 (S7). When the user finishes executing his or her job, the user can obtain the execution results of the learning process by accessing the cache cluster 7 or the cluster shared storage 4. The scheduler 1 deletes the data in the cache area after the job ends (S8).

FIG. 4 is a configuration diagram of the cache cluster 7. As shown in FIG. The illustrated cache cluster 7 includes a VPN connection section 71 (VPN Function), a cache management section 72 (Cache Manager), and one or more storages 73 (Storage).

The VPN connection unit 71 starts or stands by for a closed connection, and establishes a closed connection.

The cache management unit 72 composes one or more storages 73 into a cluster. The cache management unit 72 accesses the origin (original) storage (cluster shared storage 4, user storage 6) using a file sharing protocol, etc., and caches the data owned by the origin while sharing the data with the request source. It has a transparent caching function. When a request source requests data from the cache cluster 7, the cache management unit 72 determines whether the requested data has already been cached. If the data has been cached, the data is returned to the request source. If the data is not already cached, it requests the data from the origin storage and returns the data passed from the origin storage to the request source. The cache management unit 72 has a function of operating the cluster shared storage 4 and the user storage 6.

The storage 73 stores data cached from the origin storage. For the storage 73, high-speed storage such as NVMe or NVDIMM is used. Note that the VPN connection unit 71 may not be included in the cache cluster 7 but may exist independently from the cache cluster 7 in the GPU cluster system. Further, the cache cluster 7 may include the cluster shared storage 4.

FIG. 5 is a configuration diagram of the scheduler 1. The scheduler 1 includes a first stage queue 10, a second stage queue 20, a fetching job list 30 (FJL), an accounting DB 31, and a GPU usage monitoring unit. 32 (GPU Utilization Monitor). The account DB 31 manages each user's GPU usage. The account DB 31 may be installed outside the scheduler 1 instead of inside the scheduler 1. Furthermore, the account DB 31 may be an existing user database of the business operator. The GPU usage monitoring unit 32 acquires the GPU usage from the master 2 or the node 3 and monitors the GPU usage.

The first stage queue 10 stores jobs waiting to start fetching. The first stage queue 10 includes a first queue selector 11 (Queue Selector 1), a plurality of job queues 13-15, and a first job selector 12 (Job Selector 1). The first queue selector 11 stores the job submitted from the user terminal 5 in one of the job queues 13-15 of the first stage queue 10 that stores jobs waiting to start fetching. The processing of the first queue selector 11 will be described later.

The first job selector 12 extracts a job from the first stage queue, registers it in the fetching job list, and causes the cache cluster to start fetching data of the job stored in the storage. In this embodiment, the first job selector 12 takes out the jobs stored in the job queue 13-15 according to their priorities and registers them in the fetching job list 30. The first job selector 12 also queries the account DB 31 about the user's current GPU usage, and depending on the user's current usage, the first job selector 12 assigns jobs that exceed the fairness quota or user quota to the corresponding job queue. Relocate to. The processing of the first job selector 12 will be described later.

The first stage job queue includes a job queue 13 (JQ), an over-fairness-quota job queue 14 (OFJQ), and over-user jobs. Queue 15 (Over User-quota Job Queue (hereinafter referred to as "OUJQ")) is included.

Jobs that do not exceed the fairness quota and user quota are stored in JQ13. JQ13 is provided for each job class (priority) k. Here, class k is 1≦k≦n, the highest priority class is k=1, and the lowest priority class is k=n. JQ13 of class k is sometimes written as "JQ k".

OFJQ14 stores jobs that exceed the GPU fairness quota allocated to each user from the viewpoint of fairness. This fairness quota is set by the operator to prevent a situation where one user monopolizes the GPU and other users cannot use the GPU, and to fairly allocate GPUs to each user. This sets an upper limit on the amount. The allocated amount is an allocated amount for a predetermined period such as one month, for example. Like JQ13, OFJQ14 is provided for each job class k, where k is 1≦k≦n. The job queue 14 of class k may be written as "OFJQ k".

OUJQ15 stores jobs that exceed the user allocated amount of GPU set by the user. This user quota is an upper limit set by the user for his/her own GPU usage in order to keep the GPU usage fee within the budget. The allocated amount is an allocated amount for a predetermined period such as one month, for example. Jobs stored in OUJQ15 are not deployed and fetched. When the user quota is changed or the current usage amount is updated, the first job selector 12 takes out the job from the beginning of the OUJQ 15, and the first queue selector 11 assigns the job to the job of the corresponding class. Assign to queue 13. Examples of cases in which the current amount of usage is updated include when an upper limit for monthly usage is set, or when the usage amount is updated to 0 in the next month.

The first job selector 12 registers the job of JQ13 in the FJL30 with priority over the job of OFJQ14.

FJL30 is a list in which jobs that start fetching data are registered. The cache cluster 7 fetches (prefetches) jobs registered in the FJL 30. After registering the job in the FJL 30, the first job selector 12 may instruct the cache cluster 7 to start fetching the job added to the FJL 30. The cache cluster 7 may periodically check the FJL 30 and, when a new job is registered, start fetching the job. Jobs whose fetched data amount exceeds a predetermined threshold are moved to the second stage queue 20. The threshold value will be described later. A suspended job may be registered in the FJL 30.

The second stage queue 20 stores jobs waiting to be deployed. The second stage queue 20 includes a second queue selector 21 (Queue Selector 2), a plurality of job queues 23-25, and a second job selector 22 (Job Selector 2).

The second queue selector 21 extracts a job whose fetched data amount exceeds a predetermined threshold from the FJL 30 and stores it in one of the queues 23-25 of the second stage queue 20. The processing of the second queue selector 21 will be described later. The second job selector 22 takes out a job from one of the queues 23-25 of the second stage queue 20 and instructs the deployment of the job.

The second stage job queue includes a restore queue 23 (Restore Queue (hereinafter referred to as "RQ")), a deploy queue 24 (hereinafter referred to as "DQ"), and an over fairness queue 25 (Over Fairness-quota). Deploy Queue (hereinafter referred to as (OFDQ)) is included.

RQ23 stores jobs that are waiting to be restored and whose fetched data amount exceeds a threshold. DQ24 stores jobs waiting to be deployed whose fetched data amount exceeds a threshold. The OFDQ 25 stores jobs in which the current GPU usage of the user (job owner) of the job exceeds the fairness quota among the jobs whose fetched data amount exceeds the threshold. A job that exceeds the fairness quota becomes a target for deployment if there is free space on the GPU and there are no other jobs (jobs RQ23 and DQ24). If there are other jobs, the other jobs have priority. In the next month, when the usage amount of the user is reset to 0 and the excessive state is resolved, the first job of OFDQ 25 is taken out and stored in RQ 23 or DQ 24.

The second job selector 22 instructs to deploy the job of RQ23 with priority over the job of DQ24, and instructs the job of DQ24 to be deployed with priority over the job of OFDQ25. Furthermore, when both RQ23 and DQ24 are empty, the second job selector 22 activates the second queue selector 21, and the second job selector 22 stores the first job of FJL30 or the amount of data fetched in FJL30. The job with the largest number of jobs may be stored in any of RQ23, DQ24, and OFDQ25. Furthermore, when retrieving a job from RQ23, the job immediately after being suspended may not be subject to deployment instructions until a certain amount of time or a certain amount of fetching is performed, so as to avoid repeating restoration and suspending in a short period of time.

The threshold value of the fetched data amount when retrieving a job from the FJL 30 may be calculated, for example, by the following method.

In the first method, a value defined by a business operator or a user is used as a threshold value. For example, let it be 10% of the data amount.

The second method calculates the threshold value from the job definition. Specifically, the calculation amount order is calculated from the depth of the loop processing of the program included in the job definition and the number of instructions, and the calculation amount order is divided into stages depending on the size of the calculation amount order, and a threshold value is determined for each stage. Furthermore, the larger the order of calculation amount is, the lower the amount of data processing per time (data processing speed) is, so the larger the order of calculation amount is, the smaller the threshold value is.

In the third method, a threshold value is calculated from the job execution status up to a checkpoint, which will be described later. Specifically, the data processing speed Vp and the fetch speed Vf are calculated from the previous execution status.

In the case of Vf≧Vp, the threshold value is set as Vf×M. M is an arbitrary value.

When Vf<Vp, threshold value=(1−Vf/Vp)×S+Vp×M. S is the amount of remaining data that has not been processed, and M is an arbitrary value.

FIG. 6 is a flowchart showing the processing of the first queue selector 11. When the first queue selector 11 receives a job (S11), it acquires the priority class of the user who is the owner of the job from the account DB 31 (S12). Here, the priority class is k (S13). The first queue selector 11 compares the GPU fairness allocation with the user's current usage (S14), and if the current usage does not exceed the fairness allocation (S15: true), the GPU The allocated amount and the user's current usage amount are compared (S16).

If the current usage does not exceed the user quota (S17: true), the first queue selector 11 stores the job received in S11 at the end of JQ k13 of priority class k (S18). If the current usage exceeds the user quota (S17: false), the first queue selector 11 stores the job received in S11 at the end of OUJQ15 (S19). If the current usage exceeds the fairness quota (S15: false), the first queue selector 11 stores the job received in S11 at the end of OFJQ k14 of priority class k (S20).

FIG. 7 is a flowchart showing the processing of the first job selector 12. The process in FIG. 7 is started when a job is submitted to the first queue selector 11 as a trigger. Further, the process in FIG. 7 is started when the second queue selector 21 detects that a space becomes available in the FJL 30.

If there is a vacancy in the FJL 30 (S31: true), the first job selector 12 sets k (priority class) to 1 (S32). If there is a job in JQ k13 with k=1 (S33: true), the first job selector 12 extracts the job from JQ k13 (S34) and compares the fairness quota of the job owner with the current usage amount. (S35). If the current usage does not exceed the fairness quota (S36: true), the first job selector 12 compares the job owner's user quota with the current usage (S37).

If the current usage does not exceed the user quota (S38: true), the first job selector 12 stores the job extracted in S34 at the end of the FJL 30 (S39). If the current usage exceeds the user quota (S38: false), the first job selector 12 stores the job retrieved in S34 at the end of the OUJQ 15 via the first queue selector 11 (S40). If the current usage exceeds the fairness quota (S36: false), the first job selector 12 stores the job retrieved in S34 via the first queue selector 11 at the end of OFJQ k14 of priority class k. (S41).

If there is no job in JQ k13 with k=1 (S33: false), the first job selector 12 adds 1 to k (S42), and if k≦n (S43::true), returns to S33 and thereafter. Process. If k>n (S43: false), the first job selector 12 sets k to 1 (S44). If there is a job in OFJQ k14 with k=1 (S45: true), the first job selector 12 retrieves the job from OFJQ k14 (S48), and calculates the user quota and current usage of the job owner of the retrieved job. are compared (S37), and the process proceeds to S38. The processing from S38 onwards is the same as described above, so a description thereof will be omitted.

If there is no job in OFJQ k14 with k=1 (S45: false), the first job selector 12 adds 1 to k (S46), and if k≦n (S47: true), returns to S45 and performs the subsequent Perform processing. If k>n (S47: false), the first job selector 12 ends the process.

FIG. 8 is a flowchart showing the processing of the second queue selector 21. The process in FIG. 8 is executed periodically. The second queue selector 21 sets the variable i to 1 (S51), and if the i-th job of the FJL 30 exists (S52: true), the data amount of the fetched learning target data of the i-th job is the threshold value. is exceeded (S53). The second queue selector 21 inquires of the cache cluster 7 (cache management unit 72) about the amount of fetched data. If the fetched data amount does not exceed the threshold (S53: false), the second queue selector 21 adds 1 to i (S54), returns to S52, and performs the subsequent processing.

If the fetched data amount exceeds the threshold (S53: true), the second queue selector 21 takes out the i-th job of the FJL 30 and dequeues it (S55). The second queue selector 21 checks the metadata of the retrieved job (S56), and if the job is in a suspended state (temporarily stopped state) (S57: true), stores the job in the RQ 23 (S63). The second queue selector 21 activates the first job selector 12 because there is a free space in the FJL 30 (S61).

If the retrieved job is not in a suspended state (S57: false), the second queue selector 21 checks the job owner's fairness quota and reduced current usage in order to determine whether to proceed with job control. (S58). If the current usage does not exceed the fairness quota (S59: true), the second queue selector 21 stores the job in the DQ 24 (S63) and activates the first job selector 12 (S61). If the current usage exceeds the fairness quota (S59: false), the second queue selector 21 stores the job in the OFDQ 25 (S62) and activates the first job selector 12 (S61).

The second queue selector 21 also inquires of the GPU usage monitoring unit 32 about the GPU usage. The GPU usage monitoring unit 32 acquires the GPU usage from the master 2 or node 3 and sends it to the second queue selector 21 . If there is space in the GPU (S64: true), RQ23 is empty (S65: true), DQ24 is empty (S66: true), and the first job exists in FJL30 (S67: true), the second queue The selector 21 takes out the first job of the FJL 30 (S68) and proceeds to S56. In order to maximize the usage rate of the GPU, in this embodiment, when there are no RQ23 and DQ24 jobs to be executed, even the FJL30 job with insufficient fetching is deployed. That is, when both RQ23 and DQ24 are empty, the second queue selector 21 takes out the first job of FJL30 even if the fetch is insufficient and enqueues it into one of the queues 23-25. If at least one of S64 to S67 is false, the second queue selector 21 activates the first job selector 12 (S61).

Note that, depending on the I/O speed and communication speed of the storage where the learning target data is stored, the amount of fetched data for the first (first) job among the jobs stored in FJL30 may not be the largest. . Considering such a case, in S68, the second queue selector 21 extracts the job with the largest amount of data fetched from FJL30, proceeds to S56, and stores the job in one of RQ23, DQ24, and OFDQ25. You may let them. That is, the second queue selector 21 may take out the job that has been fetched the most among the jobs in the FJL 30.

FIG. 9 is a flowchart showing the processing of the second job selector 22. The GPU usage monitoring unit 32 activates the second job selector 22 when there is a vacant GPU, and the process shown in FIG. 9 is performed. If RQ23 is not empty (S71: false), the second job selector 22 extracts one job from RQ23 and stores it in J (S72). If RQ23 is empty (S71: true) and DQ24 is not empty (S75: false), the second job selector 22 extracts one job from DQ24 and stores it in J (S76). If DQ24 is empty (S75: true), FJL30 is empty (S77: true), and OFDQ25 is not empty (S78: false), the second job selector 22 takes out one job from OFDQ25 and stores it in J. (S79).

After S72, S76, and S79, the second job selector 22 instructs the master 2 to deploy J (S73), and activates the second queue selector 21 (S74). If the OFDQ 25 is empty (S78: true), the second job selector 22 activates the second queue selector 21 (S74). If the FJL 30 is not empty (S77: false), the second job selector 22 activates the second queue selector 21, waits for the second queue selector 21 to complete its operation (S80), and proceeds to S71. In this way, when all of RQ23, DQ24 and OFDQ25 are empty, the second job selector 22 activates the second queue selector 21 and stores the first job of FJL30 in any of RQ23, DQ24 and OFDQ25. .

(Example 1)
FIG. 10 is a configuration diagram of a GPU cluster according to the first embodiment. In this embodiment, the learning target data is stored in advance in the low-speed cluster shared storage 4 (distributed storage). When the execution of a job approaches, the cache cluster 7 (cache management unit 72) prefetches learning target data from the cluster shared storage 4 to the cache cluster 7. When the GPU becomes free, the master 2 mounts the area of the cache cluster 7 on the node 3. Mounting of the cache area is implemented using RDMA-fs (a mechanism for converting data on an RDMA device into a file system), NFS over RDMA, GlusterFS, etc. Bandwidth is guaranteed for the RDMA transfer path using TSN (Time Sensitive Networking), etc. In this embodiment, a high-speed, bandwidth-securing network such as a lossless DC fabric is constructed, and data is transferred using various switches (SW) such as a spine switch (Spine SW).

In this embodiment, (1) the scheduler 1 instructs the cache cluster 7 to prefetch the data of the job while the job is on standby. Thereby, the cache cluster 7 prefetches data from the cluster shared storage 4 according to the instruction. (2) Scheduler 1 instructs master 2 to deploy the job, and master 2 assigns the job to the GPU. (3) The master 2 mounts the cache area of the cache cluster 7 using RDMA-fs or the like. (4) GPU executes the job. (5) The scheduler 1 deletes the cache data in the cache cluster 7 after executing the job.

(Example 2)
FIG. 11 is a configuration diagram of a GPU cluster according to the second embodiment. In this embodiment, online connection is made to the user storage 6 at the user base. That is, in this embodiment, the learning target data stored in the low-speed user storage 6 is connected online.

In this example, in the GPU cluster system, a high-speed, bandwidth-securing network such as a Lossless DC fabric is constructed as in Example 1, and data is transferred using various switches (SW) such as spine switches (Spine SW). Forward. Connect the Access/Metro network with a switch such as Border Leaf to build a data transfer path (VPN, dedicated line, etc.) between the GPU cluster system and the user location. The operation of this embodiment is as follows.

(1) The cache cluster 7 (cache management unit 72) transfers data in the user storage 6 at the user base and prefetches it to the memory (NV-DIMM) of the cache cluster 7. Note that this does not correspond to downloading because it only places part of the data in the cache memory.

(2) When a certain amount of cache data is accumulated in the memory of the cache cluster 7, the GPU executes the job.

(3) When the GPU runs out of cache data, the GPU temporarily suspends the job and releases resources. By using technology such as CRIU (Checkpoint/Restore In Userspace) to release resources, there is no need to implement a temporary suspension function in the job program. CRIU is a technology that pauses, saves, and resumes processes without terminating them.

(4) The cache cluster 7 writes the process data being processed into the cache cluster 7.

(5) Once a certain amount of cache data has accumulated in the memory of the cache cluster 7, secure a GPU.

(6) Write back the process data and restart (restore) job processing.

(7) Terminate when job processing is completed. If it is not completed, return to (3) and repeat the process.

(Example 3)
FIG. 12 is a configuration diagram of a GPU cluster according to the third embodiment. In this embodiment, a plurality of data centers 40 exist in a distributed manner. The data center 40 includes a GPU cluster including a plurality of masters 2 and nodes 3, a cache cluster 7, and a cluster shared storage 4. The data center 40 does not need to include the cluster shared storage 4.

The scheduler 1 places jobs in GPU clusters that are close to the user base. When the user himself/herself uploads data to the cluster shared storage 4, the scheduler 1 selects a GPU cluster as close as possible to the cluster shared storage 4 to which the user has uploaded the data.

(Closed connection method)
Below, a closed connection system between the user base and the cache cluster 7 will be described when the cache cluster 7 fetches the learning target data stored in the user storage 6 of the user base in the second embodiment.

FIG. 13 is a schematic diagram showing closed area connection of method 1. In this method, the user storage 6 has a closed connection function and waits for a closed connection from the cache cluster 7. When prefetching learning target data, the cache cluster 7 starts a closed connection to the user storage 6. When acquisition of the learning target data is completed, the cache cluster 7 releases the closed connection. As a result, the user storage 6 returns to the closed connection standby state. The user storage 6 is always in a state of waiting for a closed connection. A subscriber line termination equipment (hereinafter referred to as "CPE") is installed at the user location. Users need to negotiate and decide on the settings for closed connections with the GPU cluster system provider in advance. The user needs to configure his/her own user storage 6 for closed connection with the cache cluster 7 .

FIG. 14 is a schematic diagram showing closed area connection of method 2. In this system, the CPE 8 at the user site includes a VPN connection section and an API (control section) for responding to control from the scheduler 1. This method configures closed connections on demand. When the user registers a job in the scheduler 1, the user includes connection information to the API of the CPE 8 in the job. The scheduler 1 instructs the cache cluster 7 to wait for a closed connection from the CPE 8. Upon receiving an instruction from the scheduler 1, the CPE 8 requests a closed connection to the specified connection destination (cache cluster 7). When the closed connection is established, the cache cluster 7 copies the learning target data on the user storage 6 to the GPU cluster system. The replication destination is either the cache cluster 7 or the cluster shared storage 4. When the job is completed, the scheduler 1 instructs the CPE 8 to delete the closed connection settings.

FIG. 15 is a schematic diagram showing closed area connection of method 3. In this method, as in method 2, the CPE 8 includes a VPN connection section and an API (control section). In this method, the scheduler 1 causes the CPE 8 to wait for a closed connection, and instructs the cache cluster 7 to start a closed connection. This method configures closed connections on demand. When the user registers a job in the scheduler 1, the user includes connection information to the API of the CPE 8 in the job. The scheduler 1 instructs the CPE 8 to wait for a closed connection from the cache cluster 7. Upon receiving an instruction from the scheduler 1, the cache cluster 7 requests a closed connection to the specified connection destination (CPE 8). When the closed connection is established, the cache cluster 7 copies the learning target data on the user storage 6 to the GPU cluster system. The replication destination is either the cache cluster 7 or the cluster shared storage 4. When the job is completed, the scheduler 1 instructs the CPE to delete the closed connection settings.

FIG. 16 is a schematic diagram showing closed area connection of method 4. In this method, a virtualized subscriber line termination equipment (hereinafter referred to as "vCPE") 92 is installed within the carrier network. The vCPE 92 has a VPN connection unit and an API (control unit) that is controlled by the scheduler 1.

This method configures closed connections on demand. When a user registers a job in the scheduler 1, the user includes line identification information for identifying the line to which the user storage 6 is connected to the job. The scheduler 1 instructs the cache cluster 7 to wait for a closed connection from the vCPE 92. In response to an instruction from the scheduler 1, the vCPE 92 requests a closed connection to the specified connection destination (cache cluster 7). When the closed connection is established, the cache cluster 7 copies the learning target data on the user storage 6 to the GPU cluster system. The replication destination is either the cache cluster 7 or the cluster shared storage 4. When the job is completed, the scheduler 1 instructs the vCPE 92 to release the closed connection. At the user base, an optical line terminal unit (hereinafter referred to as "ONU") 91 or a modem is installed and connected to the vCPE 92. ONU91 etc. provide layer 2 connection (Ethernet etc.) with vCPE92.

FIG. 17 is a schematic diagram showing closed area connection of method 5. In this method, a vCPE 92 is provided in the carrier network as in method 4. ONU91 etc. are installed at the user base. The ONU 91 and the like are connected to the vCPE 92 and provide layer 2 connection with the vCPE 92. In this method, the scheduler 1 causes the vCPE 92 to wait for a closed connection, and instructs the cache cluster 7 to start a closed connection.

This method configures closed connections on demand. When the user registers a job in the scheduler 1, the user includes line identification information for identifying the line to which the user storage 6 is connected to the job. The scheduler 1 instructs the vCPE 92 to wait for a closed connection from the cache cluster 7. Upon receiving the instruction from the scheduler 1, the cache cluster 7 requests a closed connection to the specified connection destination (vCPE 92). When the closed connection is established, the cache cluster 7 copies the learning target data on the user storage 6 to the GPU cluster system. The replication destination is either the cache cluster 7 or the cluster shared storage 4. When the job is completed, the scheduler 1 instructs the vCPE 92 to release the closed connection.

FIG. 18 is a schematic diagram showing closed area connection of method 6. In this method, a vCPE 92 is provided in the carrier network as in method 4. A CPE 8 similar to method 1 is installed at the user base and connected to the vCPE 92.

This method configures closed connections on demand. The scheduler 1 instructs the vCPE 92 to start waiting for closed connection requests from the cache cluster 7 and CPE 8. The scheduler 1 instructs the cache cluster 7 to establish a closed connection to the vCPE 92. The scheduler 1 instructs the CPE 8 to make a closed connection to the vCPE 92. When the closed connection is established, the cache cluster 7 copies the learning target data on the user storage 6 to the GPU cluster system. The replication destination is either the cache cluster 7 or the cluster shared storage 4. When the job is completed, the scheduler 1 instructs the vCPE 92 and CPE 8 to release the closed connection.

As a pattern of vCPE92 instances, those deployed in advance may be pooled, and the vCPE92 closest to the user's base may be assigned when prefetching of the learning target data of a job is started. Furthermore, an instance of vCPE may be deployed when prefetching of learning target data for a job is started.

FIG. 19 is a schematic diagram showing closed area connection of method 7. In this method, a closed connection is established using a gateway device (hereinafter referred to as "GW") 93 that relays PPPoE etc. in the carrier network to the ISP. The GW 93 of this method is added with a connection section for making a closed connection with the cache cluster 7 and an API (control section) that supports control from the scheduler 1. Normally, when accessing the Internet, a tunneling protocol such as PPPoE or DS-lite is used to connect to an ISP via a relay device within the carrier network. The CPE 8 installed at the user base is a device that terminates these protocols on the subscriber side, and in most cases always makes a closed connection to the GW 93. The scheduler 1 establishes a closed connection between the GW 93 and the cache cluster 7, and causes the GW 93 to relay communication between the user storage 6 and the cache cluster 7. Communication between devices other than the cache cluster 7 and the CPE 8 is transferred to the tunnel to the ISP as usual and is used as Internet access 94.

This method configures closed connections on demand. When setting up a closed connection, the scheduler 1 instructs the GW 93 to start waiting for a closed connection request from the cache cluster 7. The GW93 to be instructed is specified from line identification information, etc. The scheduler 1 requests the cache cluster 7 to establish a closed connection to the GW 93. When the closed connection is established, the GW 93 relays communication between the user storage 6 and the cache cluster 7, and a communication path is established.

(GPU cluster system operation)
The operation of the GPU cluster system will be explained below.

FIG. 20 is a sequence diagram showing the operation of the basic GPU cluster system shown in FIG. 1. The user uploads the learning target data to the cluster shared storage 4 (S101) and registers the job in the scheduler 1 (S102). The job registration data includes a job definition, a storage location of learning target data, authentication information such as a user ID, and the like. The scheduler 1 authenticates the user using the authentication information, but the authentication process will be omitted here.

When the job is registered, the scheduler 1 checks the GPU availability status (GPU operating status) with the master 2 (S103), and acquires the GPU availability status etc. from the master 2 (S104). The scheduler 1 schedules a job using GPU availability information (S105), and instructs the master 2 to deploy the job (S106). This deployment instruction includes job definitions, storage locations for learning target data, authentication information, etc. Master 2 instructs node 3 to deploy the job (S107). This deployment instruction includes job definitions, storage locations for learning target data, etc.

Node 3 starts executing the job and creates a virtual environment for the job (S108). Specifically, the node 3 generates a name space such as a network namespace and a virtual environment such as a container. Further, the node 3 is set so that the job can access the learning target data. As a result, the storage location (cluster shared storage 4) of the learning target data becomes accessible from the job.

The job starts the learning process (S109) and executes the learning process while accessing the learning target data. The job writes the learning results to the cluster shared storage 4 (S110). The learning results may be written out one after another or all at once at the end. When the learning process ends (S111), the job reports execution completion to the node 3 (S112). The node 3 deletes the virtual environment of the job, etc. (S113). Further, the node 3 also deletes the virtual network for the job. When the execution of the job is completed, the node 3 reports the completion of the job execution to the master 2 (S114). The master 2 reports job completion to the user as necessary. Alternatively, the user may inquire of the scheduler 1 or master 2 about job completion.

FIGS. 21A, 21B, and 21C are sequence diagrams showing the operation of the GPU cluster of this embodiment. These are sequence diagrams when the cache cluster 7 fetches and uses the learning target data uploaded to the cluster shared storage 4.

When uploading the learning target data before job registration, the user uploads the learning target data stored in the user storage 6 to the cluster shared storage 4 (S131), and registers the job in the scheduler 1 (S132). The job registration data includes a job definition, a storage location of learning target data, authentication information such as a user ID, and the like. The storage location of the learning target data is the cluster shared storage 4 if pre-uploaded, and the user storage 6 if not pre-uploaded. Furthermore, if the job is not uploaded in advance, the job registration data includes closed connection information to the user storage 6 and the like. The user authentication process in the scheduler 1 will be omitted.

If the learning target data is not uploaded in advance, the "closed connection establishment process" A, the "cluster storage process of learning target data" B, and the "closed connection cancellation process" C described below are performed without performing S131. will be held. "Closed connection establishment process" A connects a closed connection or a closed path between the user base and the cache cluster 7 under the control of the scheduler 1. “Cluster storage processing of learning target data” B stores the learning target data on the user storage 6 on the cache cluster 7 via the closed connection or closed path established between the user base and the cache cluster 7 . “Closed connection release processing” C releases the closed connection or closed path established between the user base and the cache cluster 7 under the control of the scheduler 1.

The scheduler 1 instructs the cache cluster 7 to prefetch the learning target data (S133). That is, the scheduler 1 instructs the cache cluster 7 to store the learning target data in a predetermined storage location. The cache cluster 7 starts fetching the learning target data on the cluster shared storage 4 (S134).

When fetching all the learning target data, the cache cluster 7 reports the completion of prefetching the learning target data to the scheduler 1 (S135). The scheduler 1 checks the GPU availability status with the master 2 (S136) and obtains the GPU availability status and the like from the master 2 (S137).

When not fetching all learning target data, that is, when starting a job speculatively without waiting for the cache data of all learning target data, scheduler 1 starts the subsequent processing without waiting for the completion of prefetching. Execute. The scheduler 1 checks the GPU availability status with the master 2 (S138), and acquires the GPU availability status etc. from the master 2 (S139). Further, the scheduler 1 checks the amount of data that has been fetched with the cache cluster 7 (S140), and obtains the amount of data that has been fetched from the cache cluster 7 (S141). The scheduler 1 may perform in parallel the process of checking the idle state of the GPU in S138 and S139, and the process of checking the progress of fetching the learning target data in S140 and S141.

The scheduler 1 schedules the job using GPU availability information and the like (S142), and instructs the master 2 to deploy the job (S143). This deployment instruction includes a job definition, a storage location of learning target data, authentication information such as a user ID, and the like. Master 2 instructs node 3 to deploy the job (S144). This deployment instruction includes job definitions, storage locations for learning target data, etc.

Node 3 starts executing the job and creates a virtual environment for the job (S145). Specifically, the node 3 generates a name space such as a network namespace and a virtual environment such as a container. Further, the node 3 is set so that the job can access the learning target data. As a result, the storage location (cache cluster 7) of the learning target data becomes accessible from the job.

The job starts learning processing (S146), performs "data access to cache cluster in learning processing" D, which will be described later, and executes learning processing while accessing the learning target data. The job writes the learning results to the cache cluster 7 (S147). By writing the learning results to the cache cluster 7, the cache management unit 72 transparently writes the learning results to the cluster shared storage 4. Further, the job may directly write the learning results to the cluster shared storage 4. In that case, when creating the virtual environment for the job in S145. Settings are made so that jobs can access the cluster shared storage 4.

When the learning process ends (S148), the job reports execution completion to node 3 (S149). The node 3 deletes the virtual environment of the job, etc. (S150). Further, the node 3 also deletes the virtual network for the job. When the execution of the job is completed, the node 3 reports the completion of the job execution to the master 2 (S151). The master 2 reports job completion to the user as necessary. Alternatively, the user may inquire of the scheduler 1 or master 2 about job completion.

The scheduler 1 checks the GPU availability status and job completion status with the master 2 (S152), and acquires this information from the master 2 (S153). The scheduler 1 instructs the cache cluster 7 to delete the cache data of the learning target data (S154). The cache cluster 7 deletes cache data and the like (S155). The cache cluster 7 also deletes the learning results when the learning results are temporarily stored. The cache cluster 7 writes back the data written from the job to the cluster shared storage 4 in accordance with the deletion process. The cache cluster 7 reports completion of deletion to the scheduler 1 (S156).

22A, 22B, and 22C are sequence diagrams showing the operation of the GPU cluster of this embodiment. Here, a sequence will be described in which the cache cluster 7 directly fetches and uses the learning target data on the user storage 6.

The user registers the job in the scheduler 1 (S161). The job registration data includes a job definition, a storage location for learning target data (user storage 6), authentication information such as a user ID, and the like. The job registration data includes closed connection information to the user storage 6 and the like. The closed connection information will be described later. The authentication process of the scheduler 1 will be omitted. Next, “closed connection establishment processing” A, which will be described later, is performed. "Closed connection establishment process" A connects a closed connection or a closed path between the user base and the cache cluster 7 under the control of the scheduler 1. The learning data on the user storage 6 at the user base can be accessed from the cache cluster 7 via the established closed connection or closed path.

The scheduler 1 instructs the cache cluster 7 to prefetch the learning target data (S162). That is, the scheduler 1 instructs the cache cluster 7 to store the learning target data in a predetermined storage location. The cache cluster 7 starts fetching the learning target data on the user storage 6 via a closed connection or a closed path (S163). The processing from S164 to S171 is the same as the processing from S135 to S141 in FIG. 21B, so a description thereof will be omitted here.

Then, the processing from S172 to S181 in FIG. 22B is performed, but since this processing is the same as the processing from S142 to S151 in FIG. 21C, the explanation will be omitted here. Then, in FIG. 22B, the scheduler 1 checks the GPU availability status and job completion status with the master 2 (S182), and acquires this information from the master 2 (S183). Then, "closed connection release processing" C, which will be described later, is performed. The “closed connection release process” releases the closed connection or closed route established between the user base and the cache cluster 7 under the control of the scheduler 1. The scheduler 1 instructs the cache cluster 7 to delete the cache data of the learning target data (S184). The cache cluster 7 deletes cache data and the like (S185). The cache cluster 7 also deletes the learning results when the learning results are temporarily stored. The cache cluster 7 writes back the data written from the job to the cluster shared storage 4 in accordance with the deletion process. The cache cluster 7 reports completion of deletion to the scheduler 1 (S186).

FIG. 23 is a sequence diagram showing the operation of "closed connection establishment process" A. Here, the closed connection establishment process of method 2 shown in FIG. 14 will be explained. A CPE 8 is installed at the user base, and a closed connection is established between the CPE 8 and the cache cluster 7 . Therefore, if the CPE 8 does not publish the API, the user sets the part where the scheduler 1 controls the CPE 8 using the API. The CPE 8 may also replace a vCPE deployed within the carrier network.

As a premise of this process, a job is registered in the scheduler 1. The closed connection information to the user storage 6 included in the job registration data includes "information about the closed connection with the CPE" and "information about the connection to the API of the CPE." However, if the CPE8 does not publish its API and the user configures the CPE8, the closed connection information does not include "CPE connection information to the API." This process will be explained below.

The scheduler 1 instructs the cache cluster 7 to wait for a closed connection (S191). This instruction includes information on closed connection with CPE8. After the closed connection is established, if the cache cluster 7 autonomously controls the acquisition of the learning target data, the "storage location of the learning target data" is also passed in the closed connection standby instruction. The cache cluster 7 performs settings to wait for a closed connection (S192). This establishes a closed connection standby state. The cache cluster 7 reports the completion of the closed connection standby process to the scheduler 1 (S193). Information on the closed connection to the cache cluster 7 is generated in S191. Information on the closed connection to the cache cluster 7 can be obtained from the contract procedure between the user and the operator prior to job registration if the CPE8 does not publish the API and the user configures the CPE8. The decision will be made through negotiation and the user will be notified.

The scheduler 1 instructs the CPE 8 to establish a closed connection (S194). The CPE 8 sets up a closed connection (S195) and starts a closed connection with the cache cluster 7 (S196). If the CPE 8 does not publish the API, job registration from the user and settings for establishing a closed connection to the CPE 8 by the user are performed asynchronously. Therefore, the closed connection start process is repeatedly executed by the CPE 8 until the closed connection is established. The cache cluster 7 accepts the closed connection to the CPE 8 (S197). This establishes a closed connection. The CPE 8 reports completion of the closed connection to the scheduler 1 (S198). Thereafter, the learning target data on the user storage 6 at the user base can be accessed from the cache cluster 7 or the cluster shared storage 4 via the established closed connection.

FIG. 24 is a sequence diagram showing the operation of "closed connection release processing" B. Here, closed connection release processing according to method 2 shown in FIG. 14 will be described. A closed connection has been established between the CPE 8 at the user site and the cache cluster 7, and this is released. Therefore, CPE8 publishes an API for closed connection control. If the API is not made public, the user sets the part where the scheduler 1 controls the CPE 8 using the API. The CPE 8 may also replace a vCPE deployed within the carrier network.

As a premise of this process, a job is registered in the scheduler 1. The closed connection information to the user storage 6 included in the job registration data includes "information about the closed connection with the CPE" and "information about the connection to the API of the CPE." However, if the CPE8 does not publish its API and the user configures the CPE8, the closed connection information does not include "CPE connection information to the API." A closed connection is established between the CPE 8 and the cache cluster 7 under the control of the scheduler 1. Under the control of the scheduler 1, the cache cluster 7 starts fetching the learning target data. The job is deployed under the control of the scheduler 1 and learning begins. There are two deployment timings: after the cache cluster 7 has fetched all the learning target data, and when fetching is continuing. The job is completed, and the scheduler 1 detects the completion of job execution. This process will be explained below.

The information on the closed connection to the cache cluster 7 is as described above in S193 of "closed connection connection processing" in FIG. The scheduler 1 instructs the CPE 8 to release the closed connection (S201). The CPE 8 starts releasing the closed connection to the cache cluster 7 (S202). The cache cluster 7 accepts the cancellation of the closed connection from the CPE 8 (S203). As a result, the closed connection between CPEs 8 and 7 is released. The CPE 8 deletes the closed connection (S204) and reports completion of the closed connection release to the scheduler 1 (S205).

The scheduler 1 instructs the cache cluster 7 to cancel standby for the closed connection (S206). This instruction includes information on closed connection with CPE8. The cache cluster 7 deletes the closed connection standby setting (S207), and reports the cancellation of the closed connection standby to the scheduler 1 (S208).

If the CPE8 does not publish the API and the user configures the CPE8, S206-S208 (cache cluster closed connection canceling process) will be performed before S201-S205 (CPE closed connection canceling process). There is a possibility that it will be executed. In that case, in the "CPE closed area connection release process", since the closed area connection has already been released, S202 (starting the release of the closed area connection) is not executed, but S204 (deleting the closed area connection) is executed. On the other hand, in the "cache cluster closed connection standby cancellation process", in step S207, cancellation of the closed connection is started, and the cancellation of the closed connection is accepted from the CPE 8.

FIG. 25 is a sequence diagram showing the operation of "closed connection establishment process" A. Here, the closed connection establishment process of method 7 shown in FIG. 19 will be described. A CPE 8 is installed at the user base, and a GW 93 of the carrier network holds a connection interface to the CPE 8. A closed connection has already been established between the CPE 8 and the GW 93 using PPPoE, etc., and by establishing a closed connection between the GW 93 and the cache cluster 7, the GW 93 can create a closed connection that relays the two closed connections. Generate a route. By using this closed path, the user storage 6 under the CPE 8 can be accessed from the cache cluster 7 and the cluster shared storage 4. For this control, the GW93 maintains an API for closed connection control.

As a premise of this process, a closed connection has been established between the CPE 8 and the GW 93 by PPPoE or the like, and the CPE 8 can be connected to the Internet via this closed connection. A job is registered in scheduler 1. The closed connection information to the user storage 6 included in the job registration data includes "line identification information" (used to identify the GW 93 when the CPE 8 is connected). This process will be explained below.

The scheduler 1 identifies the GW 93 to which the CPE 8 connects (S211). The scheduler 1 instructs the GW 93 to perform closed connection standby settings and closed connection relay settings (S212). In this relay setting, after establishing a closed connection with the cache cluster 7, the closed connection between the CPE 8 and the GW 93 and the closed connection between the GW 93 and the cache cluster 7 are relayed using routing, switches, etc., and the logical CPE 8 and the cache cluster 7 are connected to each other. This is a setting for generating a closed path between. By using this closed path, the cache cluster 7, the cluster shared storage 4, and the user storage 6 under the CPE 8 can be connected to each other. The GW 93 transfers data destined only for the cache cluster 7 or the cluster shared storage 4 to the closed path regarding traffic from under the CPE 8 . It can be shared with the Internet connection from under CPE8.

The GW 93 performs a standby setting for a closed connection and a relay setting for a closed connection (S213). This establishes a standby state for a closed connection and a standby state for closed connection relay. The GW 93 reports the completion of the closed connection standby setting and the closed connection relay setting to the scheduler 1 (S214). This report includes "information on closed connections to GW". The scheduler 1 instructs the cache cluster 7 to establish a closed connection (S215). This instruction includes "information on closed connection to GW". After the closed connection is established, if the cache cluster 7 autonomously controls the acquisition of the learning target data, the "storage location of the learning target data" is also passed in the closed connection standby instruction. The cache cluster 7 sets up a closed connection (S216) and notifies the GW 93 of the start of the closed connection (S217). The GW 93 accepts the closed connection to the cache cluster 7 (S218). This establishes a closed connection standby state. A closed path between the CPE 8 and the cache cluster 7 is established by relaying the closed connection by the GW 93. The cache cluster 7 reports the completion of establishing the closed connection to the scheduler 1 (S219). From then on, through the established closed path. The learning target data on the user storage 6 at the user base can be accessed from the cache cluster 7 or the cluster shared storage 4.

FIG. 26 is a sequence diagram showing the operation of "closed connection release processing" C. Here, the closed connection release process of method 7 shown in FIG. 19 will be described. A closed connection has been established between the GW 93 and the cache cluster 7, and a closed path has been established between the CPE 8 and the cache cluster 7 by the GW 93. Here, by canceling the closed connection between the GW 93 and the cache cluster 7, the closed path between the CPE 8 and the cache cluster 7 is also canceled. For this control, the GW93 maintains an API for closed connection control.

As a premise of this process, a closed connection has been established between the CPE 8 and the GW 93 by PPPoE or the like, and the CPE 8 can be connected to the Internet via this closed connection. A job is registered in scheduler 1. The closed connection information to the user storage 6 included in the job registration data includes "line identification information." Under the control of the scheduler 1, a closed connection is established between the GW 93 and the cluster shared storage 4. At the same time, a closed path is established between the CPE 8 and the cache cluster 7 by the GW 93. Under the control of the scheduler 1, the cache cluster 7 starts fetching the learning target data. Under the control of the scheduler 1, the job is deployed and learning starts. There are two deployment timings: after the cache cluster 7 has fetched all the learning target data, and when fetching is continuing. This process will be explained below.

The scheduler 1 instructs the CPE 8 to release the closed connection (S231). This instruction includes "information on closed connection to GW93". The cache cluster 7 starts canceling the closed connection with the GW 93 (S232). The GW 93 accepts the cancellation of the closed connection from the cache cluster 7 (S233). As a result, the closed connection is released, and the closed path between the CPE 8 and the cache cluster 7 is released. The cache cluster 7 deletes the closed connection (S234) and reports completion of the closed connection release to the scheduler 1 (S235). The scheduler 1 instructs the GW 93 to delete the closed connection standby setting and the closed connection relay setting (S236). The GW 93 deletes the closed connection standby setting and deletes the closed connection relay setting (S237). The GW 93 reports the completion of deletion in S237 to the scheduler 1 (S238).

FIGS. 27 and 28 are sequence diagrams showing "cluster storage processing of learning target data" B. In this process, the learning target data on the user storage 6 is stored on the cluster shared storage 4.

In FIG. 27, the cache cluster 7 repeatedly reads a block of learning target data from the user storage 6 and writes (duplicates) the block onto the cluster shared storage 4. A block is a part of learning target data, and indicates, for example, a set of one or more files, or a portion of a certain size of one file. A closed connection or a closed path is established between the cache cluster 7 and the CPE 8, and the cache cluster 7 accesses the learning target data in the user storage 6 via either one. Furthermore, upon detecting the establishment of a closed connection or a closed route, the cache cluster 7 autonomously starts storage processing. The CPE 8 may also replace a vCPE located within the carrier network.

As a premise of this process, a job is registered in the scheduler 1. Under the control of the scheduler 1, a closed connection or a closed path is established between the CPE 8 and the cache cluster 7. In order for the cache cluster 7 to autonomously start the storage process, the "storage location of the learning target data" is passed from the scheduler 1 to the cache cluster 7 during the closed connection or closed route establishment process. This process will be explained below.

Upon establishment of the closed connection, the cache cluster 7 reads the learning target data in blocks from the user storage 6 via the closed connection or the closed path (S251), and stores the read learning target data in blocks in the cluster shared storage 4. (S252). The cache cluster 7 repeats S251 and S252 until all learning target data is stored in the cluster shared storage 4. After storing all the learning target data, the cache cluster 7 notifies the scheduler 1 of completion of acquisition of the learning target data (S253). This notification includes the storage location of the learning target data on the cluster shared storage 4.

In FIG. 28, the cache cluster 7 instructs the cluster shared storage 4 to acquire learning target data on the user storage 6. Since the preconditions and the like are the same as those in FIG. 27, their explanation will be omitted here. This process will be explained below.

Taking the establishment of the closed connection as a trigger, the cache cluster 7 instructs the user storage 6 to acquire the learning target data (S271). This instruction includes "the storage location of the learning target data". The cluster shared storage 4 acquires the learning target data from the user storage 6 via a closed connection or a closed path (S272). Thereby, the learning target data in the user storage 6 is stored in the cluster shared storage 4 via a closed connection or a closed path. The cluster shared storage 4 reports completion of acquisition of the learning target data to the scheduler 1 (S273). This report includes the storage location of the learning target data on the cluster shared storage 4.

FIG. 29 is a sequence diagram showing "data access processing to cache clusters in learning processing" D. In this process, a job is speculatively deployed and learning is started before caching (fetching) of learning target data to the cache cluster 7 is completely completed. Learning target data to be cached by the cache cluster 7 is stored in the user storage 6 or the cache cluster 7.

As a premise of this process, a job is registered in the scheduler 1. Under the control of the scheduler 1, the cache cluster 7 starts caching the learning target data. The job is deployed. The learning target data in the cache cluster 7 can be accessed from the job. The job starts learning processing (S291). The job performs learning processing while accessing learning target data. This process will be explained below.

The job requests the cache cluster 7 to read the learning target data in blocks (S292). When a cache miss occurs (S293), the cache cluster 7 performs transparent connection to the learning target data (S294, S295) and prefetching the learning target data (S296) in parallel. A cache miss refers to a state in which a request source (such as a job that uses the cache cluster 7) attempts to read or write uncached data among the data that is cached by the cache cluster 7. Since the data does not exist, it is not possible to immediately respond with data to the requester. The cache cluster 7 performs processing such as requesting data to be cached from the origin while keeping the requester waiting, creating cache data, and then responding to the requester.

In the transparent connection to the learning target data, the cache cluster 7 acquires the learning target data of the block that caused the cache miss from the origin (S294), and returns the acquired learning target data of the block to the job (S295). The cache cluster 7 returns data to the job while accessing the original data of the learning target data when a cache miss occurs. This allows jobs to transparently access the origin of the learning target data while hiding the occurrence of cache misses. Note that in the cache cluster 7, since the blocks of learning target data that are being returned here are unlikely to be used in the future, data input processing may be speeded up by not caching them.

In the prefetching of learning target data, learning target data several blocks ahead is prefetched and cached (S296). After a cache miss occurs, the cache cluster 7 accesses the origin of the learning target data and returns a response to the job, and also starts caching several blocks ahead in parallel for the blocks of learning target data that the job will read in the future. . This makes cache misses temporary, reduces the occurrence of subsequent cache misses, and speeds up data input/output processing.

30A and 30B are sequence diagrams showing "job checkpoint processing". Checkpoint processing is a process that freezes the virtual space and processes included in a running job and saves the state to several files (dump). Job checkpoint processing is achieved using, for example, CRIU (https://www.criu.org/Main_Page, https://github.com/checkpoint-restore/criu).

In this process, a cache miss when a job reads learning target data from the cache cluster 7 is allowed. Specifically, when a cache miss occurs, the occurrence is detected and the job is checkpointed. In this process, a job is speculatively deployed and learning is started before fetching of learning target data to the cache cluster 7 is completely completed. Learning target data to be cached by the cache cluster 7 is stored in the user storage 6 or the cache cluster 7.

As a premise of this process, the node 3 mounts a volume on the cache cluster 7 as a storage location for job dumps. The job dump may be stored on the cluster shared storage 4. A job is registered in scheduler 1. Under the control of the scheduler 1, the cache cluster 7 starts caching the learning target data. The job is deployed. The learning target data in the cache cluster 7 can be accessed from the job. The job starts learning processing (S311). The job performs learning processing while accessing learning target data. This process will be explained below.

Cache misses occur continuously when reading the learning target data. In this case, one of the following three processes is performed.

In "when the cache cluster detects", the cache cluster 7 detects consecutive cache misses equal to or greater than a predetermined threshold (S312), and notifies the scheduler 1 of the occurrence of cache misses (S313). The threshold value is arbitrarily determined by the cluster administrator. An appropriate threshold value can be determined based on the block size, communication speed, etc.

In "When the job detects a cache miss", the job detects a cache miss from a decrease in storage IO bandwidth, etc. (S314), and notifies the scheduler 1 of the occurrence of a cache miss (S315).

In "when the scheduler 1 detects a cache miss", the node 3 reports the job's storage IO bandwidth, GPU usage rate, etc. to the master 2 (S316). Scheduler 1 inquires of master 2 about the job status (S317). Master 2 responds with the job status reported from node 3 (S318). The scheduler 1 detects the occurrence of a cache miss by checking the job status to see if the job's storage IO bandwidth has decreased or if the GPU is almost not being used.

When the scheduler 1 detects the occurrence of a cache miss, it instructs the master 2 to check the job (S319), and the master instructs the node 3 to check the job (S320). Node 3 checkpoints the job (S321). That is, the node 3 stores the dump of the job on the cache cluster 7. Node 3 has mounted cache cluster 7 in advance. Due to the job checkpoint, the job is placed in a suspended state. On the other hand, the cache cluster 7 continues to prefetch the uncached portion of the learning target data.

The node 3 reports the checkpoint completion of the job to the master 2 (S322), and the master 2 reports the completion of the job checkpoint to the scheduler 1 (S323). This report includes the location of the job's dump. Then, "job restoration processing" E, which will be described later, is performed.

FIG. 31 is a sequence diagram showing another "job checkpoint process". This process prevents cache misses when a job reads learning target data from the cache cluster 7. Specifically, the occurrence of a cache miss is detected in advance and the job is checkpointed. Note that the premise of this process is the same as that in FIG. 30A, so the explanation will be omitted here. This process will be explained below.

When the job starts learning (S331), the cache cluster 7 starts monitoring the cache usage status. Occurrence of caching is detected in advance from the change in the amount of cached learning target data and the amount of data read by the job (S332). The cache cluster 7 notifies the scheduler 1 of advance warning of the occurrence of a cache miss (S333). The scheduler 1 instructs the master 2 to check the job (S334), and the master instructs the node 3 to check the job (S335). Node 3 checkpoints the job (S336). That is, the node 3 stores the job dump on the cache cluster 7 mounted in advance. A job checkpoint puts the job in a suspended state. On the other hand, the cache cluster 7 continues to prefetch the uncached portion of the learning target data.

The node 3 reports the completion of the checkpoint of the job to the master 2 (S337), and the master 2 reports the completion of the checkpoint of the job to the scheduler 1 (S338). This report includes the location of the job's dump. Then, "job restoration processing" E, which will be described later, is performed.

32A, 32B, and 32C are sequence diagrams showing "job restoration" E. This process is a process from when the job is checkpointed until the job resumes execution. Job restoration is a process of restoring a job from a checkpointed job dump and restarting the job. Job restoration is achieved using, for example, CRIU (https://www.criu.org/Main_Page, https://github.com/checkpoint-restore/criu).

As a premise of this process, the node 3 mounts a volume on the cache cluster 7 as a storage location for job dumps. The dump may be stored on the cluster shared storage 4. A job is registered in scheduler 1. Under the control of the scheduler 1, the cache cluster 7 starts caching the learning target data. The job is deployed. The learning target data in the cache cluster 7 can be accessed from the job. The job starts learning processing (S351). The job performs learning processing while accessing learning target data. The scheduler 1 temporarily stops execution of the job by checkpointing the job. Even after the job is stopped, prefetching of the uncached portion of the learning target data continues. This process will be explained below.

When job execution is paused due to a checkpoint, one of the following three processes will occur: "Waiting for restoration based on cache cluster polling confirmation," "Waiting for restoration based on time prediction," and "When notified from the cache cluster." processing is performed.

In the "restoration standby based on cache cluster polling confirmation", the scheduler 1 queries the cache cluster 7 about the amount of cache data at the checkpoint of the job and the amount of learning target data (S352), and the cache cluster 7 asks these about the amount of cache data at the job checkpoint and the amount of learning target data. Information is acquired (S353). The amount of learning target data may be obtained from the user at the time of job registration. The scheduler 1 then inquires of the cache cluster 7 about the amount of cache data and obtains it (S354, S355). The scheduler 1 repeats the processes of S354 and S355 until “cache data amount”−“cache data amount at checkpoint”>=“data amount threshold”.

In the "restoration standby based on time prediction", the scheduler 1 inquires of the cache cluster 7 about the amount of cache data at the time of the checkpoint, the cache speed of the cache cluster 7, and the amount of learning target data (S356). information is acquired (S357). The amount of learning target data may be obtained from the user at the time of job registration. The cache speed of the cache cluster 7 indicates the data input throughput when the cache cluster 7 caches learning target data.

Scheduler 1 calculates waiting time candidate 1 (S358). Specifically, the scheduler 1 calculates, as a time candidate 1, the time until the cache data amount exceeds a threshold in the future from the cache data amount and cache speed at the time of the checkpoint. Scheduler 1 calculates waiting time candidate 2 (S359). Specifically, the scheduler 1 calculates the time until all of the learning target data is cached from now on as the time candidate 2 from the cache data amount and cache speed at the time of the checkpoint. Scheduler 1 compares candidate wait time 1 and candidate wait time 2, and waits for the shorter time (S360).

In "the case of notification from the cache cluster", the scheduler 1 instructs the cache cluster 7 about the required amount of cache data (S361). The cache cluster 7 caches the uncached portion of the learning target data (S362), and notifies the scheduler 1 when the instructed amount of data has been cached (S363).

Scheduler 1 registers the checkpointed suspended job in RQ23 (S364). The scheduler 1 inquires of the master 2 about the availability status of the GPU (S365) and obtains the information (S366). If there is free space on the GPU, the scheduler 1 schedules the job (S367). Specifically, the scheduler 1 schedules the RQ23 job with priority over the DQ24 normal job. Scheduler 1 instructs master 2 to restore the job (S368), and master 2 instructs node 3 to restore the job (S369). This instruction includes the location of the dump. Node 3 executes job restoration (S370) and resumes job execution (S371). For example, a virtual environment such as a network namespace is restored, and the learning process is restored from the state at the checkpoint to a state where it can be restarted. Node 3 restarts the learning process (S372).

(Effects of this embodiment)
The scheduler 1 in the GPU cluster system of the present embodiment described above includes a first queue selector 11 that stores submitted jobs in a first stage queue 13-15 that stores jobs waiting to start fetching, and a first stage queue A first job selector 12 extracts the job No. 13-15 and registers it in the fetching job list 30, and causes the cache cluster 7 to start fetching the data of the job stored in the storage 4, and the amount of fetched data is determined by a predetermined amount. A second queue selector 21 extracts jobs that exceed the threshold from the fetching job list 30 and stores them in a second stage queue 23-25 that stores jobs waiting to be deployed. and a second job selector 22 that instructs to extract and deploy the job, and in the job deployment instruction, the cache cluster 7 is specified as the storage location of the data of the job, and the GPU cluster is assigned to the cache cluster 7. access and execute the job.

As a result, in this embodiment, idle time of the GPU caused by insufficient storage speed can be reduced, and the utilization rate of the GPU can be improved. In other words, it is possible to speed up the reading of data such as learning target data, and it is possible to increase the utilization rate of GPUs by GPU cluster system providers.

Further, in this embodiment, a job to be executed is registered in the fetching job list 30, and the cache cluster 7 is caused to start prefetching data. In this way, by prefetching data in parallel with job execution by the GPU, the GPU can be used efficiently.

Furthermore, in this embodiment, the GPU utilization rate can be improved by temporarily stopping a job when the GPU is idle due to waiting for data and giving the GPU to another job.

(Hardware configuration)
For the scheduler 1 described above, for example, a general-purpose computer system as shown in FIG. 33 can be used. The illustrated computer system includes a CPU (Central Processing Unit) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device. 906. Memory 902 and storage 903 are storage devices. In this computer system, each function of the scheduler 1 is realized by the CPU 901 executing a predetermined program loaded onto the memory 902.

Furthermore, the scheduler 1 may be implemented on one computer or on multiple computers. Further, the scheduler 1 may be a virtual machine implemented in a computer.

The program for Scheduler 1 can be stored on a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or distributed via a network. You can also.

Note that the present invention is not limited to the above-described embodiments and modifications, and can be modified in many ways within the scope of the gist thereof.

1: Scheduler 11: First queue selector 12: First job selector 13: Job queue (JQ)
14: Over-Fairness Job Queue (OFJQ)
15: User excess job queue (OUJQ)
21: Second queue selector 22: Second job selector 23: Restore queue (RQ)
24: Deployment queue (DQ)
25: Over-Fairness Queue (OFDQ)
30: Fetching job list (FJL)
31: Account DB
32: GPU usage monitoring unit 2: Master 3: Node 4: Cluster shared storage 5: User terminal 6: User storage 7: Cache cluster

Claims (7)

A scheduling method performed by a GPU cluster system,
The scheduler is
storing the submitted job in a first stage queue that stores jobs waiting to start fetching;
retrieving a job from a first stage queue, registering it in a fetching job list, and causing a cache cluster to start fetching data for the job;
extracting a job whose fetched data amount exceeds a predetermined threshold from the fetching job list and storing it in a second stage queue that stores jobs waiting to be deployed;
retrieving the job from the second stage queue and instructing the deployment of the job;
The cache cluster is
fetching job data registered in the fetching job list from the storage where the data is stored and storing it in the cache cluster;
GPU cluster is
A scheduling method that performs a step of accessing data in the cache cluster and executing a job. A scheduler for a GPU cluster system,
a first queue selector that stores the submitted job in a first stage queue that stores jobs waiting to start fetching;
a first job selector that retrieves a job from a first stage queue, registers it in a fetching job list, and causes a cache cluster to start fetching data of the job stored in storage;
a second queue selector that extracts jobs whose fetched data amount exceeds a predetermined threshold from the fetching job list and stores them in a second stage queue that stores jobs waiting to be deployed;
a second job selector that retrieves a job from the second stage queue and instructs to deploy the job;
The cache cluster is specified as a storage location for data of the job in the job deployment instruction, and the GPU cluster accesses the cache cluster and executes the job. 3. The scheduler according to claim 2,
From the viewpoint of fairness, the first stage queue is divided into a job queue in which jobs that do not exceed the GPU quota assigned to each user are stored, and an excess job queue in which jobs that exceed the quota are stored. Prepare,
A first job selector is a scheduler that registers jobs in the job queue in a fetching job list with priority over jobs in the excess job queue. The scheduler according to claim 2 or 3,
The second stage queue consists of a restore queue where jobs waiting to be restored are stored, a deploy queue where jobs waiting to be deployed are stored, and jobs that exceed the GPU quota allocated to each user from a fairness perspective. Equipped with a stored excess queue and
A second job selector is a scheduler that instructs to deploy jobs in the restore queue with priority over jobs in the deploy queue, and instructs to deploy jobs in the deploy queue with priority over jobs in the excess queue. 5. The scheduler according to claim 4,
If the restore queue, the deploy queue, and the excess queue are all empty, the second job selector activates the second job selector and selects the job or fetch job that has the largest amount of fetched data in the fetch job list. a scheduler that stores a first job in a processing job list in one of the restore queue, the deployment queue, and the excess queue; A GPU cluster system comprising the scheduler according to any one of claims 2 to 5, a cache cluster, and a GPU cluster,
The cache cluster fetches the data of the job registered in the fetching job list from the storage where the data is stored and stores it in the cache cluster,
The GPU cluster accesses data in the cache cluster to execute the job.
GPU cluster system. A program that causes a computer to function as the scheduler according to claim 2.

Images