US20140208072A1

US20140208072A1 - User-level manager to handle multi-processing on many-core coprocessor-based systems

Info

Publication number: US20140208072A1
Application number: US13/858,034
Authority: US
Inventors: Srihari Cadambi; Kunal Rao; Srimat T. Chakradhar; Rajat Phull; Giuseppe Coviello; Murugan Sankaradass; Cheng-Hong Li
Original assignee: NEC Laboratories America Inc
Current assignee: NEC Laboratories America Inc
Priority date: 2013-01-18
Filing date: 2013-04-06
Publication date: 2014-07-24
Also published as: US20140208327A1; US20140208331A1; US9086925B2; US9152467B2

Abstract

A method is disclosed to manage a multi-processor system with one or more multiple-core coprocessors by intercepting coprocessor offload infrastructure application program interface (API) calls; scheduling user processes to run on one of the coprocessors; scheduling offloads within user processes to run on one of the coprocessors; and affinitizing offloads to predetermined cores within one of the coprocessors by selecting and allocating cores to an offload, and obtaining a thread-to-core mapping from a user.

Description

This application is a non-provisional of and claims priority to provisional applications with Ser. No. 61/754,371 filed on Jan. 18, 2013 and Ser. Nos. 61/761,969 and 61/761,985 both filed on Feb. 7, 2013, the contents of which are incorporated by reference.

BACKGROUND

The present application relates to multi-core processing.
The Intel Xeon Phi is a recently introduced x86-based 60-core, 240-thread coprocessor that is increasingly being deployed in servers and clusters. It is easier to program than other manycore processors, and runs the Linux operating system. The operating system provides services such as virtual memory and context switching, and enables multiple processes to run concurrently and share the coprocessor. Multi-processing on the manycore is also necessary to fully utilize the hardware resources of the Xeon Phi.
It is remarkably easy to offload processing the Xeon-Phi: it supports a popular ISA (x86), a popular OS (Linux), and a popular programming model (OpenMP). Unfortunately, quick and easy portability rarely results in an implementation that executes faster on the Xeon-Phi. In order to use the Xeon Phi, a programmer employs pragmas to identify code regions to be “offloaded” to the coprocessor, for which the compiler automatically generates coprocessor instructions along with glue code to transfer data. This is referred to as the “offload programming model” where the main trunk of the code runs on the host processor while regions identified by the pragmas are offloaded to run on the Xeon Phi coprocessor. Thus, if portions of a program are already parallelized using OpenMP, porting them to the Xeon Phi is easy. Unfortunately, quick and easy portability rarely results in an implementation that executes faster on the Xeon Phi. Rather, additional programmer effort such as carefully selecting the number of threads, mapping them to cores, ensuring no thread or memory oversubscription, and load balancing the application across multiple coprocessors is necessary.
Such application tuning can work well for individual applications when they “own” the coprocessor, but not in a multi-processing environment. The Xeon Phi coprocessor runs Linux, which makes it easy for multiple processes to share the coprocessor. Such a use case of a manycore like the Xeon Phi where multiple processes compete for coprocessor resources is likely not only in cluster and cloud deployments, but also in servers since good hardware utilization is essential.
In a multi-processing environment, multiple application processes compete for coprocessor resources, and one programmer is unaware of another programmer's intentions. Thus any programmer-driven steps taken to improve performance can in fact degrade it. In such an environment, processes must adhere to the following guidelines in order to avoid performance degradation and benefit from the manycore.
Additional programmer effort such as carefully selecting the number of threads, mapping them to cores, ensuring memory is not over-subscribed, and using multiple coprocessors well are necessary to tune application performance But programmer directives alone are insufficient: multi-processing on the manycore is required to improve hardware utilization. Linux makes it easy for processes to share the Xeon Phi, but in an environment where applications compete for manycore resources, any programmer effort intended to boost individual application performance can in fact end up doing the opposite.
In a multi-processing environment, directives introduced by the programmer specifically to enhance manycore coprocessor performance can be counter-productive. For instance, programmers must select the number of threads, map them to cores, ensure memory is not over-subscribed, and manage their workload across multiple coprocessors. But these can degrade performance in a multi-processing environment when one programmer is unaware of other programmers' intentions.
Various solutions are available to help programmers take advantage of the co-processor. Programming model such as SWARM provides an API to represent workloads as tasks, compiles them to specific processors including the Xeon Phi and uses a runtime to manage the tasks on distributed heterogeneous nodes. Compilers such as CAPS can produce host and Xeon Phi code starting from OpenACC directives. Libraries with parallelized high performance numerical code for the Xeon Phi have been done. Cluster management middleware can schedule workloads on the Xeon Phi. Virtualization approaches such as ScaleMP provide a hypervisor that can “virtualize” the Xeon Phi and host into a single entity visible to programmers. An operating system runtime (MPSS) is available on top of the Xeon Phi micro kernel (OS) to perform primitive scheduling of offloads.

SUMMARY

In one aspect, a method is disclosed to manage a multi-processor system with one or more multiple-core coprocessors by intercepting coprocessor offload infrastructure application program interface (API) calls; scheduling user processes to run on one of the coprocessors; scheduling offloads within user processes to run on one of the coprocessors; and affinitizing offloads to predetermined cores within one of the coprocessors by selecting and allocating cores to an offload, and obtaining a thread-to-core mapping from a user.
In another aspect, a server populated with one or more multiple core Xeon Phi coprocessors includes a manager to control user processes containing offload blocks by intercepting COI API calls and schedules user processes to run on one of the Xeon Phi coprocessors; schedules offloads within user processes to run on one of the Xeon Phi coprocessors; and affinitizes offloads to specific cores within one of the Xeon Phi coprocessors by selecting and allocating cores to an offload, and obtaining the thread-to-core mapping from the user.
Implementations of the above system can

- Employ an aging-based first-fit algorithm for process and offload scheduling
- Use thread and memory over-scheduling factors to enhance performance
- Employ a greedy algorithm for core selection such that offloads from the same process get preference to use the same cores
- Use bitmaps to enhance the performance of core selection algorithm.

Advantages of the above system may include one or more of the following. The system provides a middleware on top of the Xeon Phi micro kernel and the Intel runtime. The middleware handles multi-processing on Xeon Phi coprocessor-based servers by automatically avoiding thread and memory oversubscription and load balancing processes across the cores of the Xeon Phi and across several Xeon Phi coprocessors. The system is completely transparent to the users and requires no changes to the underlying software such as the MPSS and the Linux kernel running on the coprocessor. It uses a scheduling technique to schedule processes and Xeon Phi offload regions within processes simultaneously. It also uses algorithms to set thread affinity and load balance processes across coprocessors.
The system achieves faster operation when multiple processes share a many integrated core coprocessor system. Faster operation includes end-to-end turn-around-time per process (latency), as well as the number of processes completed per unit time (throughput).
The system protects against thread and memory over-subscription resulting in severe performance loss and crashes. Within a coprocessor, it manages cores such that offloads of different processes run on separate sets of cores, and offloads in the same process use the same cores (thus respecting data affinity). The system balances the load of multiple processes across multiple Xeon Phi coprocessors. The manager provides a transparent user-level middleware that includes a suite of run-time techniques explicitly designed to enhance performance portability in the presence of multi-processing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary process manager for a multiprocessor system.

FIG. 2 shows an exemplary software stack with host and co-processor components of the multi-processor software stack.

FIG. 3 shows an exemplary flow of the process manager of FIG. 1.

FIG. 4 shows an exemplary architecture of the process manager of FIG. 1.

FIG. 5 shows an exemplary scheduling procedure for the system of FIG. 1.

FIG. 6 shows an exemplary method for aging-based first-fit procedure for process selection.

DESCRIPTION

FIG. 1 shows a high-level view of a process manager 20 called the COSMIC system. In a server with multiple Xeon Phi 30, 32 and 34, each with different amounts of memory and cores, COSMIC manages offloads from several user processes 10, 12 and 14. Each user process contains several offload blocks that are executed sequentially. The process has a single memory requirement for all its offloads, while the offloads themselves have their own thread requirements. Thus, before execution, every process requests the process manager 20 for memory, and every offload requests the process manager 20 for threads. COSMIC arbitrates the requests by taking into consideration the different available coprocessors, the available cores within each device and the available memory. It then schedules and allocates resources for the offloads in such a way that thread and memory oversubscription are avoided, and the devices as well as the cores within them are load balanced. COSMIC manages processes and coprocessor resources in order to:

- Avoid over-subscribing coprocessor hardware threads.
- Avoid over-subscribing and carefully manage the limited coprocessor main memory.
- Map threads to cores, ensuring minimal thread migration, while respecting data affinity and persistence across offload regions.
- Load balance applications across multiple coprocessors while ensuring locality of data.
  Given a server populated with multiple integrated core coprocessors such as the Xeon Phi coprocessors, with several users and processes competing for coprocessor resources, the goals of the process manager 20 are to manage processes and coprocessor resources and:
- Avoid over-subscribing coprocessor hardware threads.
- Avoid over-subscribing and carefully manage the limited coprocessor main memory.
- Map threads to cores, ensuring minimal thread migration, while respecting data affinity and persistence across offload regions.
- Load balance applications across multiple coprocessors while ensuring locality of data.

To simplify memory management, one implementation requests that the programmer specify the maximum memory required on the Xeon Phi for each process. This is similar to job submission requirements in cluster schedulers. In typical cases, different offloads of the same process often share data in order to reduce data movement between the host and Xeon Phi. Thus, as long as the process exists, it will use memory on the card. However, unlike cluster schedulers, this embodiment does not require the process specify core, devices or other resources, but infers it automatically from the number of threads requested by the offload. Unlike memory that is reserved for the life of a process, threads (and cores) are given to an offload when it starts executing and released when the offload completes for use by other offloads.
Before execution, every process requests COSMIC for memory, and every offload requests COSMIC for threads. COSMIC arbitrates the requests by taking into consideration the different available coprocessors, the available cores within each device and the available memory. It then schedules and allocates resources for the offloads in such a way that thread and memory oversubscription are avoided, and the devices as well as the cores within them are load balanced.
COSMIC has several parameters that may be set by the server administrator or user that can affect its policies and behavior. An administrator can configure the following parameters of COSMIC to affect its scheduling decisions:

- Aging threshold: how many times the scheduler attempts a process or offload before progress is blocked.
- Thread factor T_o(1 or larger): the offload “fit” function considers T_o*hardware threads as the total number of threads in a coprocessor. If T_ois greater than 1, the number of threads is oversubscribed in a measured way to leverage the fact that slight oversubscriptions may actually be beneficial since otherwise offloads may have to wait longer.
- Memory factor M_o(1 or larger): the process “fit” function considers M_o*physical memory as the total physical memory of the coprocessor. If M_ois greater than 1, the memory is oversubscribed in a measured way to leverage the fact that not all processes will require their maximum requested memory at the same time.
  COSMIC also expects from the owner of each process the following directives:
- Memory limit: the peak memory the process will use over its lifetime. COSMIC kills any process that exceeds its memory usage as described later in this section.
- Preferred thread affinity: In order to allocate Xeon Phi cores for an offload, COSMIC needs to know how user threads must be mapped to cores. A SCATTER mapping indicates 1 thread per core, a COMPACT mapping 4 threads per core and BALANCED 3 threads per core.

FIG. 2 shows a block diagram of the Xeon Phi software stack, and where COSMIC fits in. The left half of the figure shows the stack on the host processor, while the right half shows the stack running on the coprocessor. The top half represents user space, and the bottom half represents kernel space. The host processor runs a Linux kernel 122 with a PCI and card driver 124-126 to communicate with the card. Along with the operating system, a Symmetric Communication Interface (SCIF) driver 120 is provided for inter-node communications. A node can be a Xeon Phi device or the host processor. SCIF 120 provides a set of APIs for communication and abstracts the details of communicating over the PCIe bus. On top of SCIF 120, the Coprocessor Offload Infrastructure (COI) 112 is a higher-level framework providing a set of APIs to simplify development of applications using the offload model. COI provides APIs for loading and launching device code, asynchronous execution and data transfer between the host and Xeon Phi. The coprocessor portion of the software stack consists of a modified Linux kernel 156, the PCI driver 154 and the standard Linux proc file system 152 that can be used to query device state (for example, the load average). The coprocessor portion also has a SCIF driver 158 for communicating over the PCI bus with the host and other nodes. The COI 112 communicates with a COSMIC host component 110 that communicates with user processes 100-104. The host component 110 interacts with a COSMIC coprocessor component 160 that handles offloaded portions of user processes 162.
The COSMIC host middleware component has a global view of all processes and offloads emanating from the host, and knowledge of the states of all coprocessor devices. COSMIC is architected to be lightweight and completely transparent to users of the Xeon Phi system. As shown in FIG. 2, COSMIC exists in the user space, but interacts with both user processes and other kernel-level components. It controls offload scheduling and dispatch by intercepting COI API calls that are used to communicate with the device. This is a key mechanism of COSMIC that enables it to transparently gain control of how offloads are managed. We first briefly describe by way of example how offload blocks are expressed using the COI API.
The Xeon Phi compiler converts all offload blocks that are marked by pragmas into COI calls. The user's program with offload pragmas is compiled using Intel's icc or a gcc cross-compiler for the Xeon Phi. The compiler produces a host binary, and Xeon Phi binaries for all the offload portions. The offload portions are first translated into a series of COI API calls. The figure shows the important calls for a simple example: first COIEngineGetCount and COIEngineGetHandle get a handle to the coprocessor specified in the pragma. Then COIProcessCreateFromFile creates a process from the binary corresponding to the offload portions. Each offload block is represented as a function, and COIProcessGetFunctionHandles acquires the handles to these functions. COIPipelineCreate creates a “COI pipeline” which consists of 3 stages: one to send data to the coprocessor, one to perform the computation and one to get data back from the coprocessor. Then COIBufferCreate creates buffers necessary for inputs and outputs to the offload. In this example, three COI buffers corresponding to the arrays a, b and c are created. COIBufferCopy transfers data to the coprocessor, and COIPipelineRunFunction executes the function corresponding to the offload block. Finally, another COIBufferCopy gets results (i.e., array c) back from the Xeon Phi.
Since every offload is converted into a series of COI calls (which has a standard API), COSMIC can transparently control offload scheduling and dispatch.
COSMIC is architected as two components implemented as separate processes: the COSMIC client and the COSMIC server. The COSMIC client is the front-end, while the COSMIC server is the back-end consisting of the scheduler and the monitor. The monitor comprises a host portion and a card-side portion, as depicted in FIG. 2. Inter-process interfaces are clearly defined: each process communicates with the other two using explicit messages.
The COSMIC client is responsible for intercepting COI calls and communicating with the scheduler in the COSMIC server to request access to a coprocessor. It accomplishes this using library interposition. Every user process links with the Intel COI shared library that contains definitions for all API function modules. COSMIC intercepts and redefines every COI API function: the redefined COI functions perform COSMIC-specific tasks such as communicating with the COSMIC scheduler, and then finally calls the actual COI function. With the redefined functions, COSMIC creates its own shared library that is pre-loaded to the application (using either LD_PRELOAD or redefining LD_LIBRARY_PATH). The pre-loading ensures that COSMIC's library is first used to resolve any COI API function.
Based on the type of COI API intercepted, the client sends the following different messages to the scheduler in the COSMIC server:
NewProcess: When an offload is first encountered for a a process, the client sends a NewProcess message to the scheduler indicating that the scheduler should account for a new process in its book-keeping. Every new process is annotated with its memory requirement provided by the user.
NewOffload: For every offload, the client sends a NewOffload message to the scheduler indicating the process to which the offload belongs and the number of threads it is requesting. It also indicates the size of the buffers that need to be transferred to the coprocessor for this offload.
OffloadComplete: When an offload completes, the client sends an OffloadComplete message to the scheduler so that it can account for the newly freed resources such as coprocessor cores and threads.
ProcessComplete: When a process completes, the client sends a ProcessComplete message to the scheduler to account for the freed memory used by the process.
The COSMIC scheduler is the key actor in the COSMIC system and manages multiple user processes with offloads and several coprocessor devices by arbitrating access to coprocessor resources. It runs completely on the host and has global visibility into every coprocessor in the system. In scheduling offloads and allocating resources, it ensures no thread and memory oversubscription and load balances coprocessor cores and devices to most efficiently use them.
A key distinction between the COSMIC scheduler and traditional operating system schedulers is that COSMIC concurrently schedules processes and offloads within the processes. Each process has a memory requirement, while each offload has a thread requirement. Various coprocessors in the system may have different memory and thread availabilities.
Under these constraints, the goal of the scheduler is to schedule processes and offloads by mapping processes to Xeon Phi coprocessors and offloads to specific cores on the coprocessors. The scheduler also ensures fairness, i.e., makes sure all processes and offloads eventually get access to coprocessor resources.
The scheduler is event-based, i.e., a scheduling cycle is triggered by a new event. A new event can be the arrival of a new process, the arrival of a new offload in an existing process, the dispatching of an offload to a Xeon Phi device, the completion of an offload or the completion of a process. A queue of pending processes is maintained: each arriving new process is added to the tail of the pending process queue. A process is eventually scheduled to one Xeon Phi coprocessor. The scheduler also maintains a queue of pending offloads for each Xeon Phi coprocessor in the system. Each new offload is added to the tail of the offload queue belonging to the Xeon Phi coprocessor on which its process has been scheduled.
FIG. 3 shows the orkflow through the Xeon Phi software stack when multiple processes are issued. They are all fed to the Xeon Phi MPSS runtime, which often serializes them in order to avoid crashing the coprocessor. The manager 20 avoids this by intercepting COI calls at 202, and the manager 20 takes control of the processes and offloads. Specifically, in 210 the manager 20 performs process scheduling, offloads scheduling and affinitizes offloads to specific cores on the co-processor. Once this is done, it issues the processes and offloads to the MPSS at 204 and continues with the Linux operating system 206.
FIG. 4 shows an exemplary architecture of the process manager of FIG. 1. COSMIC is architected as two components implemented as separate processes: clients 310-312 that communicate with a library 316, scheduler 320 and monitor 326, the latter comprising a host portion and a card-side portion. Inter-process interfaces are clearly defined: each process communicates with the other two using explicit messages.
FIG. 5 shows an exemplary scheduling procedure. When a new event occurs, a pending process is selected and scheduled to a coprocessor that has enough free memory. Then offload queues corresponding to each Xeon Phi are examined, and the scheduler dispatches an offload to each coprocessor if it has enough free threads. Both processes and offloads are selected based on an aging-based first-fit heuristic.
When a new event occurs, a pending process is selected and scheduled to a coprocessor that has enough free memory. Then offload queues corresponding to each Xeon Phi are examined, and the scheduler dispatches an offload to each coprocessor if it has enough free threads. Both processes and offloads are selected based on an aging-based first-fit heuristic.
At the start of a scheduling cycle, let P be the process at the head of the pending process queue (402). The scheduler maintains a circular list of the Xeon Phi coprocessors in the system. Let D be the next coprocessor in the list (404). The scheduler checks to see if the memory required by P fits in the available memory of D (406). If it does, P is removed from the queue and dispatched to D (408). If not, the next coprocessor in the circular list is examined (410). If P does not fit in any coprocessor, its age is incremented, and the next pending process is examined (412). When a process' age reaches a threshold, all scheduling is blocked until that process is scheduled (414). This ensures fairness since all processes will get a chance at being scheduled.
Scheduling an offload is similar to scheduling a process, with one difference. Instead of memory, an offload has a thread requirement; COSMIC checks if the threads requested by an offload are available on the coprocessor on which the offload's owner process has been scheduled. If so, the offload is dispatched. If not, it increments the offload's age, and examines the next offload in the queue.
An administrator can specify the following parameters to tailor the scheduler's behavior: (i) aging threshold, (ii) thread over-scheduling factor and (iii) memory over-scheduling factor. The latter two indicate to what extent threads and memory may be oversubscribed.
The COSMIC monitor collects data pertaining to the state of the coprocessors. It has a host-side component, and a component that is resident on each of the coprocessors. The host-side component is primarily responsible for communicating with the scheduler and all the coprocessor-side components. The coprocessor-side components monitor the load on the device, the number of threads requested by each offload and the health of each offload process. If a process dies for any reason, it catches it and reports the reason to the COSMIC scheduler.
Next, the affinity setting is discussed. COSMIC selects the cores that are used by an offload, and affinitizes threads to these cores using programmer directives. The core selection procedure for an offload is discussed next. COSMIC's core selection algorithm scans one or more lists of free physical cores to select cores until it finds enough cores for a given offload region. The number of cores assigned to an offload region is the number of threads used by the offload region divided by the thread-to-core ratio N. The order of the core lists from which COSMIC selects cores reflects the preference of COSMIC's core selection strategy. The first list of physical cores for a new offload region consists of cores that are both free and only used by the earlier offloads coming from the same offload process. If more physical cores are needed, COSMIC picks from a second list of physical cores, which are both free and not currently assigned to other offload processes. If still more cores are needed, COSMIC forms a third list of physical cores, which are the remaining free cores not yet selected.
To ensure efficient execution of offloads COSMIC adopts several policies to make sure the hardware resources of a Xeon Phi processor are well utilized but not oversubscribed. These policies and their implementations are discussed below.
COSMIC limits the total number of actively running software threads on a Xeon Phi device to ensure that the device's physical cores are not oversubscribed. When an offload region is running, all of the threads spawned by the offload process are considered active. Otherwise the threads are considered dormant. COSMIC keeps track of the number of active and inactive software threads spawned by offload processes on a Xeon Phi device. It maintains the ratio between the total number of active software threads spawned by all offload processes and the physical cores to be no more than an integer N, which is configured as 4 in our current implementation of COSMIC. Therefore COSMIC only schedules an offload region to run if the thread-to-core ratio will not exceed N after the offload region starts.
COSMIC uses several mechanisms to detect the number of software threads created by an offload process on a Xeon Phi device. First COSMIC inspects the environment variable MIC_OMP_NUM_THREADS of a submitted job. COSMIC also intercepts omp_set_num_threads_target( ) function calls on the host. Finally on a Xeon Phi device an offload process's call to omp_set_num_threads( ) is also intercepted and the number of threads is reported back the host COSMIC process.
COSMIC relies on the OMP library to get the number of physical cores on a Xeon Phi device. During COSMIC's initialization, the monitor (running on each Xeon Phi device) queries the maximum number of threads by calling omp_get_max_threads( ) and communicates the returned value to COMSIC's host process. COSMIC then divides this number by 4 to obtain the number of the processing cores on a Xeon Phi device. Notice that the number of physical cores derived in this approach is generally one less than the real number of the physical cores. We believe this is because one core is reserved for the OS, and thus we do not adjust the derived number.
To further improve the efficiency of offload executions, COSMIC creates physical-core containers and sets thread affinity automatically. The goal is to avoid threads migrating from one core to another so data in the registers and the private L1 cache can be reused, and to have multiple concurrent offload processes so the overall system utilization remains high.
COSMIC assigns different, non-overlapping sets of physical cores to execute concurrent offload regions. For an offload region the selected cores constitute a physical-core container that the offload can use exclusively. The physical-core containers minimize the interference between multiple concurrent offloads on the same Xeon Phi device due to resource contention. The physical-core container expires after the execution of its offload region completes, and the assigned cores are released for use by other offloads. Notice that a physical core on a Xeon Phi device can be assigned to multiple offload processes but only used by at most one active offload region at any given point of time. A free core is a core not currently used by any offload region (but it may be assigned to one or more offload processes.)
COSMIC's core selection algorithm scans one or more lists of free physical cores to select cores until it finds enough cores for a given offload region. The number of cores assigned to an offload region is the number of threads used by the offload region divided by the thread-to-core ratio N. The order of the core lists from which COSMIC selects cores reflects the preference of COSMIC's core selection strategy. The first list of physical cores for a new offload region consists of cores that are both free and only used by the earlier offloads coming from the same offload process. If more physical cores are needed, COSMIC picks from a second list of physical cores, which are both free and not currently assigned to other offload processes. If still more cores are needed, COSMIC forms a third list of physical cores, which are the remaining free cores not yet selected.
The core selection algorithm uses simple linear arrays to track of the status of the physical cores so that it can efficiently construct various lists of physical cores to choose from. For each Xeon Phi device the algorithm maintains two arrays, called F and G. Entry i of an array stores the status of physical core i. Array F indicates which cores are free, and array G records the number of the offload processes that a core is assigned to. Further the algorithm also maintains for each active offload process an array, called P, listing the cores assigned to the process's latest offload region (P[i] is 1 iff core i is assigned.)
From the status arrays the algorithm constructs the various lists of cores to select for an offload region. For an offload region, the first list of cores to be considered (the most preferred) consists of core i such that F[i]=1, G[i]=1, and the offload process's P[i]=1. If not enough cores can be found from the first list, the algorithm creates a second list of cores, where a core i on the list satisfies F[i]=1, G[i]=0 (and P[i]=0, which is implied by G[i]=0). Finally the third list of cores (the least preferred) consist of any core i such that F[i]=1 (and G[i]>1, that is, core i is assigned to more than one offload processes).
These status arrays are updated after offload starts and completes or when an offload process finishes. The initial values of the entries in F and G are 1 and 0, respectively. The update rules are as follows:

- 1. If core i is selected for an offload region, F[i] is set to 0 after the offload region starts, and to 1 after the offload region ends.
- 2. After an offload region starts, G[i] is incremented by 1 if core i is selected and P[i] is 0, or decremented by 1 if core i is not selected and P[i] is 1. In any other case G[i] is unchanged. Following the adjustment of values in G, the values in P are adjusted: P[i]=1 if and only if core i is selected.
- 3. When an offload process finishes, G[i] is decremented by 1 if P[i] is 1 for each core i, otherwise G[i] is unchanged.

COSMIC uses the thread affinity API of the OpenMP library to keep the software threads of an offload region running on the selected cores. This thread-to-core binding procedure is conducted before or at the beginning of every offload region after the number of software threads is detected and the cores are selected. Since the thread binding must be conducted in the offload process at which the binding is targeted, COSMIC preloads every offload process a special function to perform the binding and calls the function when the binding is required. The binding is similar to the “compact” option provided by OpenMP: each core is assigned N=4 threads.
Oversubscribing Xeon Phi memory can lead to two undesirable results: application crashes or excessive performance degradation due to memory swapping. Therefore COSMIC ensures that the overall memory requirement of the offload processes running on a single Xeon Phi device does not exceed the amount of the device's physical memory.
To avoid memory oversubscription COSMIC keeps track of the amount of available physical memory for each Xeon Phi device. When COSMIC starts running, it queries each MIC device the amount of free physical memory using a COI function (COIEngineGetInfo). When a user submits a job to COSMIC, the user needs to inform COSMIC the total amount of memory the process needs on a Xeon Phi device through an environment variable COSMIC_MEMORY. COSMIC only launches a submitted job if there is one Xeon Phi device with enough free memory to meet the memory requirement of the job.
COSMIC can be optionally configured to terminate any running process that uses more Xeon Phi memory than the amount specified by the user. COSMIC relies on Linux's memory resource controller to set up a memory container for each offload process on a Xeon Phi device. Each container limits the real committed memory usage of the offload process to the user-specified maximum value. If a process's memory footprint goes over the limit, the memory resource controller invokes Linux's out-of-memory killer (oom-killer) to terminate the offending process.
Enforcing this maximum memory usage rule requires an extra installation procedure and incurs minor runtime performance overhead. The memory resource controller is not enabled in the default Xeon Phi OS kernel. To install a new kernel with the memory resource controller requires adding one line to the kernel configuration file, recompiling the kernel, and rebooting Xeon Phi cards with the new kernel image. The runtime performance overhead due to using the Linux memory controller ranges from negligible to about 5% in real applications.
A many integrated cores (MIC) co-processor can have the cores, PCIe Interface logic, and GDDR5 memory controllers are connected via an Interprocessor Network (IPN) ring, which can be thought of as independent bidirectional ring. The L2 caches are shown as slices per core, but can also be thought of as a fully coherent cache, with a total size equal to the sum of the slices. Information can be copied to each core that uses it to provide the fastest possible local access, or a single copy can be present for all cores to provide maximum cache capacity. In one embodiment, the co-processor is the Intel® Xeon Phi™ coprocessor that can support up to 61 cores (making a 31 MB L2) cache) and 8 memory controllers with 2 GDDR5 channels each. Communication around the ring follows a Shortest Distance Algorithm (SDA). Co-resident with each core structure is a portion of a distributed tag directory. These tags are hashed to distribute workloads across the enabled cores. Physical addresses are also hashed to distribute memory accesses across the memory controllers. Each Xeon Phi core is dual-issue in-order, and includes 16 32-bit vector lanes. The performance of each core on sequential code is considerably slower than its multi-core counterpart. However, each core supports 4 hardware threads, resulting in good aggregate performance for highly parallelized and vectorized kernels. This makes the offload model, where sequential code runs on the host processor and parallelizable kernels are offloaded to the Xeon Phi, a suitable programming model. The Xeon Phi software stack consists of a host portion and coprocessor portion. The host portion asynchronous execution and data transfer between the host and Xeon Phi. The coprocessor portion of the software stack consists of a modified Linux kernel, drivers and the standard Linux proc file system that can be used to query device state (for example, the load average). The coprocessor portion also has a SCIF driver to communicate over the PCI bus with the host and other nodes. Together the current Xeon Phi software stack is referred to as the Many Integrated Core (MIC) Platform Software Stack or MPSS for short.
The invention may be implemented in hardware, firmware or software, or a combination of the three. Preferably the invention is implemented in a computer program executed on a programmable computer having a processor, a data storage system, volatile and non-volatile memory and/or storage elements, at least one input device and at least one output device.
Each computer program is tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
The invention has been described herein in considerable detail in order to comply with the patent Statutes and to provide those skilled in the art with the information needed to apply the novel principles and to construct and use such specialized components as are required. However, it is to be understood that the invention can be carried out by specifically different equipment and devices, and that various modifications, both as to the equipment details and operating procedures, can be accomplished without departing from the scope of the invention itself.

Claims

What is claimed is:

1. A multi-processor system, comprising:

a computer populated with one or more multiple-core coprocessors; and

a management code to control user processes containing offload blocks, including code to:

intercept coprocessor offload infrastructure application program interface (API) calls;

schedule user processes to run on one of the coprocessors;

schedule offloads within user processes to run on one of the coprocessors;

select one or more cores and create a core container consisting of the selected on a coprocessor to be used by an offload; and

obtain an optional thread-to-core mapping from a user and bind threads of an offload to the selected cores.

2. The system of claim 1, comprising one or more procedures, each of which selecting a process and offloading the process for scheduling.

3. The system of claim 1, comprising code with thread over-scheduling factors to enhance performance.

4. The system of claim 1, comprising code with memory over-scheduling factors to enhance performance.

5. The system of claim 1, comprising code to select cores to be assigned to an offload.

6. The system of claim 1, comprising code to apply instances of data structures from which a status of cores and information about selected cores are used to enhance core selection.

7. The system of claim 1, comprising code to communicate using messages between components of the management code.

8. The system of claim 1, comprising providing management code client module to control each user process and monitor processes from the host-side.

9. The system of claim 1, wherein each coprocessor comprise a plurality of cores running an operating system, one or more buses connecting one or more host processors and each coprocessor.

10. The system of claim 1, wherein the management code:

selects and schedules a pending process to a coprocessor with free memory; and

examines offload queues corresponding to each coprocessor, and dispatches an offload to each coprocessor with free threads,

wherein processes and offloads are selected based on an aging-based first-fit heuristic.

11. A method to manage a multi-processor system with one or more multiple-core coprocessors, comprising:

intercepting coprocessor offload infrastructure application program interface (API) calls;

scheduling user processes to run on one of the coprocessors;

scheduling offloads within user processes to run on one of the coprocessors; and

affinitizing offloads to predetermined cores within one of the coprocessors by selecting and allocating cores to an offload, and obtaining a thread-to-mapping from a user.

12. The method of claim 11, comprising applying an aging-based first-fit procedure for process and offload scheduling.

13. The method of claim 11, comprising applying thread or memory over-scheduling factors to enhance performance.

14. The method of claim 11, comprising applying greedy core selection such that offloads from the same process get preference to use the same cores.

15. The method of claim 11, comprising applying bitmaps to enhance core selection.

16. The method of claim 11, comprising messaging between components of the management code.

17. The method of claim 11, comprising providing management code client module to control each user process.

18. The method of claim 11, comprising monitoring processes from the host-side.

19. The method of claim 11, wherein each coprocessor comprise a plurality of X86 cores running Linux, Peripheral Component Interconnect Express Interface, and memory controllers connected with a bidirectional Interprocessor Network (IPN) ring.

20. The method of claim 11, further comprising:

selecting and scheduling a pending process to a coprocessor with free memory; and

examining offload queues corresponding to each coprocessor, and dispatch an offload to each coprocessor with free threads,