JP2008243203A - Workload control in virtualized data processing environment - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system, a method and a computer-readable medium, for balancing accesses of a computer system to physical system resources using system virtualization among a large number of logical partitions. <P>SOLUTION: The respective logical partitions are classified according to use levels of allocated dispatch windows, in initial start-up. Performance evaluation indexes about one or plurality of the physical system resources are determined in association with one or plurality of the logical partitions. The determination of the performance evaluation index is executed in a hardware level, independent of programming interruption. A given set of the physical system resources is allocated again to an alterative logical partition, according to the performance evaluation index determined in association with the alterative logical partition, and according to the dispatch window use classification of the alterative logical partition, in the dispatch window composed of the given set of physical system resources to be allocated to one of the logical partitions. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates generally to managing workloads in data processing systems. More specifically, the present invention relates to managing workloads in a partition system, such as a logical partition system.

  Logical partitioning of computer resources allows multiple system images to be established within a single physical machine or processor complex. Virtualization refers to the imaging of a system where each system image, also known as a virtual machine (VM), operates logically independent of other VMs that use the shared resources of that physical computer system. It is a term. In this way, each logical partition corresponding to a VM resets independently, loads an operating system that can be different for each partition, and uses different input / output (I / O) devices. Can be operated with different software programs. A commercially available form of logical partition system is, for example, IBM's POWER5 multiprocessor architecture.

  One important aspect of logical partitioning is managing the workload of each partition. For example, in POWER5, a workload manager called a hypervisor manages the workload between partitions. In this type of shared resource environment, the hypervisor uses memory, central processing unit (CPU), I / O, in a broad sense, using alternating time slot scheduling techniques similar to general multitasking computer scheduling. Allocate physical system resources such as O to logical partitions. The hypervisor attempts to balance the partition workload by dispatching the partition's work as a logical processor to physical system resources as needed and / or pre-assigned.

  One aspect of partition scheduling specifically relates to the use and sharing of processor resources. That is, partitions that use the processor capacity of the shared processor pool are defined as either having an upper limit or not having an upper limit for scheduling purposes. A capped partition cannot exceed the configured processor entitlement. Unbounded support for a logical partition allows an unbounded partition to exceed its set capacity if there is unused capacity in the shared processor pool. Such unused capacity is caused by other partitions not fully using all of their configured capacity, or otherwise the shared pool capacity is not fully allocated.

  When dispatched, the logical partition incorporates the allocated physical processor resources as a logical processor. The scheduling of a logical processor (sometimes called a virtual processor) is a pre-specified period of time during which processing cycles, memory, and physical system resources are allocated for use by a partition during a given dispatch window. It is necessary to allocate a time period, that is, a time slice. For example, an AIX operating system running on POWER5 has a 10 millisecond dispatch window by default. Any unused portion of the assigned dispatch window can be assigned to one or more of the uncapped partitions of the system. A lottery mechanism based on the priority level of an uncapped partition is often used to determine which uncapped partition replaces the original scheduled partition for unused portions of the dispatch window .

  Although the above alternative dispatch techniques are relatively simple and low in computational cost, they do not adequately address the potential inefficiencies associated with partition logical structure and functional characteristics. A significant source of scheduling inefficiency occurs when replacing so-called interactive partitions during their respective dispatch windows. A partition is characterized as "interactive" or "batch" based on its dependency on external processing events and its corresponding interruptability during a given dispatch window. Batch partitions are largely independent of responses from external events and therefore typically use the entire dispatch window. Interactive partitions, in contrast, typically suspend activity during the dispatch window and wait for external event responses.

  In order to take advantage of other unused cycles in the dispatch window where the interactive partition is suspended, the hypervisor uses the prioritized lottery mechanism described above to replace the suspended partition. Can try. In many cases, however, the suspended partition is waiting for an urgent external event response, and therefore terminates in the current dispatch window without replacing the partition, regardless of the partition's current suspension state. It is likely that an additional cycle will be required to complete the intended task.

  The dispatch window cycle is wasted if the suspended partition is not replaced during the partition inactivity period. On the other hand, the conventional partition replacement technology makes effective use of dispatch window cycles that would otherwise be wasted, but it reduces the computational cost of interrupting the interactive processing of the replaced interactive partition. I can't deal with it. Such an interrupt requires re-queuing the alternate interactive partition and cycling back through the queue to dispatch the partition again. Unlike dedicated systems, virtual systems need to re-establish the memory footprint for each dispatch. Therefore, in addition to requiring requeueing, alternate interactive partitions must spend additional cycles to recover memory footprint, which is a virtualized system Cause significant workload management inefficiencies.

  Conventional logical partition management cannot address the above-mentioned problems and many other problems related to partition scheduling and runtime workload balancing. Thus, it can be seen that there is a need for methods, systems, and computer programs for managing scheduling and workload balancing among logical partitions. The present invention addresses these and other needs that have not been resolved by the prior art.

  Disclosed herein are systems, methods, and computer-readable media for balancing access to physical system resources of a computer system using system virtualization across multiple logical partitions. Each logical partition is categorized according to the assigned dispatch window usage level at startup. Performance metrics for one or more of the physical system resources are determined in association with one or more of the logical partitions. The performance metric determination is performed during partition dispatch using hardware detection and tracking logic independent of programming interrupts. During a dispatch window in which a given set of physical system resources is configured to be assigned to one of the logical partitions, the given set of physical system resources is determined to be associated with an alternate logical partition Reassigned to the alternate logical partition according to the performance evaluation index and the dispatch window usage classification of the alternate logical partition.

  In another aspect, a method, system, and computer program for balancing workload among multiple logical partitions that share physical system resources is eligible for partition substitution using memory footprint statistics. Determine gender and priority. The method includes determining a performance metric for one or more of the physical system resources in association with the logical partition and using the performance metric to determine a memory footprint value. During a dispatch window in which a given set of physical system resources is assigned to one of the logical partitions, the given set of physical system resources is transferred to another logical partition according to the determined memory footprint value. Reassigned.

  In another aspect, disclosed is a method, system, and computer program for dynamically tuning a scheduler that schedules multiple logical partitions that share physical system resources during a given dispatch window. The The method includes dispatching the logical partition using a preset dispatch window period at system startup. During logical partition dispatch, performance metrics for one or more of the physical system resources are determined in association with the logical partition. The logical partition's memory used to dynamically determine partition scheduling among other scheduling heuristics during the dispatch window using performance metrics associated with the partition A footprint value is determined.

  The above as well as additional objectives, features, and advantages of the present invention will become apparent in the following detailed written description.

  The novel features believed characteristic of the invention are set forth in the appended claims. However, the invention itself, as well as its preferred mode of use, further objects and advantages, are best understood by referring to the following detailed description of exemplary embodiments when read in conjunction with the accompanying drawings. The

  The system and method of the present invention enables more efficient resource allocation and workload balancing within a virtualized computer environment. An exemplary virtualized computer environment includes a number of logical partitions with the workload managed between two or more of the partitions. As used herein, the term partition generally refers to a subset of data processing hardware resources allocated to the operating system. A partition may refer to a thread or any other computing unit. Preferred embodiments for implementing the systems and methods of the present invention are illustrated and described below with reference to the drawings, wherein like reference numerals refer to like corresponding parts throughout.

  The present invention enables dynamic redistribution of sharable resources among logical partitions under the direction of a workload manager. In one aspect, the present invention achieves improved workload management and system efficiency by using a dynamically adjustable partition scheduling mechanism. The partition scheduling mechanism features a hardware tracking mechanism that, in one embodiment, determines a partition performance metric associated with defining a memory footprint as a partition scheduling metric. In another aspect, the present invention dynamically adjusts the scheduling of each logical partition within the dispatch window using a hardware footprint memory footprint metric.

  The present invention enables dynamic redistribution of sharable resources among logical partitions under the direction of a workload manager, which in one embodiment may be a hypervisor. These resources can include, for example, CPU resources, logical processor resources, I / O resources, coprocessors, channel resources, and the like. In one embodiment, dynamic adjustment of resource allocation is performed in hypervisor functions and hardware and firmware in a performance tuning feedback loop to achieve workload balancing and overall system efficiency gains. This is achieved by integrating with a partition monitoring mechanism.

  In one aspect, the present invention addresses system throughput limitations caused by memory access latency. The present invention reduces the impact of memory latency by determining and using memory access statistics for partition scheduling. Such memory-dependent partition scheduling improves partition scheduling decisions and provides greater dispatch window scheduling freedom. In one embodiment, the present invention addresses the cost of setting the memory footprint in selecting a partition that preempts or otherwise replaces the original dispatched partition. It is. The present invention further clarifies the footprint setting cost when the replacement partition is redispatched to regain its part of the original dispatch window.

  Referring now to the drawings in which like reference numerals refer to like corresponding parts throughout, one embodiment of a virtualized computer environment implementing the workload management functionality of the present invention is shown in FIG. . Illustrated is a virtual computer system 100 that includes many of the functions included in a POWERS5 server provided by International Business Machines Corporation of Armonk, NY. The virtual computer system 100 generally includes a firmware layer resource 120 that includes a hypervisor 115 and a hardware layer resource 122 that includes a shared processor pool 117 and memory devices 121 and 125. The shared processor pool 117 preferably includes a number of processors 108, 110, 112, and 114, each labeled CPU1-CPU4, with a cache memory M1-M4 121 associated with each processor. A processor complex is provided. Virtual computer system 100 further includes a number of logical partitions 105A-105D, each labeled LP1-LP4. The hypervisor 115 manages and coordinates the allocation of hardware layer resources 122 among multiple logical partitions 105A-105D.

  CPU1-CPU4 and the cache memories M1-M4 associated with them represent some of the physical system resources that are allocated by the hypervisor 115 to the logical partitions LP1-LP4 to be resource virtualized. Physical system resources are generally distinguished from non-physical abstract system resources such as program layered organizations and program protocols such as those associated with operating systems, memory devices, processors, drivers, Tangible system devices, components, and related physical phenomena such as buses, processor / bus cycles. Physical system resources can also be distinguished from logically or virtually definable entities such as virtual machines. Each of the logical partitions LP1-LP4 includes one or more logical processors (not explicitly shown), and each of the logical processors is an entire or one of one of the physical processors CPU1-CPU4 assigned to the partition. Represents a part. A given logical processor in partitions 105A-105D can dedicate the lower processor resources to that partition to reserve for that partition, or defer lower processor resources to other partitions Can also be shared to make it available.

  In the illustrated embodiment, each of the logical partitions LP1-LP4 functions as a separate system having a resident operating system 104 and one or more applications 102 that may vary from partition to partition. In one embodiment, one or more of the operating systems 104A-104D may be a Linux operating system or an i5 / OS ™ operating system provided by IBM. In addition, operating systems 104A-104D (or a subset thereof) include respective OS workload managers 106A-106D to manage application workload within each of the respective partitions.

  In one embodiment, the hypervisor 115 operates as a hidden partition that does not have entitled capacity. The allocation of system resources to the logical partitions LP1-LP4 is managed by a hypervisor 115 that can be implemented by microcode running on the processors CPU1-CPU4. Hypervisor calls support a scheduling rule of thumb that any of the operating systems 104A-104D communicates with the hypervisor 115 to minimize partition idle time using techniques described in more detail below. Thereby providing a means for enabling more efficient use of physical processor capacity. Logical partitions LP1-LP4 and hypervisor 115 typically include one or more tangible program modules in respective portions of central memory associated with processors CPU1-CPU4.

  FIG. 2 is a high-level conceptual diagram illustrating an example architecture 200 adapted to facilitate partition scheduling, according to one embodiment of the present invention. The partition scheduling architecture 200 integrates a partition management unit (PMU) 204 with other system components such as a shared processor pool 117, hypervisor 115, and cache memory 206. Although PMU 204 is shown in FIG. 2 as a separate module, it should be noted that some or all of the hardware, firmware, and software components of PMU 204 may be integrated within hypervisor 115. is there. In addition, the cache block 206 may include some or all of the collection of cache memory resources M1-M4 used by one or more of the CPUs 108, 110, 112, 114 included in the shared processor pool 117. It should be noted that

  PMU 204 monitors physical system resource performance metrics, such as metrics related to memory usage, for resources allocated to partitions LP1-LP4, logic, program modules, and other hardware modules, firmware Includes modules and / or software modules. Such performance metrics preferably include cache usage and specifically include metrics related to each cache footprint of partitions LP1-LP4. The high level conceptual diagram of FIG. 2 illustrates an interface for integration and interaction between the PMU 204 and other system components that allow such monitoring of physical system resources associated with the logical partition.

  When the currently dispatched logical partition executes its instruction stream using the CPU of the shared processor pool 117 and accesses the contents of the memory location via a load or store operation, the CPU Issue to its associated cache 206 through the CPU-cache interface 212. The task of the cache 206 at that time determines whether or not the memory contents exist in the cache storage, and if (a) exists, the cached data is returned to the CPU, and (b) does not exist. Is to fetch its memory contents from main memory, such as shared memory 125, before performing the load or store. If the requested memory content is already in the cache 206, the data is returned to the CPU without accessing the shared memory 125 via the cache-memory interface 210 or the like. At this point, no interaction with the PMU 204 is necessary. However, if the requested data is not available in the cache 206, the data must be fetched from the main shared memory 125 through the cache-memory interface 210.

Referring to FIG. 3, a high level conceptual diagram showing the internal architecture of partition monitoring unit 204, hypervisor 115, and partition history table 305 that can be implemented within the architecture shown in FIG. The input side includes a PMU 204, which is shown as including a tracking logic module 302 that processes input from the CPU interface 208 and cache interface 214 to generate partition vectors 308, 310, and 312. On the output side, the architecture includes a hypervisor 115 and a partition history table 305. In the illustrated embodiment, the hypervisor 115 includes a priority value calculation module 304 that processes the partition vectors 308, 310, and 312 to generate and update the contents of the partition history table 305. The partition history table 305 includes items (entries) for all N logical partitions in the virtual computer system 100. In the illustrated embodiment, the partition history table 305 is shown as including a row record, each corresponding to one of the N logical partitions in the system, and each partition's record contains a number of columns. Contains data fields. Among the column fields are, for each logical partition, a logical partition (LP) identifier field, a field for cycle per instruction (CPI) value, and a cache line count (CLC) value. Fields, and fields for cache miss count (CMC) values, which are described in more detail below. The column field for each partition history table entry includes a field for the memory footprint period value T _FP in addition to the hardware detected CPI value, CLC value, and CMC value, and the footprint value variation VAR. And a field for a cache affinity (CA) value that can be derived from one or more of the CPI value, CLC value, and CMC value described above.

  Within the PMU 204, the tracking logic module 302 is a logic for detecting, processing, and temporarily storing performance metrics of physical system resources such as the processing resources and memory resources shown in FIGS. And data storage hardware devices. The performance evaluation index is detected in association with a logical partition to which physical system resources such as a memory and a CPU are allocated at the time of detection. The detected and processed performance evaluation index is stored in association with the identification information of the logical partition to which the physical resource is currently allocated. The association between a given set of performance metrics and a logical partition may be provided by a CPU interface register 314 that contains identification information for one or more currently dispatched partitions. The partition ID value in register 314 is preferably set when dispatch is determined.

  Exemplary performance metrics collected by the tracking logic module 302 include CPI counts, cache values, which can be determined directly or computationally from signals detected on the CPU interface 208 and / or the cache interface 214. Line counts, cache miss counts, and other metrics related to memory access or processing efficiency may be included. The processor usage resource register 322 can be used to provide a cycle count that measures activity during a time slice in which a partition is dispatched on a physical processor. In the illustrated embodiment, the CPI count for each of the currently dispatched partitions am detected by the tracking logic 302 is stored in the dispatched partition vector 308. Similarly, cache line counts and cache miss counts for dispatched logical partitions am detected by tracking logic 302 are stored in dispatched partition vectors 310 and 312 respectively. .

  In the preferred embodiment, the detection logic and data storage devices in tracking logic module 302 collect signals on CPU interface 208 and cache interface 214 independently of program interrupt mechanisms such as operating system interrupts. Hardware devices and firmware devices for processing. Such tracking devices and storage devices may include hardware such as logic gates, registers, etc., and firmware encoding as used by system bus snoopers. The tracking logic module 302 implements its detection, processing, and storage functions at the hardware and / or firmware level, independent of software program interrupts. Accordingly, such detection functions, processing functions, and storage functions are implemented independently of operating system kernel management constraints. In this way, the sampling rate at which the tracking logic module 302 collects performance metrics is used to determine the reference data used for partition scheduling and replacement as described in more detail below, eg, 0. It can have a sufficiently fine granularity, such as 1 millisecond.

  The system performance metrics in dispatched partition vectors 308, 310, and 312 are the CPI count, cache line for each of the currently dispatched or previously dispatched logical partition numbers am, respectively. Includes counts and cache miss counts. In FIG. 3, the association between each recorded partition vector value and the corresponding logical partition is visually represented by subscripts am, each representing a particular logical partition. The storage devices in tracking logic 302 that store partition vectors 308, 310, and 312 are preferably dedicated registers.

  The performance metrics detected by the tracking logic 302 and collected in the partition vectors 308, 310, and 312 are processed by the hypervisor 115 to update the partition history table 305. Interrupt signals generated or received by the hypervisor 115 determine when the partition history table 305 should be updated using system data collected in dispatched partition vectors 308, 310, and 312. Is done. The interrupt indicates the end of the dispatch window for at least one of the dispatched partitions am, or otherwise coincides with this. Upon receipt of the interrupt, the priority value calculation module 304 obtains and processes the partition vectors 308, 310, and 312 to add or update entries in the partition history table 305. In the illustrated embodiment, the CPI, CLC, and MC metrics in dispatched vectors 308, 310, and 312 are responsive to the update interrupt signal for each column for the corresponding logical partition record. It is compared with the evaluation index item or otherwise processed. Assuming that FIG. 3 shows the initial system startup time when the logical partition am is dispatched first, the priority value calculation module 304 stores a record for each of the partitions am in the partition history table 305. Add an entry and enter the partition vector data from the vectors 308, 310, and 312 into the corresponding record entry. The record generation process continues until records are stored in the partition history table 305 for all N logical partitions during system initialization.

  Following the initial dispatch and the subsequent partition record generation for each logical partition, the priority value calculation module 304 performs the dispatched vectors 308, 310, And 312 to add, replace, or otherwise modify the performance metrics and / or alternative priority value entries in the partition history table 305 as described in more detail below. Keep doing.

  FIG. 4 is a high level block diagram of a sequence for determining scheduling priorities for logical partitions in accordance with the present invention. Specifically, FIG. 4 illustrates an exemplary time and event based priority factor that can be utilized by the hypervisor 115 in making a dispatch decision, such as replacing or preempting a currently dispatched partition. .

  In order to incorporate priority into the overall partition dispatch scheduling as a dynamically adjustable factor, hypervisor 115 associates the relative or absolute priority of each logical partition with the cache footprint. Therefore, the performance evaluation index detected by the hardware must be acquired from the PMU 204.

  As described above, physical system performance metrics derived directly from the memory interface and CPU bus interface are detected using hardware level logic and registers within the tracking logic module 302 and are initially registered and processed. The The cache footprint metrics detected or generated by the tracking logic module 302 preferably include the CPI count, cache line count, and cache miss count for each dispatched partition, Collected as partition vectors 308, 310, and 312 that can be combined with other scheduling priority factors to derive dispatch priority.

  5-9, in conjunction with FIG. 4, specifically illustrate partition monitoring functions associated with substituting or preempting a dispatched partition or scheduling partition after the minimum partition usage entitlement has been assigned. Fig. 4 illustrates an exemplary embodiment of a method for incorporation in partition scheduling. First, the partition prioritization sequence shown in FIG. 4 includes a number of priorities including physical system performance metrics detected by hardware that are used to set and adjust priority values for the partition. Shows the accumulation and use of factors. FIG. 6 illustrates in more detail the process performed by the computer, which determines the performance metrics of the hardware detected physical system, particularly metrics related to memory access performance, Specifically used to characterize one or more partition scheduling priority factors for a given partition. FIGS. 7-9 illustrate computer-implemented processes for determining priorities and tuning dispatch windows using memory footprint metrics associated with partitions. The prioritization characterization and / or scheduler tuning shown in FIGS. 5 and 6 can be used with a hypervisor dispatcher, such as a dispatcher incorporated within the hypervisor 115. However, it should be noted that the features and techniques of the invention described herein are not necessarily limited to any one or more of the illustrated embodiments. Those skilled in the art will be able to modify various aspects of the process of determining and using partition priorities without departing from the spirit and scope of the present invention, as well as the mechanisms described herein. And it will be readily appreciated and understood that the basic process aspect or aspects can be used with other scheduling algorithms.

Continuing with FIG. 4, a partition P having a base priority value bp (P _i ) 402 that can be initially zero or otherwise intermediate so as not to affect partition scheduling / alternative decisions. _i is generated. The base priority value bp (P _i ) is a numeric or numerical indicia or other quantitative value or indicia that can be used by the hypervisor 115 to make partition preemption and / or alternative decisions. be able to. The overall priority value 425 of the partition P _i preferably includes a temporal fairness component that dynamically adjusts the base priority value bp (P _i ) 402. As shown in FIG. 4, the current priority cp (P _i , t) 406 at time t is cumulatively represented by the partition base priority value bp (P _i ) as the priority sum dΣt _j 404. It is calculated for each amount of time by increasing it by some time dependent increment d. Thus, the current priority cp (P _i , t) 406 includes the base priority plus an incremental fairness component that falls somewhere within the priority sum dΣt _j 404 according to the current time interval j. For partition dispatch that is not based on performance metrics, the logical partition priority is equal to the current priority cp (P _i , t) 406.

The present invention provides a partition scheduling mechanism that further provides and incorporates physical system performance metrics associated with the partition to determine the scheduling priority 425 of the partition P _i . This can be achieved by simultaneously calculating the priority of different partitions for a number of logical partitions running in a virtualized system using a shared set of physical processing resources (processors, memory, etc.). Means that may occur. Specifically, the performance-independent partition priority cp (P _i , t) 406 is adjusted 420 for each scheduling interval by a performance-based factor Δp 418. The performance-based factor Δp 418 represents one of a range of possible specific values, which can be used to increase or decrease the partition priority level using the system performance metrics associated with the partition. In one embodiment, the factor Δp 418 is calculated by using a reliability factor calculation module ζ 416 that processes the partition's cache footprint value CFP (P _i ). CFP (P _i ) itself is calculated as the cache footprint value of partition P _i quantified by the mechanism of the present invention via one or more logical functions represented by the cache footprint calculation module Θ 412. Is done. That is, the set of physical resource metrics PRM (P _i ) associated with one or more partitions is a specific memory footprint such as CFP (P _i ) that helps determine partition scheduling priority. To determine the value, it is processed by the cache footprint calculation module Θ 412. The footprint calculation module Θ 412 and the reliability factor calculation module ζ 416 determine a performance-based factor Δp 418. Thus, as shown in FIG. 4, the overall priority value of partition P _{i at} time t is a composite of the base partition priority, the time fairness adjustment, and the physical system metrics that are measured in relation to the partition. It is a function.

FIG. 5 is a high level block diagram of a partition dispatcher state that can be implemented in the hypervisor 115 to replace a dispatched but suspended partition in accordance with the present invention. An alternative partition vector or alternative queue 505 is generated and dynamically adjusted by the hypervisor 115 according to the methods disclosed herein. The replacement queue 505 may provide absolute replacement priority or relative replacement when, for example, a currently executing partition suspends processing before its dispatch window time expires during the current dispatch window. Accessed to determine priority. The alternate queue 505 is organized as a queue of preferred alternate objects LP _a -LP _n , each of which includes partition settings and coordinated dispatch scheduling states and alternate priority states among the partition specific data. Includes or otherwise links to corresponding partition control blocks (PCBs) 502a-502n, each holding partition state information. As shown in FIG. 5, the replacement objects LP _a -LP _n are changed from “maximum” (ie, highest priority among the available alternative partitions) to “minimum” (ie, of the available alternative partitions). Can be prioritized up to the lowest priority). The components of the alternate queue 505 have the advantage that they can be utilized to determine scheduling priorities, as will now be described with reference to FIG.

  Referring to FIG. 6, a high level flow diagram illustrating the steps performed by the partition monitoring unit 204 and the hypervisor 115 to determine the partition scheduling priority value according to the present invention is shown. The process begins by initializing and identifying the logical partition, such as by generating a corresponding partition control block as shown in FIG. 5 for each logical partition, as shown in steps 602 and 604. . As shown in steps 606 and 608, a base alternative priority is set for each of the partitions and the value is incremented. The hypervisor 115 preferably sets an alternative priority for the partition in the manner shown in FIG. 4 using a time fairness scheduling function to dynamically adjust the partition priority value in some normalized manner. And increase the value.

  Proceeding as shown in step 610, the logical partition is dispatched according to a set dispatch window assignment specified by the hypervisor 115, in one example. During the partition dispatch window, hardware-based tracking devices and modules, such as those in the tracking logic module 302, are used to associate each logical partition to which resources are allocated during the dispatch window. Track performance metrics for one or more of the physical system resources (step 615). Tracking performance metrics is performed at the hardware level independent of programming interrupts, such as operating system interrupts, and preferably includes tracking CPI and other physical resource processing metrics.

As shown in FIG. 6, physical resource metrics are tracked by the hypervisor 115 using the substep of tracking metrics associated with dispatched partitions and the hardware detected metrics. Calculating or otherwise determining a scheduling priority value that can be used for alternative or other dispatch decisions. As shown in step 612, the physical resource metric associated with the partition has a recording time increment Δt _rec that is less than the ratio of the partition dispatch period T _DP defined by the dispatch period divided by an increment factor greater than one. Collected and stored using hardware level logic and registers at speed. Following and / or in conjunction with the collection of physical resource metrics by hardware detection, a priority value associated with the partition is calculated (step 614). The priority value associated with the partition may be calculated from the hardware detected / stored physical system metrics, or may be the detected / stored metrics themselves. For example, referring to the embodiment shown in FIG. 3, the physical resource metrics collected in step 612 include the CPI values collected in the dispatched partition vector 308, and the priority value calculated in step 614. Includes memory footprint values calculated from CPI values in the process shown and described in more detail below with reference to FIGS. 8A and 8B. Concurrently with the determination of the performance metric in step 615, a reliability factor such as the VAR value shown in the partition history table 305 is calculated and updated in each dispatch window for the partition, as shown in step 616. It is preferable.

Referring to FIG. 8A, a more detailed representation of the processing steps incorporated in step 615 is provided. Specifically, FIG. 8A shows a high-level flow diagram showing the steps performed by PMU 204, hypervisor 115, etc. to determine the memory footprint performance value used for dispatch decisions. The process begins at partition dispatch step 610 and proceeds to step 806, where CPI data is determined and recorded at time interval point t _n within the dispatch period of the dispatched partition. The CPI value collected at t _n is compared with the previously recorded CPI value for the same partition. The previously recorded CPI value represents the CPI value determined at an earlier time within the dispatch period of the same partition. An exemplary graph of CPI data collected at various points in the dispatch period cycle is shown in FIG. 8B. In FIG. 8B, over a dispatch period indicated by T _DP, when the dispatch period starts is denoted as t _0, the time of completion is expressed as t _DP. In such one or more dispatches for the same partition, the point in time when CPI data is sought and recorded is _denoted as t _rec0 , t _rec1 , t _rec2 , t _rec3 , etc. Returning to step 808 of FIG. 8A, the comparison between the CPI data values may be, for example, is intended to include a comparison of the CPI values recorded at time _{t rec2,} a CPI value recorded at _{t rec1.} In the preferred embodiment, the determination of CPI data at different time increments is performed over different dispatch periods for a given logical partition.

The present invention reveals the substantial cost of dispatch associated with the need to redefine the memory footprint for each partition dispatch. The purpose of the comparison performed in step 808 is to determine the time period required for the partition to define its memory footprint. Determining the memory footprint definition includes determining a _cornerpoint as shown at t _rec3 in FIG. 8B, where the CPI value is the previous time increment of the previous dispatch period. From the recorded CPI value by a value smaller than a specific threshold value. In Figure 8B, the time _{t rec3 the} difference between the recorded CPI value in CPI value and _{t rec2} recorded at _{t rec3} threshold DerutaCPI _Thrshld smaller, represent these end points. Therefore, the footprint period T _fp is a period from t ₀ to t _rec3 . A determination is made at step 810 whether the difference between the CPI values is less than a particular threshold.

In response to the CPI value difference being greater than or equal to the threshold, the CPI value recorded at step 806 (ie, the last CPI value) is recorded (step 812) and returns to the next dispatch of the same partition at step 610. Before, the value of the recording interval is increased for the next dispatch period (step 814). When given comparison such as a comparison between the collected CPI value previously in CPI value and t _rec2 collected at t _rec3 has resulted in meeting the threshold criteria, the footprint defined for that partition The period is recorded in association with the partition identification information as in the partition history table 305 (step 816).

  Returning to FIG. 6 and proceeding as shown in step 618, the hypervisor 115 and / or PMU 204 may determine the use of each dispatch period of the logical partition. Using the determination in step 618, each logical partition is either a batch partition that uses substantially all of the configured dispatch period assignments, or interrupt handling that the partition must wait for external events during the dispatch period. Classified as either of the interactive partitions that you receive. A dispatched partition, categorized as a batch partition, uses substantially the entire dispatch period, while an interactive partition is interrupted, such as when processing is interrupted, and receives a response from another process. have to wait. If the use of the dispatch period for a given partition is above a certain threshold, which in one embodiment is 95%, the partition is classified as a batch partition (steps 618 and 622). Otherwise, the partition is classified as interactive, as shown in steps 618 and 620.

  The dispatch period usage classification is included as a field in the partition control block of each partition, as previously shown in FIG. 5, and can be used by the hypervisor 115 as a scheduling priority criterion. For example, the hypervisor 115 may exclude a partition that has been identified as interactive from eligibility as an alternate partition in the alternate queue 505 (see FIG. 5). Further, as will be described below with reference to FIG. 7, the classification of dispatch period usage is whether the hypervisor 115 reassigns the rest of the dispatch window to the original partition that was intercepted or replaced. Can be used to determine whether.

  The process of determining partition scheduling priority values enters or updates priority values such as priority values stored in partition history table 305 and PCBs 502a-502n and ends as shown in steps 624 and 626. To do. As shown and described above with reference to FIGS. 3-5, hardware detected performance metrics are associated with logical partitions, such as in the partition history table 305, and are replaced by the hypervisor 115 with an alternate queue 505. Used to determine the priority as established by. In a preferred embodiment, the dispatch period usage classification and performance metrics determination can be measured in time (eg, 5 minutes) or by a specific number of rotations through the partition dispatch queue. Can be implemented at startup. After collecting at least an initial set of priority data, the hypervisor 115 uses that data for replacement and possibly preemption of dispatched partitions, as well as other scheduling decisions.

  FIG. 7 is a high-level flow diagram illustrating the steps performed by a dispatcher, such as a dispatcher in hypervisor 115, to balance the workload between logical partitions in accordance with the present invention. The process begins with the hypervisor 115 dispatching the next set of one or more partitions according to the partition scheduling set by the system, as shown in steps 702 and 710. During the current dispatch cycle, physical system resource metrics, including CPI and CLC of the dispatched partition, are recorded and updated (step 712), such as the alternative priority value calculated in step 614 of FIG. The alternative priority value is updated (step 713).

  As shown in step 714, the hypervisor 115 may determine whether a given dispatched partition can be intercepted according to the partition replacement priority data. As used herein, partition preemption is similar to and includes many of the same mechanisms used to replace dispatched partitions that were yielded during a dispatch window. The difference is that pre-emption of a partition does not require the replacement partition to be interrupted as a condition that initiates the replacement step described herein. As shown in steps 714 and 720, for a given dispatched partition, it was determined to meet pre-specified preemption criteria in terms of priority values, such as the priority contained in the partition history table 305. In response, the hypervisor 115 suspends the dispatched partition and allocates system resources to the selected partition for the remainder of the dispatch window (possibly its entirety). The alternate partition is selected according to the alternate priority data in the same or similar manner as the scheduling prioritization step described below with reference to step 718 and FIG. The hypervisor 115 uses the priority data in the alternate queue 505 and partition control blocks 502a-502n to determine whether the next dispatched partition or otherwise the currently dispatched partition is selected by the selected alternate partition. Determine whether it will be intercepted.

  If the preemption criteria are not met (step 714) and the original scheduled partition uses the entire dispatch allocation period (ie, the partition does not interrupt processing), then the hypervisor dispatch / load balancing process is The next dispatch in step 710 continues.

  If a partition that has been dispatched and has not been preempted suspends processing during the dispatch window (step 716), the hypervisor 115 may replace the alternate queue 505 and / or partition control for available partitions. The replacement priority data provided by blocks 502a-502n is processed to determine whether pre-specified partition replacement criteria are met (step 718).

The alternative determination shown in step 718 preferably evaluates the alternative priority value of the currently idle logical partition to determine which partition is eligible to replace the currently suspended partition. Determining. Substitution eligibility determination is based on partition substitution priorities such as memory footprint values contained in partition control blocks 502a-502n in view of imposed restrictions such as the remaining portion of the dispatch window is limited. Evaluate one or more of the values. For example, the original dispatched partition has a partition window period _TDW that is 10 milliseconds for the IBM POWER5 architecture. The dispatch window period T _DW is the minimum run-time increment set for the partition and, as a result, the smallest increment that the hypervisor 115 can replace or preempt the dispatched partition. Effectively divided into dispatch increments T _DPSTCH . Under such circumstances, the determination of the alternative criteria in step 718 is that of the available alternative partitions in view of the restrictions imposed by T _DPSTCH and the rest of the T _DW for the suspended partition. Including determining which are eligible.

  The present invention includes a dispatch selection function that guarantees a certain level of scheduling efficiency. That is, referring to FIG. 9, in determining whether an alternative criterion is met (step 718) and when selecting an alternative partition to ensure a certain level of processing efficiency, the partition scheduler A high level flow diagram showing the steps performed is shown. The process begins by prompting the alternative criteria to be determined by an alternative event, such as meeting preemption criteria or suspending the dispatched partition, as shown in steps 902 and 904. The alternative criteria determination shown in step 718 is a sub-step for selecting an alternative partition using both the memory footprint priority value for the available partition and the remaining available portion of the dispatch window. 906 and 908 are included.

Step 906 determines whether the footprint stipulated period T _fp for the available alternate partition that can be recorded in the partition control block and / or the partition history table 305 is less than the _remaining portion T dispatching of the dispatch window. Indicates a decision. Further, the determination shown in step 906 is that the T _fp value for a partition is sufficiently small that the alternate partition can achieve a certain level of processing efficiency within the available dispatch window period T _remaining . To determine whether we use a tunable factor x that can be set to a value greater than one. For example, in one embodiment, the hypervisor 115 provides a predetermined memory footprint value at which the alternate logical partition satisfies the relationship xT _fp ≦ T _remaining when x is greater than 1, preferably at least 10. An alternative decision is made in step 718 depending on whether T _fp is present.

  As shown in steps 906 and 908, if the criteria for the alternative priority value are not met, the partition is excluded from the alternative. Step 906 is performed for one or more of the partitions until the replacement criteria are met, and the process ends as indicated in step 910.

  Returning to FIG. 7, in response to selecting an alternate partition, hypervisor 115 deallocates the dispatch window resource from the replaced partition and allocates the resource to the selected alternate partition (step 720). . If the replacement partition is classified as non-interactive (ie, batch) as shown and described with reference to FIGS. 5 and 6, the replacement partition consumes the remaining dispatch window period. The process returns to the next dispatch cycle (steps 722 and 710). However, if the replacement partition is an interactive partition (step 722), the replacement partition is dispatched over a subset of the remaining dispatch window periods (step 723). Following the end of the alternate dispatch period, the dispatch window resource is reallocated to the original alternate partition (step 724) and the process continues (step 726) or ends (step 728).

FIG. 10 illustrates partition scheduling within a dispatch window according to the present invention. The dispatch window shown in FIG. 10 is defined in part as having a set dispatch window period T _DW that starts at t _start and ends at t _finish , which is a partition over a specific dispatch window interval. Implemented by a counter function (not explicitly shown) in the hypervisor 115 that gives the hypervisor 115 a timed interrupt of activity. In the illustrated embodiment, the partitions represented as P ₁ , P ₂ , and P ₃ are interleaved time intervals starting at times t _start , t ₁ , and t ₂ , respectively, in the dispatch window. , And dispatched by the hypervisor 115. According to the logical partition scheduling rules, each of the partitions P ₁ , P ₂ , and P ₃ is preconfigured that the hypervisor 115 uses for scheduling to ensure that the partition receives its respective minimum usage entitlement. Eligible to use the dispatch period.

In the illustrated embodiment, the minimum usage entitlements for partitions P ₁ , P ₂ , and P ₃ are shown in FIG. 10 as periods of t _start −t ₁ , t ₁ −t ₂ , and t ₂ −t ₃ , respectively. It is. After the eligibility period, there is a remaining period t ₃ -t _finish where the partitions P ₁ , P ₂ , and P ₃ may be dispatched by the hypervisor 115 according to fairness and scheduling priority factors. Further, as explained above, the partitions, as shown for partition P _2, which may interrupt the process to the dispatch period of entitlement.

11 is a high level flow diagram illustrating tuning of partition scheduling during the dispatch window etc. shown in FIG. 10 according to the present invention. The process begins by receiving a hard interrupt signal (ie, independent of partition activity) for the start of the next dispatch window, as shown in step 1102. Proceeding as shown in step 1104, a logical partition, such as one of partitions P ₁ , P ₂ , and P ₃ , is dispatched for a period of time corresponding to the partition's configured cycle usage. Referring to FIG. 10, the first dispatch period is for partition P ₁ dispatched at t _start for a specific number of cycles.

When the logical processor belonging to the dispatched partition does not interrupt the process for the partition P ₁ dispatched to, for example, t _start -t ₁ (step 1106), and further usage entitlement needs to be assigned (step 1108). The next scheduled partition (eg, partition P ₂ ) is dispatched (step 1104). If the entire dispatch window period has been consumed after assigning the minimum usage entitlement (step 1110), the process returns to step 1102 for the next dispatch window period.

In addition to dispatching to meet partition usage eligibility, the hypervisor 115 also determines a priority factor, such as a memory footprint value derived from the performance metrics associated with the partition, from a scheduling rule of thumb. It is preferable to use and dynamically schedule partitions. One situation in which the hypervisor 115 uses priorities derived from performance metrics for scheduling is interrupted as shown in FIG. 10, at time t _sus , when partition P ₂ interrupted processing. The logical processor state (step 1106) is shown in FIG. Another such situation shown in FIG. 11 is a scheduling after use credentials additional cycle begins at t ₃ when remaining in the dispatch window (steps 1108 and 1110).

  In response to the partition processing interruption (step 1106) or the availability of additional cycles in the dispatch window, the hypervisor 115 begins dispatching in step 720, which includes the following sub-steps. The dispatch eligibility of the partition is determined by comparing the time defining the memory footprint for each partition with the remaining time of the dispatch window (step 1112). The partitions that are eligible through step 1112 are then prioritized according to their respective footprint definition costs, as shown in step 1114. For example, the stored statistics indicate the time required for each partition to define a memory footprint. As described above with reference to FIG. 9, the relative efficiency of dispatching a particular partition with a limited memory window remaining using a memory footprint definition period Can be determined. Accordingly, the prioritization performed in step 1114 may include the steps shown in FIGS. The partitions selected according to the prioritization at step 1114 are dispatched as indicated at step 1116.

  The disclosed method can be implemented as software using objects or using an object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation hardware platforms. It can be easily implemented. In this case, the method and system of the present invention includes a routine embedded in a personal computer, such as a Java or CGI script, a resource residing on a server or graphics workstation, a dedicated source code editor. It can be implemented as a routine embedded in the management system.

  Although the invention has been particularly shown and described with reference to preferred embodiments, those skilled in the art can make various changes in form and detail without departing from the spirit and scope of the invention. You should understand what you can do. All such alternative embodiments are within the scope of the invention.

1 is a virtualized computer system adapted to implement workload balancing and dispatch window tuning according to the present invention. FIG. FIG. 3 is a high-level conceptual diagram illustrating an example architecture adapted to facilitate partition scheduling, according to one embodiment of the invention. FIG. 3 is a high-level conceptual diagram illustrating the internal architecture of a partition monitoring unit, hypervisor, and partition history table that can be implemented within the architecture shown in FIG. FIG. 4 is a high level block diagram of a sequence for determining alternative priorities for logical partitions according to the present invention. FIG. 4 is a high level block diagram of a partition dispatcher state implemented in accordance with the present invention. FIG. 5 is a high level flow diagram illustrating the steps performed by a partition monitoring unit and dispatcher to determine alternative priorities according to the present invention. FIG. 5 is a high level flow diagram illustrating steps performed during a partition dispatch process that balances workload among logical partitions using alternative priorities in accordance with the present invention. (A) A high level flow diagram illustrating the steps performed by a partition monitoring unit to determine a memory footprint performance evaluation index utilized for dispatch substitution decisions, according to one embodiment of the present invention. (B) is a graph of cycle per instruction data collected over a dispatch window cycle in accordance with the present invention. FIG. 5 is a high level flow diagram illustrating the steps performed by the partition scheduler in selecting an alternate partition. Fig. 5 shows partition scheduling within a dispatch window according to the present invention. FIG. 6 is a high level flow diagram illustrating steps performed by a partition monitoring unit to dynamically tune a partition scheduler during a dispatch window, in accordance with the present invention.

Claims (20)

In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the physical system between the number of logical partitions A method for balancing access to resources,
Using hardware detection logic to collect performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Classifying each of the logical partitions according to the level of usage of the assigned dispatch window;
Collected performance metrics associated with an alternative logical partition and the alternative logical partition during a dispatch window in which a given set of physical system resources is preconfigured to be assigned to one of the logical partitions Assigning the given set of physical system resources to the alternate logical partition according to a dispatch window usage classification of:
Including methods. Storing the performance metrics in association with each of the respective associated logical partitions;
Assigning the given set of physical system resources to the alternative logical partition further comprises selecting the alternative logical partition according to the stored performance metrics;
The method of claim 1. Dispatching one logical processor of the logical partition during a dispatch window, the dispatching step being pre-designated to allocate the physical system resources to be used by the one of the logical partitions Assigning a specified period of time; and
In response to the dispatched logical processor suspending processing during the pre-specified period, the logic to reallocate the physical system resources to the remaining portion of the pre-specified period. Selecting another logical partition of the partitions, wherein the selecting step is performed according to the collected performance metrics associated with the other logical partition of the logical partitions;
The method of claim 1, further comprising: In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the physical system between the number of logical partitions A system for balancing access to resources,
Hardware detection logic for collecting performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Partition scheduling logic to classify each of the logical partitions according to a level of usage of an assigned dispatch window;
Collected performance metrics associated with an alternative logical partition and the alternative logical partition during a dispatch window in which a given set of physical system resources is preconfigured to be assigned to one of the logical partitions Partition scheduling logic for assigning the given set of physical system resources to the alternative logical partition according to a dispatch window usage classification of
Including system. Means for storing the performance metrics in association with each of the respective associated logical partitions;
The partition scheduling logic for assigning the given set of physical system resources to the alternative logical partition further comprises partition scheduling logic for selecting the alternative logical partition according to the stored performance metrics. Including,
The system according to claim 4. Partition scheduling logic for dispatching one logical processor of the logical partition during a dispatch window, the dispatching being used by the one of the logical partitions as the physical system resource Partition scheduling logic, including allocating a pre-specified period in which
In response to the dispatched logical processor suspending processing during the pre-specified period, the logic to reallocate the physical system resources to the remaining portion of the pre-specified period. Partition scheduling logic for selecting another logical partition of the partitions, wherein the selection is performed according to the collected performance metrics associated with the other logical partition of the logical partitions; Partition scheduling logic;
The system of claim 4 further comprising: In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the computer system includes the plurality of logical partitions. A program for executing processing for balancing access to physical system resources among logical partitions, the program comprising:
Using hardware detection logic to collect performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Classifying each of the logical partitions according to the level of usage of the assigned dispatch window;
Collected performance metrics associated with an alternative logical partition and the alternative logical partition during a dispatch window in which a given set of physical system resources is preconfigured to be assigned to one of the logical partitions Assigning the given set of physical system resources to the alternate logical partition according to a dispatch window usage classification of:
A program adapted to perform a method comprising: The method further comprises storing the performance metrics in association with each of the respective associated logical partitions;
Assigning the given set of physical system resources to the alternative logical partition further comprises selecting the alternative logical partition according to the stored performance metrics;
The program according to claim 7. The method
Dispatching one logical processor of the logical partition during a dispatch window, the dispatching step being pre-designated to allocate the physical system resources to be used by the one of the logical partitions Assigning a specified period of time; and
In response to the dispatched logical processor suspending processing during the pre-specified period, the logic to reallocate the physical system resources to the remaining portion of the pre-specified period. Selecting another logical partition of the partitions, wherein the selecting step is performed according to the collected performance metrics associated with the other logical partition of the logical partitions;
The program according to claim 7, further comprising: In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the physical system between the number of logical partitions A method for balancing access to resources,
Using hardware detection logic to collect performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Processing the performance metric to determine a memory footprint value for the logical partition;
During a dispatch window in which a given set of physical system resources is assigned to one of the logical partitions, the given set of physical system resources according to the determined memory footprint value. Assigning a set to another of the logical partitions;
Including methods.   The method of claim 10, wherein the performance metric includes data per cycle of instructions, and the memory footprint value includes a period for a logical partition to define a memory footprint.   The period for defining the memory footprint is such that the cycle value per instruction at some point in the dispatch period for a partition is the cycle value per instruction at an earlier point in the dispatch period for the same partition. The method of claim 11, wherein the method is determined by detecting an end point that varies by less than a certain threshold from. In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the physical system between the number of logical partitions A system for balancing access to resources,
Hardware detection logic for collecting performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Partition monitoring logic for processing the performance metrics to determine a memory footprint value for the logical partition;
During a dispatch window in which a given set of physical system resources is assigned to one of the logical partitions, the given set of physical system resources according to the determined memory footprint value. Partition scheduling logic for assigning a set to another of the logical partitions;
Including system.   The system of claim 13, wherein the performance metric includes data for cycles per instruction, and the memory footprint value includes a period for a logical partition to define a memory footprint.   15. The system of claim 14, wherein the hardware detection logic includes hardware circuit means for collecting data for cycles per instruction over successive dispatches for a given one of the logical partitions. Collecting data for cycles per instruction over successive dispatches for a given one of the logical partitions;
Collecting data for cycles per instruction in the nth time increment within the mth dispatch window for the given logical partition;
Let m, n, p, and q be integers greater than or equal to 1, per instruction at (n + p) th time increment in the (m + q) th dispatch window for the given logical partition. Collecting data for the cycle;
The system of claim 15, further comprising: The partition scheduling logic for allocating the given set of physical system resources to another logical partition of the logical partition is such that T _unitized represents the unused portion of the dispatch window period and x is 1. _Assuming a larger efficiency multiplier, the physical system depends on whether the other logical partition of the logical partitions has a determined memory footprint value T _fp that satisfies the relationship xT _fp ≦ T _unitized 14. The system of claim 13, further comprising partition scheduling logic for assigning the given set of resources to the other logical partition of the logical partitions. In a computer system using system virtualization partitioning, wherein each of a number of logical partitions operates logically independent of other partitions that use the computer system's shared physical resources, the computer system includes the plurality of logical partitions. A program for executing processing for balancing access to physical system resources among logical partitions, the program comprising:
Using hardware detection logic to collect performance metrics for one or more of the physical system resources in association with one or more of the logical partitions;
Processing the performance metric to determine a memory footprint value for the logical partition;
During a dispatch window in which a given set of physical system resources is assigned to one of the logical partitions, the given set of physical system resources according to the determined memory footprint value. Assigning a set to another of the logical partitions;
A program adapted to perform a method comprising:   The program according to claim 18, wherein the performance evaluation index includes data of a cycle per instruction, and the memory footprint value includes a period for the logical partition to define a memory footprint.   The period for defining the memory footprint is such that the cycle value per instruction at some point in the dispatch period for a partition is the cycle value per instruction at an earlier point in the dispatch period for the same partition. 20. The program according to claim 19, wherein the program is obtained by detecting an end point that changes by an amount smaller than a specific threshold value.

Images