DE102013207603B4 - Run jobs efficiently in a shared pool of resources - Google Patents

Abstract

Embodiments of the invention relate to a shared set of resources and the efficient processing of one or more jobs in the shared set of resources. The shared set of resources provides tools to access and organize a topology of the shared resources, including physical and virtual machines, as well as storage devices. The topology is stored in a known location and used for the efficient assignment of one or more jobs in response to the hierarchy.

Description

BACKGROUND

The present invention relates to an efficient approach to using job processing in a shared pool of resources. In particular, the invention relates to evaluating the virtual and physical topology of the shared resources and processing jobs in response to the combined topology.

MapReduce is a framework for processing highly distributable tasks over huge data sets using a large number of computer nodes. In cases where all nodes use the same hardware, or in a grid where the nodes use different hardware, the framework is generally referred to as a cluster. Computer-related processing can be carried out for data that is either stored in a file system or in a database. In particular, a master node (or higher-level node) receives a job entry and divides the job into smaller sub-jobs, which are distributed to the other nodes in the cluster or grid. In one embodiment, the nodes in the cluster or grid are arranged in a hierarchy and the sub-jobs can be further divided and distributed. The nodes responsible for processing the sub-jobs return processed data to the master node. In particular, the processed data is collected and combined by the master node to form an output. Accordingly, MapReduce is an algorithm-based technique for the distributed processing of large amounts of data belonging to a job.

As described above, MapReduce enables data processing to be distributed across a network of nodes. Although MapReduce is very comfortable to use, current problems with MapReduce mean that performance problems arise when processing jobs.

Various documents have already been published in the context of this technological status: The document Lee, G. et al .: Topology-Aware Resource Allocation for data-Intensive Worksloads, in Newsletter ACM SIGCOMM, Vol. 4, Jan. 2001, pp. 120-124 describes an optimized resource allocation in IaaS-based cloud systems. For this purpose, a prototype for a topology-aware resource allocation (TARA) system was developed and tested with 80 servers in a cluster with two representative MapReduce-based benchmarks.

The document from Diakhate, F. et al .: Efficient Shared Memory Passing for Inter-VM Communications, in Euro-Par 2008 Workshops, LNCS 5415, Springer-Verlag Berlin Heidelberg 2009, pp. 53-62 addresses the topic of inter-VM-MPI communication if VMs are present on the same physical machine. Based on this, an efficient MPI library for virtual machines can be implemented.

Despite the progress made so far in processing large amounts of data, there is still a need - especially with MapReduce technologies - for further improvements to avoid bottlenecks for individual assigned VMs.

BRIEF SUMMARY

Such improvements are described in the subject matter of the independent claims.

Further embodiments are specified by the respective dependent claims.

Further features and advantages of the present invention will become apparent from the following detailed description of the current preferred embodiment of the invention in conjunction with the accompanying drawings.

Figure list

• 1 stellt einen Cloud-Computingknoten gemäß einer Ausführungsform der vorliegenden Erfindung dar.
• 2 stellt eine Cloud-Computingumgebung gemäß einer Ausführungsform der vorliegenden Erfindung dar.
• 3 stellt Abstraktionsmodellschichten gemäß einer Ausführungsform der vorliegenden Erfindung dar.
• 4 stellt ein Blockdiagramm dar, das die Architektur zum Einsatz des die Cloud erfassenden MapReduce darstellt.
• 5 stellt ein Flussdiagramm dar, das die Ressourcenzuweisung und die Verwaltung der Daten sowie die Anordnung der virtuellen Maschine veranschaulicht.
• 6 stellt einen Flussgraphen dar, der eine Beispielnetzwerktopologie für die Datenanordnung veranschaulicht.
• 7 stellt einen Flussgraphen für eine beispielhafte Anordnung der Daten dar.
• 8 stellt einen Flussgraphen für die Anordnung einer virtuellen Maschine dar.
• 9 stellt eine Tabelle mit Werten dar, die dem Flussgraphen für die Anordnung der virtuellen Maschine und zur Knotenkatalogisierung zugeordnet werden.
• 10 stellt ein Ablaufdiagramm dar, das einem Prozess der Auswertung und Nutzung der Topologie der physischen und virtuellen Maschine in einem gemeinsamen Pool von Ressourcen veranschaulicht.
• 11 stellt ein Flussdiagramm dar, das die Schritte zur Unterstützung des Aspekts der Nutzung der Speichertopologie des gemeinsamen Pools von Ressourcen veranschaulicht.
• 12 stellt ein Blockdiagramm dar, das in einem Computersystem integrierte Tools zur Unterstützung einer Technik veranschaulicht, die zur Auswertung der Ressourcennutzung für die Zuweisung eines Jobs in einem gemeinsamen Pool von Ressourcen zum Einsatz kommt.
• 13 stellt ein Blockdiagramm dar, das ein System zur Implementierung einer Ausführungsform der vorliegenden Erfindung zeigt.

The drawings to which reference is made form part of the patent. Features shown in the drawings serve only to illustrate some embodiments of the invention and not all embodiments of the invention, unless expressly stated otherwise.

• 1 illustrates a cloud computing node according to an embodiment of the present invention.
• 2nd illustrates a cloud computing environment according to an embodiment of the present invention.
• 3rd illustrates abstraction model layers according to an embodiment of the present invention.
• 4th is a block diagram showing the architecture for using the MapReduce that captures the cloud.
• 5 is a flowchart that shows resource allocation and data management, as well as the arrangement of the virtual machine.
• 6 Figure 14 is a flowchart that illustrates an example network topology for data ordering.
• 7 represents a flow graph for an exemplary arrangement of the data.
• 8th represents a flow graph for the arrangement of a virtual machine.
• 9 represents a table with values that are assigned to the flow graph for the arrangement of the virtual machine and for node cataloging.
• 10th is a flowchart illustrating a process of evaluating and using the topology of the physical and virtual machines in a shared pool of resources.
• 11 FIG. 13 is a flowchart illustrating the steps to support the aspect of using the shared pool of resources storage topology.
• 12 FIG. 3 is a block diagram illustrating tools integrated in a computer system to support a technique used to evaluate resource usage for assigning a job to a shared pool of resources.
• 13 FIG. 12 is a block diagram showing a system for implementing an embodiment of the present invention.

DETAILED DESCRIPTION

It is readily appreciated that the components of the present invention, as generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the device, system and method of the present invention, as shown in the figures, is not intended to limit the scope of the invention according to the claims, but is only for selected embodiments of the invention.

The functional unit (s) described in this patent specification were identified with tools in the form of manager (s) and director (s). A manager or director can be implemented in programmable hardware devices, such as field programmable gate arrays (or universal circuits, field programmable gate arrays), programmable array logic, programmable logic devices or the like. The manager or director (s) can also be implemented in software for processing by different processor types. For example, an identified manager or director consisting of executable code may have one or more physical or logical blocks of computer instructions that may be organized, for example, as an object, procedure, function, or other construct. However, the executable code of an identified manager or director does not have to be physically in one place, but can have separate instructions stored in different locations that, when logically combined, contain the managers and director and the stated purpose of the manager and director realize.

In fact, an executable code manager or director can be a single instruction or many instructions, and can even be distributed across different code segments, across different applications, and across different storage devices. Likewise, operational data in the manager or director can be identified and illustrated here and can be executed in any suitable form and organized in any suitable data structure type. The operational data can be captured as a single data set or can be distributed across different storage locations, including across different storage devices, and can, at least in part, exist as electronic signals on a system or network.

Any reference in this specification to "a selected embodiment", "a single embodiment", "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment in at least one embodiment of the invention is included. Thus, the occurrences of the terms "a selected embodiment", "in a single embodiment", "in one embodiment" at different places in this specification do not necessarily refer to the same embodiment.

Furthermore, the described features, structures or properties can be combined in any suitable manner in one or in several embodiments. The following description provides various specific details, such as examples of a topology manager, a hook manager, a storage topology manager, a resource usage manager, an application manager, a director, etc., to provide a thorough understanding of the embodiments of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without one or more of the specific details or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

The illustrated embodiments of the invention can best be understood by referring to the drawings, wherein like elements are designated by like reference numerals. The following description is provided by way of example only and simply illustrates certain selected embodiments of the devices, systems, and processes consistent with the invention as claimed herein.

A cloud computing environment is service-oriented, with a focus on stateless, modular and semantic interoperability with little coupling. At its core, cloud computing consists of an infrastructure that has a network of interconnected nodes. The following is now going on 1 Reference, in which a schematic representation of a cloud computing node is shown. The cloud computing node ( 10th ) is only one example of a suitable cloud computing node and is not intended to indicate a limitation on the scope or functionality of the embodiments of the invention described herein. Regardless, the cloud computing node ( 10th ) are implemented and / or perform any of the functions described above. In the cloud computing node ( 10th ) there is a computer system / server ( 12 ) that can work with many other general or specific computer system environments or configurations. Examples of known computer systems, environments and / or configurations that are suitable for use with the computer system / server ( 12 ) may be, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, portable or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networks -PCs, minicomputer systems, mainframe computer systems and distributed cloud computing environments that include any of the above systems or devices, and the like.

The computer system / server ( 12 ) can be described in the general context of instructions executable on a computer system, such as program modules that are executed by a computer system. In general, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform certain tasks or implement certain abstract data types. The computer system / server ( 12 ) can be performed in distributed cloud computing environments where jobs are performed by remote processing devices that are connected through a communication network. In a distributed cloud computing environment, program modules can reside on both local and remote storage media of the computer system, including main memory storage devices.

As in 1 shown, a computer system / server ( 12 ) in the cloud computing node ( 10th ) as a general computing device. The components of a computer system / server ( 12 ) may include, but are not limited to, one or more processors or processing units ( 16 ), system memory ( 28 ) and a bus ( 18th ), the various system components including system memory ( 28 ) with the processor ( 16 ) couples. The bus ( 18th ) represents one or more of any of various types of bus structure, including a memory bus or memory controller, a peripheral bus, an accelerated graphic port, and a processor or local bus Use any of various bus architectures. These architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) - Bus, the Enhanced ISA (EISA) bus, the local Video Electronics Standards Association (VESA) bus and the Peripheral Component Interconnect (PCI) bus. To the computer system / server ( 12 ) usually includes a variety of computer-readable media. Such media can be any available media on which a computer system / server ( 12 ), and it includes both volatile and non-volatile media, removable and non-removable media.

The system memory ( 28 ) media that can be read by a computer system in the form of volatile memory, such as random access memory (RAM, Random Access Memory) ( 30th ) and / or cache memory ( 32 ) include. The computer system / server ( 12 ) may further include other removable / non-removable storage media of the computer system. For example, a storage system ( 34 ) are provided to read and write to non-removable, non-volatile magnetic media (not shown and typically referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading and writing to a removable, non-volatile magnetic disk medium (such as a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk medium, such as a CD -ROM, DVD-ROM or another optical medium can be provided. In such cases, everyone can take the bus ( 18th ) connected by one or more data media interfaces. As will be further illustrated and described below, a memory ( 28 ) contain at least one program product with a set of (e.g. at least one) program modules which are designed to carry out the functions of the embodiments of the invention.

A program / utility program (or utility) (40) with a set of (at least one) program modules ( 42 ) can, for example, and without limitation in the memory (Memory) ( 28 ) are saved, as is an operating system, one or more application programs, further program modules and program data. Each of the operating systems, one or more application programs, further program modules and program data or a specific combination thereof can include an implementation of a network environment. Program modules ( 42 generally perform the functions and / or methodologies of the embodiments of the invention as described in this document.

The computer system / server ( 12 ) can also be used with one or more external devices ( 14 ) communicate, such as a keyboard, a pointing device, a display ( 24th ) etc .; one or more devices that enable a user to communicate with the computer system / server ( 12 ) to interact; and / or any devices (e.g. network card, modem, etc.) that a computer system / server ( 12 ) enable communication with one or more other computing devices. Such communication can be done via input / output (I / O or input / output) interfaces ( 22 ) respectively. And further a computer system / server ( 12 ) with one or more networks via a network adapter ( 20th ) communicate, such as a local area network (LAN, Local Area Network), a general wide area network (WAN) and / or a public network (e.g. the Internet). As shown, the network adapter communicates ( 20th ) with the other components of the computer system / server ( 12 ) on the bus ( 18th ). It should be understood that, even if not shown, other hardware and / or software components in connection with the computer system / server ( 12 ) can be used. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and storage systems for data archiving, etc.

It is now going on 2nd Referred to in which a cloud computing environment ( 50 ) is shown. As shown, the cloud computing environment ( 50 ) one or more cloud computing nodes ( 10th ) with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or a mobile phone ( 54A) , a desktop computer ( 54B) , a laptop computer ( 54C) and / or an automotive computer system ( 54N) , to be able to communicate. Knot ( 10th ) can communicate with each other. They can be grouped physically or virtually in one or more networks, in the form of private, community, public or hybrid clouds, as described above, or in a combination thereof (not shown). This enables a cloud computing environment ( 50 ) to offer an infrastructure, platforms and / or software as services for which a cloud consumer does not have to keep resources on a local computing device. It is understood that the in 2nd shown types of computing devices ( 54A - 54N ) are for illustration only and that computing nodes ( 10th ) and the cloud computing environment ( 50 ) can communicate with any type of computer-based device over any type of network and / or in a network addressable connection (e.g. with a web browser).

The following is now going on 3rd Where a group is from the cloud computing environment ( 50 ) provided functional abstraction layers. It goes without saying that the in 3rd The components, layers and functions shown are only intended to be illustrative and embodiments of the invention are not limited to these. As shown, the following layers and corresponding functions are provided: Hardware and software layer ( 60 ), Virtualization layer ( 62 ), Management layer ( 64 ) and workload layer ( 66 ). The hardware and software layer ( 60 ) includes the hardware and software components. Examples of hardware components include mainframes, in one example systems from IBM® zSeries®; Servers based on the RISC (Reduced Instruction Set Computer) architecture, in one example systems from IBM pSeries®; IBM xSeries® systems; IBM BladeCenter® systems; Storage devices; Networks and network components. Examples of the software components include software for the network application server, in one example software for the IBM WebSphere® application server; and database software, in one example the database software IBM DB2®. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere and DB2 are trademarks of International Business Machines Corporation and registered in many jurisdictions worldwide).

A virtualization layer ( 62 ) provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers; virtual memory; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients.

In one example, the management layer ( 64 ) provide the following functions: provision of resources, metering and pricing, user portal, service level management and SLA planning and execution (fulfillment). The functions are described below. The provision of resources enables the dynamic procurement of computing resources and other resources that are used to carry out jobs in the cloud computing environment. Measurement and pricing allows you to track costs when resources are used in the cloud computing environment, as well as billing or billing for the use of those resources. In one example, these resources may include application software licenses. Security enables verification of identity for cloud consumers and jobs as well as protection of data and other resources. A user portal provides access to the cloud computing environment for consumers and system administrators. Service level management enables the allocation and management of cloud computing resources so that the required service levels are met. The planning and fulfillment of the Service Level Agreement (SLA) enables the preparatory arrangement for and procurement of cloud computing resources, for which future requirements are anticipated in accordance with the SLA.

A workload shift ( 66 ) provides examples of functionality for which the cloud computing environment can be used. Examples of the workloads and functions that can be provided via this layer include, but are not limited to: mapping and navigation, software development and lifecycle management, the provision of training in virtual training environments, the processing of data analysis , processing jobs, and processing one or more jobs in response to the hierarchy of virtual resources in the cloud computing environment.

A virtual machine is a software and / or hardware implementation of a computer that executes programs similar to a physical machine. The virtual machine supports an instance of an operating system along with one or more applications to run on an isolated partition on the computer. In one embodiment, the virtual machine enables different operating systems to run simultaneously on the same computer. A physical machine can support multiple virtual machines. Multiple operating systems can run on the same physical machine, and each of the virtual machines can process jobs with different operating systems. Accordingly, the use of one or more virtual machines supports the efficient use of the hardware when processing multiple jobs.

The efficient use of a virtual machine configuration in a cloud computing system is due to the distributed nature of the physical topology of the physical machines, i.e. H. the nodes, is a challenge. In particular, there are difficulties in evaluating and rendering the physical topology underlying the virtual topology of the cloud computing system and in using job processing in response to the physical and virtual topology.

A cloud platform, referred to in this document as CAM, is provided to combine a cluster file system with resource planning. CAM has a three-tier approach to avoid ordering anomalies, the three tiers including the ordering of the data, the ordering of the virtual machine job, and the job ordering. Regarding the data arrangement, data is arranged in the cluster based on offline profiling of the jobs that are most frequently carried out for the data. The order of jobs is influenced by CAM, which selects the best possible physical node (s) for arranging the sets of virtual machines assigned to a job. Otherwise, to further minimize the possibility of a placement anomaly, CAM exposes the job scheduler to hidden computing, storage and network topologies so that it can make the best job assignments. CAM reconciles resource allocation with a variety of other competitive conditions, such as memory usage, CPU load change, and network connection capacity, using an algorithm that is based on a flow network that meets the specified conditions for both initial placement and can be reduced for the readjustment during the migration of the virtual machine and the data at the same time.

4th is a block diagram ( 400 ) that illustrates the overall architecture for using CAM. The physical resources to support the cloud consist of a cluster of physical nodes, with local storage directly associated with each node. As shown, CAM uses a file system that, in one embodiment, is a General Parallel File System-Shared Nothing Cluster ( 410 ) to provide the storage layer for it. GPFS-SNC is designed as a cloud storage platform that supports the timely and resource-efficient provision of virtual machines in the cloud. GPFS-SNC manages the local disks that are directly assigned to a cluster of commercially available physical machines.

As shown, the physical layer ( 420 ) in the form of three commercially available physical machines ( 430 ), ( 440 ) and ( 450 ) illustrates. The number of physical machines shown here is for illustration purposes only. In one embodiment, the number may include a smaller or larger number of commercially available physical machines. Each of these machines has a local plate ( 432 ), ( 442 ) and ( 452 ). The file system ( 410 ) supports a common arrangement of all blocks of a file in one location instead of distributing the file over the network (using striping). This enables an I / O request of a virtual machine to be served locally from the stored location rather than remotely via physical hosts over the network. CAM uses this function to ensure that images (images) of a virtual machine located in a common (co-located) storage location are stored in one storage location and can be accessed efficiently.

The file system ( 410 ) also supports an efficient, pipelined block-level replication scheme that ensures fast distributed recovery (recovery) and high I / O throughput through fast parallel reads. This function is advantageous for CAM in order to achieve efficient recovery in the event of failures. In addition, the file system ( 410 ) an application program interface (API) at the user level, with which physical locations of files can be queried. With this API, CAM retrieves specific block location information to determine the proximity of storage for the arrangement of data and virtual machines.

CAM uses three components to always be informed about the topology. As shown, a server ( 460 ) in communication with the physical layer ( 420 ) provided. The server ( 460 ), which is also referred to in this document as the CAM topology server, provides topology information that is necessary for a scheduler to make the optimal job assignment. From the server ( 460 ) Information presented consists of network and storage topologies and other dynamic information at the node level, such as the CPU load. Furthermore, as shown, each of the physical machines ( 430 ), ( 440 ) and ( 450 ) one agent each ( 434 ), ( 444 ) and ( 454 ) on. Each agent runs on the corresponding physical machine and collects and transmits various pieces of data relating to the respective machine to the server ( 460 ), such as the use of the outgoing and incoming network bandwidth, the I / O usage and the CPU, memory and storage load. The server ( 460 ) consolidates the dynamic information it receives from the agents ( 434 ), ( 444 ) and ( 454 ), and processes it along with topology information about each job running in the cluster. The topology information is derived from the arrangement configuration of the existing virtual machines. A job scheduler ( 470 ) forms an interface with the server ( 460 ) to get exact and current topology information. The scheduler ( 470 ) adjusts the job arrangement accordingly with each observed change in the configuration. Accordingly, the job scheduler uses the storage and physical host resources in CAM.

Network topology information is represented by a distance matrix that encodes a distance between each pair of virtual machines as racks, nodes, or virtual machines. When two virtual machines are placed on the same node, they are connected by a virtual network connection. Due to the fact that the virtual machines use the same node hardware, the virtual network provides a high-speed medium that is significantly faster than links (or connections) between nodes or between racks. Network traffic between virtual machines on the same node does not have to go through a hardware link (or hardware connection). The virtual network device forwards the data traffic in the main memory via highly optimized ring buffers.

Storage topology information is provided as a mapping between each virtual device containing the data set and the virtual machine for which it is local. In a native hardware context, a disk assigned to a node can be accessed directly via a PCI bus. However, in the cloud, the physical blocks associated with a virtual machine image associated with a virtual machine can reside on a different node. Even if a virtual device appears to be directly connected to the virtual machine, the image file that acts as a backup for the device may be located elsewhere on the network and possibly closer than another virtual machine in the cluster the directly assigned virtual machine. The server ( 460 ) queries the location of the physical image via an API and provides the scheduler ( 470 ) Information ready.

Examples of special from the server ( 460 ) APIs provided are described in Table 1 below: Table 1 API description Int get VM distance (string vm1, string vm2) Returns the distance between two virtual machines struct block_location get block location Returns the actual location of (string src, long offset, long length) Blocks back int get vm networking (string vm, struct networkinfo) Returns network usage information of the physical host on which the virtual machine is running int get_vm _diskinfo (string vm, string device, struct diskinfo) Returns disk usage information int get_vm_cpuinfo (string vm, struct cpuinfo) Returns CPU usage information of the physical host on which the virtual machine is running

The get vm distance statement provides the job scheduler ( 470 ) Jobs with hints about network distance between two virtual machines ready. The distance is estimated based on the observed data transfer rates between the virtual machines, and in one embodiment it is expressed in units of bandwidth. The get block _location statement supports getting the specific block location instead of the location of the virtual machine, which ensures locality. The instructions get_vm_networkinfo, get_vm_diskinfo and get_vm_cpuinfo provide for the job scheduler ( 470 ) simplifies querying I / O and CPU access conflict information related to network and disk usage. In one embodiment, the scheduler ( 470 ) use this additional information for smarter decisions, including, in one embodiment, placing I / O intensive jobs on physical hosts that have idle I / O resources.

5 is a flowchart ( 500 ) that illustrates resource allocation and data management, as well as the arrangement of the virtual machine. As described, CAM is a cloud platform with special interfaces and supports the execution of MapReduce jobs. A data set to be processed is initially stored on a general file system (GPFS, General Purpose Filesystem). In particular, the storage and computing resources are not isolated from each other. A MapReduce job is issued by the application, e.g. B. Relevant Java class files that specify a previously uploaded data set for the job, and the number and type of files for the job ( 502 ) virtual machines to be used to be provided. In one embodiment, each virtual machine supports multiple MapReduce job slots (areas for jobs, job slots) depending on the number of virtual CPUs and virtual RAM assigned to each virtual machine. The higher the number of virtual machines assigned to a job, the faster the job will be completed.

CAM determines an optimal arrangement for the set of the requested new virtual machines ( 504 ) by taking into account factors such as, but not limited to, the current workload distribution across the cluster nodes, the distribution of the input data sets required for the job, and the physical locations of the required master images of the virtual machine. The images of virtual machines that are required to boot the virtual machines on the selected nodes are taken from the respective master images ( 506 ) created. In one embodiment, a copy-on-write mechanism provided by the general file system is used to create the virtual machine because it can quickly provide an instance of an image of a virtual machine without a data copy of the Master Images is required. In the next step ( 506 ) the class files of the job are copied to a virtual machine image ( 508 ) copied. In one embodiment, the copying takes place in step ( 508 ) instead by mounting the image as a loop back filesystem. Finally, the data images are assigned to the virtual machine and the corresponding files are stored in the virtual machine ( 510 ) mounted, so that jobs can access the data contained therein.

It is now going on 4th Referred to in which each physical machine ( 430 ), ( 440 ) and ( 450 ) each with local plates ( 432 ), ( 442 ) and ( 452 ) Is provided. A distributed file system ( 410 ), also referred to in this document as GPFS-SNC, is on the physical machines ( 430 ), ( 440 ) and ( 450 ) Installed. The virtual machine images ( 436 ), ( 446 ) and ( 456 ) are stored in the distributed file system ( 410 ) saved. In one embodiment, a cloud manager assigns resources to jobs and manages the data arrangement and arrangement of the virtual machine.

The arrangement of the data and virtual machines are aspects that need to be considered in terms of cost. In particular, the arrangement takes into account that the proximity of the virtual machines is ensured, that hotspots are avoided and that the use of physical memory is balanced taking into account the different job types. The proximity of the virtual machine expresses the cost of data access and how data should be arranged with respect to the associated virtual machines to minimize access times. The hotspots factor expresses the expected load for a machine and identifies machines that do not have sufficient computing resources to support the assigned virtual machines. In order to avoid a hotspot, the data must be arranged on the machine with the lowest load. This can be determined by measuring the computing resources currently assigned to the machine and adding the expected assignment request of the virtual machines that would work with the data to be arranged on the machine. Finally, memory usage is a percentage of the total physical machine memory space used.

The following table, Table 2, shows a table that illustrates the meaning of the three factors listed above that affect performance under different workloads. Table 2 Job type Proximity of the virtual machine Hotspot factor Memory usage Allocation and reduction intensive Yes Yes Yes Assignment intensive No Yes Yes Reduction-intensive No No Yes

In the case of both assignment (map) and reduction-intensive (reduce) workloads, the associated data should be arranged close to one another and on the machine with the lowest load. In the case of assignment-intensive workloads, the data should be arranged on the machine with the lowest load, but they do not necessarily have to be arranged close to one another, since moderate traffic (light shuffle traffic) occurs with such a workload. With reduction-intensive workloads, only the memory usage is the machine on which the virtual machine is to be placed. For all types of workloads, it is desirable to distribute the data evenly across the racks to minimize the need to rearrange the data over time to help migrate virtual machines.

The factors given in Table 2 are used when creating a flow chart with the lowest cost (min-cost flow graph) in which the factors are encoded. 6 represents a flow graph ( 600 ) that illustrates a sample network topology for the ordering of the data. In particular, the sample network topology consists of six physical nodes ( p ₁ , p ₂ , p ₃ , p ₄ , p ₅ , p ₆ ), as ( 610 ), ( 612 ), ( 614 ), ( 616 ), ( 618 ) and ( 620 ) be identified. The physical nodes are in three racks ( r ₁ , r ₂ , r ₃ ), as ( 630 ), ( 632 ) and ( 634 ) are organized. A master rack r ₄ , like a switch that as ( 650 ) connects the racks. In one embodiment, the topology shown here can support any topology in which network traffic can be estimated. The unit of data arrangement is an image of a virtual machine to ensure that a complete image is available in one location.

Based on the flow graph of 6 is in 7 a second flow graph ( 700 ) for the exemplary arrangement of data. Two data elements, d ₁ , ( 702 ) and d ₂ , ( 704 ), with requirements for five and two images of virtual machines, respectively, are transferred to the cloud, e.g. B. a shared pool of resources. The number of images of the virtual machine requested by a data element is referred to as the input (supply) of the data element for the flow graph. A node without a successor (sink node), S ( 790 ) is added to the graph to support virtual machines. The number of virtual machines that a node can manage without a successor is assigned as a demand value. In the example shown here, the node has no successor ( 790 ) has a set point of 7 and it is the only place that can get all the rivers. There are two parameters associated with each edge of the flow graph, including the capacity of the edge and the cost of a flow through that edge. The data nodes d ₁ ( 702 ) and d ₂ ( 704 ) have outgoing links (or line connections) to each rack, whereby the proximity to the virtual machine is the cost. As shown, the data node has d ₁ ( 702 ) via a link (or line connection) ( 712 ) that too rack r ₁ ( 722 ) runs, the data node d ₁ ( 702 ) has a link ( 714 ) that too rack r ₂ ( 724 ) runs; the data node d ₁ ( 702 ) has a link ( 716 ) that too rack r ₃ ( 726 ) runs; the data node d ₁ ( 702 ) has a link ( 718 ) that too rack r ₄ ( 728 ) runs; the data node d ₂ ( 704 ) has a link ( 732 ) that too rack r ₁ ( 722 ) runs; the data node d ₂ ( 704 ) has a ( 734 ) that too rack r ₂ ( 724 ) runs; the data node d ₂ ( 704 ) has a link ( 736 ) that too rack r ₃ ( 726 ) runs and the data node d ₂ ( 704 ) has a link ( 738 ) that too rack r ₄ ( 728 ) runs. Six physical nodes are shown here, in detail these are p ₁ ( 740 ), p ₂ ( 742 ), p ₃ ( 744 ), p ₄ ( 746 ), p ₅ ( 748 ) and p ₆ ( 750 ). The hotspot factor is in the links ( 750 ), ( 752 ), ( 754 ), ( 756 ), ( 758 ) and ( 760 ) from the racks to each physical node within the scope thereof. Even if r ₄ ( 728 ) acts as a switch between the racks, it is shown as being directly connected to all physical nodes to ensure that the machine with the lowest load can be selected for assignment-intensive jobs without being restricted by the network topology.

All physical nodes p ₁ ( 740 ), p ₂ ( 742 ), p ₃ ( 744 ), p ₄ ( 746 ), p ₅ ( 748 ) and p ₆ ( 750 ) are with the node without a successor ( 790 ) connected, the storage usage forming the link costs. There is no direct link from the data element node d _j to the associated physical host p _i . The reason for this is to support scaling up the system. The following table, Table 3, contains the values assigned to the flow graph for the arrangement of the data. Table 3 Data set d _j Rack r _k Physical host Knot without a successor input Σ (Nd _j ) 0 0 - Σ (Nd _j ) Incoming link from N / Nd _j Rack r _k Physical host Outbound link (capacity, cost) To the rack (Nd _j , a _jk ) To the physical host (Cap _i , β _i ) To the knot without successor (Capi, Yi) N / A

wobei distance_max die Netzwerkdistanz zwischen zwei beliebigen Knoten im Rack r_k ist. As shown, Nd _{j is} the number of virtual machine images requested by data set d _j , a _jk to detect the proximity of the virtual machines. The cost, a _jk , of the outbound link from the data set d _j to the physical host p _i on which the data is placed on the rack r _{k is} estimated conservatively from the traffic in the shuffle phase as follows: $$a_{jk}=si\mathrm{e.g.}{e}_{intermediate-data}*\underset{\underset{\_}{num}\underset{\_}{{}_{Reducer}}}{{\underset{\_}{num}}_{\underset{\_}{Reducer}}{}^{\underset{\_}{-1}}}*distanc{e}_{max}$$

where distance _{max is} the network distance between any two nodes in the rack r _k .

wobei load_curr und load_min jeweils die aktuelle Last und die minimale aktuelle Last darstellen, und a ein Parameter ist, der als Möglichkeit („Knopf“, Knob) der Feineinstellung der Gewichtung des Hotspot-Faktors bezüglich anderer Kosten dient. Die erwartete Last, load_exp, lautet wie folgt: $$loa{d}_{exp}={\displaystyle {\sum }_j(p_j/\left(1-p_j\right)*CRes}\left(d_j\right)$$

wobei p_j = λ_j/µ_j, und λj die Anzahl der d_j zugeordneten Jobs darstellt, die in einem bestimmten Zeitintervall eingehen, und µ_j für jede virtuelle Maschine die mittlere Zeit zur Verarbeitung eines Blocks angegeben.The hotspot factor is recorded with β _i for the physical node p _i and is estimated as follows on the basis of the current and expected load: $${\beta }_i=a*\left(lOa{d}_{exp}+lOa{d}_{curr}-lOa{d}_{min}\right)$$

where load _curr and load _min each represent the current load and the minimum current load, and a is a parameter which serves as a possibility (“knob”, knob) for fine-tuning the weighting of the hotspot factor with respect to other costs. The expected load, load _exp , is as follows: $$lOa{d}_{exp}={\displaystyle {\sum }_j(p_j/\left(1-p_j\right)*\mathrm{C.}Res}\left(d_j\right)$$

where p _j = λ _j / µ _j , and λj represents the number of jobs assigned to d _j that arrive in a specific time interval, and µ _j indicates the average time for processing a block for each virtual machine.

wobei b ein Parameter ist, um die Gewichtung der Speichernutzung bezüglich anderer Faktoren feinabzustimmen. Schließlich wird für die Kapazität die folgende Formel als Schätzung der Kapazität für jeden physischen Host verwendet: $$Ca{p}_i=freespac{e}_{pi}/siz{e}_{VMImg}$$

The memory usage of a physical node p _i is recorded by y _i , which is determined by the current memory usage in comparison to the minimum memory usage of all p _i s as follows: $${\gamma }_i=b*\left(stOraGeUtili\mathrm{e.g.}atiOn{p}_i-stOraGeUtili\mathrm{e.g.}atiO{n}_{min}\right)$$

where b is a parameter to fine-tune memory usage weighting with respect to other factors. Finally, for capacity, the following formula is used to estimate capacity for each physical host: $$\mathrm{C.}a{p}_i=freespac{e}_{pi}/si\mathrm{e.g.}{e}_{VM\mathrm{I.}mG}$$

In order that the correlation between the arrangements of the virtual machine images for a data request can be recorded in the graph, a split factor parameter is provided to indicate whether the flows from a node can be divided over different links. In one embodiment, the value of this parameter is defined as true or false. For example, if the split factor for all links from d ₁ and d ₂ If "true" results, all flows of data nodes run through one of r ₁ , r ₂ , r ₃ , r ₄ , but are not split between the racks. As soon as a new data upload request is received, the cloud server updates the graph and calculates a global optimal solution for the newly updated data based on the calculated solution. Accordingly, the scheduler is updated periodically based on the new solution and can take into account varying loads.

The goal of arranging the virtual machine is to maximize the locality of the global data and job throughput. Our model takes into account both the migration of virtual machines and the delayed planning of a job as part of the optimal solution. The delay in a job is used to examine better ways of localizing data that may arise in the near future, while minimizing the time spent waiting. By migrating a virtual machine belonging to a job, the scheduler can create space for other suitable jobs or determine better options with regard to the storage location.

8th represents a flow graph ( 800 ) for the arrangement of virtual machines. Each job v _j ( 802 ) and ( 804 ) is passed to the system at the source node with the number of requested virtual machines, N _vj , as an input value. The aim of the allocation function (allocator) of the virtual machines is to keep the job as not planned (or not planned for execution, unscheduled, hereinafter only: not planned). B. each request 0 or N _vj to assign virtual machines. There is a node S without a successor ( 890 ) with a setpoint equal to minus the sum of the input value. Each job's request acts as a flow, either through the rack nodes ( 810 ), ( 812 ), ( 814 ) and ( 816 ) or through the unplanned nodes ( 820 ), ( 822 ) and ( 824 ) and finally to the node without a successor ( 890 ) runs. If a job is not scheduled, none of its virtual machines are assigned. Otherwise, the flow passes through the physical nodes ( 830 ), ( 832 ), ( 834 ), ( 836 ), ( 838 ) and ( 840 ). Based on the lowest cost solution, an allocation scheme with the lowest cost can be derived. When the virtual machines are mapped to the top-level rack, it indicates that the virtual machines can be mapped to any set of nodes in the virtual machines on the rack.

The job type information is modeled as the cost of the edge from each job to the rack nodes in the flow-based graph. The higher level rack is more expensive than the lower level rack in terms of reduced traffic. In one embodiment, the cost of the top-level rack is estimated by arranging the virtual machines in the event of the worst allocation and reduced traffic. The cost of the edges of the unplanned nodes is set to increase over time so that delayed jobs are assigned faster than recently posted jobs. These costs also control when a job stops and waits for better locality, and therefore offer a setting option to optimize the compromise between locality of data and latency. The unplanned aggregate nodes control how many virtual machines can remain in an unplanned state to control the use of system resources and the compromise on the locality of the data. The cost of the edges to the executing node set increases over time and takes into account the progress of the job execution.

wobei M die Gesamtzahl der virtuellen Maschinen ist, die im Cluster unterstützt werden können, und M_idle die im Cluster zulässige Anzahl der im Leerlauf befindlichen virtuellen Maschinenslots angibt. 9 represents a table (900) with values assigned to the flow graph for the arrangement of the virtual machine and the node categorization. Different nodes in the graph are categorized as different types, as shown in the table. A preferred knot set is a set (pr _j ) ( 910 ) from graph nodes that point to a set of physical nodes p _i that have a data set associated with the job v _j stored therein. An edge from a preferred node to a physical node p _i has the cost 0 and the capacity of the number of virtual machine disk images stored on the physical nodes p _i . An executed node set (ru _j ) ( 920 ) is a set of dynamically added nodes that point to physical nodes (p _i s) that currently host the job's virtual machines (v _j ). An edge from ru _j to p _i has the cost 0 and the capacity in the number of virtual machines running on the physical nodes p _i . An unplanned set of knots, u _j , ( 930 ) is a node set that provides information about currently unplanned jobs. The unplanned node set, u _j , has an outgoing edge with the capacity N _vj and the code 0 to an unplanned aggregator. An unplanned aggregate node, u, ( 940 ) has an outgoing edge with cost 0 to the node without successor, whereby the capacity is defined as: $$N_{unscHed}={\displaystyle \sum \left(N_{vj}\right)-M+M_{idle}}$$

where M is the total number of virtual machines that can be supported in the cluster, and M _{idle is} the number of virtual machine slots that are allowed to be _idle in the cluster.

The rack knot set, r _k , ( 950 ) represents a rack in the topology of the cluster. It has outgoing links with the cost 0 to the subordinate racks or, if it is at the lowest level, to physical nodes. The links have the capacity N _rk , this is the total number of virtual machine slots that can be served by the underlying nodes. The physical host node set, p _i , ( 960 ) has an outgoing link to the node without a successor, where the capacity is the number of virtual machines that can exist on the physical host N _vm and the cost 0. The graph has a node without a successor ( 970 ) with a setpoint, which is represented as Σ (N _vj ). The job node set, v _j , ( 980 ) represents each job node v _j with the input N _vj . It has several outgoing edges that correspond to the possible assignment decisions of the virtual machines for the job set.

The edges include a rack knot set, a preferred knot set, an executed knot set, and an unplanned knot set. The rack node set, r _k , has an edge to r _k that indicates that r _k can accommodate v _j . The cost of the edge is p _j , which is calculated by the mapping and which reduces traffic costs. If the capacity of the edge is greater than N _vj , it indicates that v _j's virtual machines are assigned to some preferred nodes in the rack. The preferred node set, p _rj , has an edge from job v _j to the preferred node set for the entire job, p _rj , it has a capacity N _vj and cost θ. The costs are only estimated on the basis of the data traffic in the reduced phase, since in this case an assigned data traffic of 0 is assumed. The node set executed, r _uj , has a link of job v _j , a capacity N _vj and cost ϕ = c * T, where T is the time that the job was executed on the set of machines and c is a constant, with which the costs are adjusted relative to the other costs. The unplanned node set, u _j , has an edge to the node u _j not planned in the entire job with the capacity N _vj and the costs ε _j , which correspond to the penalty that the job v _{j was} not planned. ε _j = d * T, where T is the time that the job v _{j is} not scheduled and d is a constant with which the costs are adjusted relative to other costs. The split factor for this link is marked as true, which means that the assignments of all virtual machines are either fulfilled or are postponed until the next round.

Based on output of a flow solution with minimal costs, the assignment of the assignment of the virtual machines can be obtained from the graph by determining where the associated flow leads to for each request from v _{j of} a virtual machine. The flow to an unplanned node indicates that the virtual machine request is skipped in the current round. If the flow leads to a preferred node set, the virtual machine request for that node set is scheduled. If the flow goes to a rack node, it indicates that the job's virtual machines are mapped to any host in that rack.

The number of flows specified on a physical host through rack nodes or a set of preferred nodes is less than the number of available virtual machines for each physical host. This is ensured by the specified link capacity from the physical host to the node without successor. Accordingly, all virtual machine requests that are assigned are mapped to a corresponding physical host.

10th provides a flow chart ( 1000 ) that illustrates a process of evaluating and using the topology of the physical and virtual machines in a shared pool of resources. The status of the virtual machines in the shared pool is collected ( 1002 ). It can be seen that one or more virtual machines can be mapped to a single physical host, thereby supporting the virtualization of an underlying physical machine. Based on the data collected, associated topology information is recorded and communicated to a server at a root node of the hierarchical organization of the physical and virtual machines ( 1004 ).

Each virtual machine is deployed with a built-in agent, and each physical machine to which the virtual machines are assigned is deployed with a built-in monitor. A server machine is provided in communication with each of the physical machines and its function is to periodically collect information from the integrated monitors of the underlying physical machines. The function of the integrated monitors is to collect local topology, disk and network information. In one embodiment, the integrated monitors are in the form of software that runs on the physical machine to collect status data of the associated virtual machines. The integrated agent of each virtual machine also communicates the specific topology and system usage information to the server machine. Accordingly, each virtual and physical machine contains integrated tools to collect usage information belonging to the topology and to transmit the collected information to the server machine.

After the step of collecting the topological data in step ( 1004 ) the collected data is organized in a single location ( 1006 ). In one embodiment, the single location may be a root node that represents a physical server that is in communication with each of the virtual machines and their associated physical machine. As in step ( 1004 ), the data communicated by the integrated agents of the virtual machines contain the specific topology and system usage information. The step ( 1004 ) recorded data have a storage topology, which is based on a virtual topology of the shared pool of resources, together with the associated resource usage information. A job is assigned to a selected virtual machine in the shared pool in the knowledge of the recorded data, the organized ones being assigned Storage topology information can be used ( 1008 ). In one embodiment, the job assignment in step ( 1008 ) designed to support the efficient performance of the I / O belonging to the job. Accordingly, jobs can be intelligently assigned to a selected virtual machine through the process of collecting and organizing the local topology information.

The job assignment in step ( 1008 ) can take place on one virtual machine or on several virtual machines. The job can also be a read job or a write job. In both scenarios, a virtual topological distance is returned in response to the job assignment ( 1010 ). The virtual topological distance can be a distance between two or more virtual machines if the job is assigned to multiple machines, or the virtual topological distance can be between a virtual machine and a data block if the job is a single virtual machine for a read or Write job supported. In the next step ( 1010 ) it is determined whether the returned topological distance between at least two virtual machines ( 1012 ) consists. If the answer to the determination in step ( 1012 ) positive, a shared memory channel is created for data communication between virtual machines between two virtual machines that exist locally on the same physical machine ( 1014 ). Creating the shared memory channel supports efficient data transfer between the two virtual machines. In particular, a memory copy can be used for communication between the virtual machines, thereby avoiding communication via a virtual network stack ( 1016 ). Conversely, if the answer to the determination in step ( 1012 ) subsequently the virtual network stack is used for communication between the virtual machines performing the assigned job ( 1018 ) support. Accordingly, the physical proximity of the virtual machines can contribute to an efficient transfer of communication between virtual machines.

A physical machine supports the virtual machine and data blocks support the job. The location of the data blocks in the shared pool affects the assignment of the job to the physical machine and the associated virtual machine (s). In particular, efficient use of resources in the shared pool ensures the physical proximity of the physical machine to the data blocks. In one embodiment, the job is mapped to a physical machine in the same physical data center as the dependent data blocks. Accordingly, part of the job assignment process consists of step ( 1008 ) ensuring that a physical location of the data blocks in the shared pool supports the job.

In addition to the location of the blocks, the bandwidth of the underlying physical machine to support the job is critical. The step of using the storage topology information in step ( 1008 ) may require one or more additional steps. 11 is a flowchart ( 1100 ), which illustrates the next steps to support the aspect of storage topology usage of the shared pool of resources. As described above, in response to the storage topology, a virtual machine is specified that receives the job for processing ( 1 102 ). Before concrete job processing, usage information of the physical machine that is available locally for the virtual machine is determined ( 1104 ). The usage information includes, but is not limited to, the processing unit and network usage information. It determines whether the underlying physical machine has the bandwidth and ability to support the job ( 1106 ). If the answer to the determination in step ( 1106 ) is returned to the selection and assignment of another virtual machine in the shared pool. In contrast, a positive answer for the determination in step ( 1106 ) an indication that the selected virtual machine has both sufficient bandwidth to support the job and a close topological distance to the physical location of the data block (s) supporting the job ( 1108 ). In one embodiment, the close topological distance includes, but is not limited to, data that is in the same data center as the virtual machine. Accordingly, the aspect of memory topology usage includes evaluating the operation of the machine along with the location of the data supporting the job.

As in 10-11 a method is provided in order to use the topological organization of the machines together with the storage location of the data blocks supporting the job for an intelligent assignment of a job. The job is assigned to a machine that has been evaluated to support efficient processing. 12 presents a block diagram ( 1200 ), which illustrates tools integrated into a computer system to support a technology that are used for the evaluation of resource use for the assignment of a job in a shared pool of resources. In particular, a common pool of configurable computer resources with a first data center ( 1210 ), a second data center ( 1230 ) and a third data center ( 1250 ). Although three data centers are shown in the example, the invention should not be limited to this number of data centers Computer system may be limited. Each of the data centers represents a computing resource. Accordingly, one or more data centers can be used to support the efficient and intelligent allocation of jobs in terms of resource use and proximity to the data blocks supporting the job (s).

Each of the data centers in the system is provided with at least one server that is in communication with the data store. In particular, the first data center ( 1210 ) with a first server ( 1220 ) with a processing unit ( 1222 ) provided via a bus ( 1226 ) with a memory ( 1224 ) is in communication and in communication with a data storage device ( 1228 ) stands; the second data center ( 1230 ) with a second server ( 1240 ) which a processing unit ( 1242 ) via a bus ( 1246 ) in communication with a memory ( 1244 ) and is in communication with a second local storage ( 1248 ) stands; and the third data center ( 1250 ) is connected to a third server ( 1260 ) with a processing unit ( 1262 ) provided via a bus ( 1266 ) with a memory ( 1264 ) is in communication and with a third local storage ( 1268 ) is in communication. The first server ( 1022 ) everything in this document is also referred to as a physical host. Communication between the data centers is via one or more network connections ( 1205 ) supports.

The second server ( 1240 ) contains two virtual machines ( 1232 ) and ( 1236 ). The first virtual machine ( 1232 ) has an integrated agent ( 1232a) , and the second virtual machine ( 1236 ) has an integrated agent ( 1236a) . In addition, the second server ( 1240 ) a monitor ( 1234 ) to communicate with the first and second virtual machines, respectively ( 1232 ) and ( 1236 ) to simplify. The third server ( 1260 ) contains two virtual machines ( 1252 ) and ( 1256 ). The first virtual machine ( 1252 ) has an integrated agent ( 1252a) , and the second virtual machine ( 1256 ) has an integrated agent ( 1256a) . In addition, the third server ( 1260 ) a monitor ( 1254 ) to communicate with the first and second virtual machines, respectively ( 1252 ) and ( 1256 ) to simplify. Although only two virtual machines ( 1232 ) and ( 1236 ) than in communication with the second server ( 1240 ) are shown and only two virtual machines ( 1252 ) and ( 1256 ) than in communication with the third server ( 1260 ) are shown, the invention should not be limited to this number, since this number is used only for the purpose of illustration. The number of virtual machines each in communication with the second and third servers ( 1240 ) and ( 1260 ) can be increased or decreased in each case.

As shown in this document, each supports the second and third server ( 1240 ) and ( 1260 ) two virtual machines each ( 1232 ), ( 1236 ) and ( 1252 ), ( 1256 ). The display ( 1234 ) of the server ( 1230 ) collects status data from each of the virtual machines ( 1232 ) and ( 1236 ). The display ( 1234 ) communicates with integrated agents ( 1232a) and ( 1236a ) to see the virtual machine status of the virtual machines ( 1232 ) and ( 1236 ) to raise. Likewise, the monitor ( 1254 ) Status data from each of the virtual machines ( 1252 ) and ( 1256 ) and especially from the integrated agents ( 1252a) and ( 1256a ).

The first server ( 1220 ) with a functional unit ( 1270 ), which has one or more tools to support the intelligent assignment of one or more jobs in the shared pool of resources. The functional unit ( 1270 ) for the first data center ( 1210 ) locally and in particular in communication with the storage ( 1224 ). In one embodiment, the functional unit ( 1270 ) be local to one of the data centers in the shared pool of resources. The in the functional unit ( 1270 ) built-in tools include, but are not limited to, a director ( 1272 ), a topology manager ( 1274 ), a hook manager ( 1276 ), a storage topology manager ( 1278 ), a resource usage manager ( 1280 ) and an application manager ( 1282 ).

The director ( 1272 ) is provided in the common pool to periodically with the monitors ( 1234 ) and ( 1254 ) to organize and maintain a storage topology that underlies a virtual topology of the shared pool of resources along with related resource usage information in a single location. In particular, the communication of the Director ( 1272 ) with the monitors ( 1234 ) and ( 1254 ) Collecting and organizing the topology of the shared pool of resources. By organizing and understanding the topology data, the director ( 1272 ) use the resource usage information to intelligently assign a job to one or more of the shared resources in the pool, in a manner that supports the efficient performance of the job's I / O. Accordingly, the director ( 1272 ) the topology and uses it to support the efficient processing of read and write jobs in the shared pool of resources.

As described above, various managers are employed to support the functionality of the director ( 1272 ) provided. The topology manager ( 1274 ) with the Director ( 1272 ) is in communication, gives the director ( 1272 ) virtual topological distance data back. The virtual topological distance data includes, but is not limited to, a distance between two virtual machines and a distance between a virtual machine and a data block.

For example, two virtual machines that are communicating with the same server are considered to be in relatively close proximity to each other. However, a second virtual machine that is in communication with a second server and a third virtual machine that is in communication with a third server are compared to the two virtual machines that are in communication with the same server considered distant. The storage topology manager ( 1278 ) with the Director ( 1272 ) in communication, returns a physical location of one or more data blocks to support a job in the shared pool of resources. In one embodiment, the storage topology manager ( 1278 ) the physical location of the data blocks to the Director ( 1272 ) back, allowing the director to intelligently assign a job to a virtual machine in response to the location of the dependent data block (s). Accordingly, the topology manager addresses ( 1274 ) in its function distances in the hierarchy with regard to efficient job processing, and the storage topology manager ( 1278 ) addresses the storage location of the data block supporting the job.

Three more managers are also provided, including a resource usage manager ( 1280 ), an application manager ( 1282 ) and a hook manager ( 1274 ). The resource usage manager ( 1280 ) manages the use of one or more physical or virtual resources. Each resource has its own limitations. The resource usage manager ( 1280 ) gives the processing unit usage information and network usage information related to the underlying physical and virtual machines to the director ( 1272 ) back. The application manager ( 1282 ) in communication with the resource use manager ( 1280 ) assigns the job to a virtual machine in response to the resource usage information. In particular, the application manager ( 1282 ) ensures that the assignment of a job to a machine in the topology ensures that the machine has sufficient bandwidth to support the job as well as a sufficient close topological distance to the data blocks supporting the job. Accordingly, the resource usage manager ( 1280 ) and the application manager ( 682 ) both usage and bandwidth are taken into account.

In addition to the managers described in detail above, the hook manager ( 674 ) to simplify communication between virtual machines. In particular, the hook manager ( 674 ) who is in communication with the Director ( 672 ) is provided to create a shared memory channel for communication between virtual machines. The shared memory channel simplifies communication between two virtual machines that are on the same physical machine by allowing data transfer between two such virtual machines to take place in the same memory stack, e.g. B. via the memory channel. Accordingly, the hook manager ( 674 ) shared memory channel created the efficient data communication in the hierarchical structure of the shared pool of resources.

As determined above, the Director ( 1272 ), the topology manager ( 1274 ), the hook manager ( 1276 ), the storage topology manager ( 1278 ), the resource use manager ( 1280 ) and the application manager ( 1282 ) locally in the first data center ( 1210 ) than in the functional unit ( 1270 ) of the server ( 1220 ) shown. In one embodiment, although the functional unit ( 1270 ) and the assigned director and managers as hardware tools external to the storage ( 1224 ) of the server ( 1220 ) of the first data center ( 1210 ), they can be implemented as a combination of hardware and software, or they can be located locally in the second data center ( 1230 ) or in the third data center ( 1250 ) are in the shared pool of resources. Likewise, in one embodiment, the director and the manager can be combined in a single functional element that contains the functionality of the individual elements. As shown in this document, each of Director and the managers is represented locally in a data center. In one embodiment, however, they can be distributed together or individually over the common pool of configurable computer resources and act as a unit to evaluate the topology of the processing units and data storage in the common pool, and to process one or more jobs in response to the hierarchy. Accordingly, the managers can be implemented as software tools, hardware tools or a combination of software and hardware tools.

Those skilled in the art will understand that aspects of the present invention can be implemented as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of a hardware-only embodiment, a software-only embodiment (including firmware, resident software, microcode, etc.) or an embodiment that combines software and hardware aspects, all of which are generally referred to herein as "circuit", "module.""Or" System "can be called. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable media that contain computer readable program code.

Any combination of one or more computer readable media can be used. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. For example, but not limited to, a computer readable storage medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, device, or any suitable combination of the foregoing. Other special examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more cables, a portable computer diskette, a hard disk, random access memory (RAM or random access memory), read-only memory (ROM or read- only memory), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM or compact disc read-only memory), an optical storage device, a magnetic storage device or any suitable Combination of the previous. In the context of this document, a computer readable storage medium may be any tangible medium that can store a program for use by or in connection with an instruction execution system, instruction execution device, or instruction execution device.

The computer-readable signal media can include a forwarded data signal with computer-readable program code therein, for example in the baseband or as part of a carrier wave. Such a relayed signal may take any of a variety of forms, including, but not limited to, an electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, forward, or transport a program for use or in conjunction with an instruction execution system, instruction execution device, or instruction execution device.

Program code contained on a computer readable medium can be transmitted using an appropriate medium, including, but not limited to, radio, cable, fiber, RF, or any suitable combination of the foregoing.

Computer program code for performing the operations for aspects of the present invention may be written in any combination of one or more languages, including object-oriented programming languages such as Java, Smalltalk, C ++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external one Computers are manufactured (for example over the Internet with an Internet service provider).

Aspects of the present invention are described in this document with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to the embodiments of the invention. It is understood that each block of the flowchart illustrations and / or block diagrams and combinations of the blocks in the flowchart illustrations and / or block diagrams can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a standard computer, special purpose computer or other programmable data processing device in order to generate a machine, so that the instructions which are executed via the processor of the computer or another programmable data processing device, means for implementing the flowchart and / or or create block diagram block or blocks of specified functions / operations.

These computer program instructions can also be stored in a computer readable medium that can instruct a computer, other programmable computing device, or other device to function in a particular manner so that the instructions stored on the computer readable medium create a product that includes instructions that Implement the function / process specified in the flowchart and / or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable computing device, or other devices to cause a series of operational steps to be performed on the computer, or on other programmable devices, or other devices, to create a computer-implemented process such that those on the computer Instructions executed on the computer or the programmable device provide the processes for implementing the functions / operations specified in the flowchart and / or block diagram block or blocks.

It is now going on 13 Which is a block diagram ( 1300 ) representing a system for implementing an embodiment of the present invention. The computer system contains one or more processors, such as a processor ( 1302 ). The processor ( 1302 ) is with a communication infrastructure ( 1304 ) connected (e.g. a communication bus, a cross-over connection or a network). The computer system can have a display interface ( 1306 ) that contain graphics, text and other data from the communication infrastructure ( 1304 ) (or from a frame buffer, not shown) for display on a display unit ( 1308 ) forwards. The computer system can also have a main memory ( 1310 ), preferably a random access memory (RAM), and it can also be a secondary memory ( 1312 ) contain. The secondary storage ( 1312 ), for example, a hard drive ( 1314 ) and / or a drive for removable storage media ( 1316 ) which, for example, represents a floppy disk drive, a magnetic tape drive or a drive for optical storage media. The removable storage drive ( 1316 ) reads from and / or writes to a removable storage media unit ( 1318 ) in a manner well known to those skilled in the art. The removable storage device ( 1318 ) represents, for example, a floppy disk, a CD, a magnetic tape or an optical disk, etc., which is removed from the removable storage drive ( 1316 ) is read and written on it. Please note that the removable storage unit ( 1318 ) contains a computer-readable medium on which the computer software and / or the data are stored.

In alternative embodiments, the second memory ( 1312 ) contain other similar means so that computer programs or other instructions can be loaded onto the computer system. A removable storage unit ( 1320 ) and an interface ( 1322 ) belong. Examples of these means include a program package and a package interface (as found in video game machines), a removable memory chip (such as an EPROM or PROM) and an associated socket and other removable memory units ( 1320 ) and interfaces ( 1322 ) with which software and data from the removable storage device ( 1320 ) can be transferred to the computer system.

The computer system can also have a communication interface ( 1324 ) contain. The communication interface ( 1324 ) enables software and data to be transferred between the computer system and external devices. To the examples of communication interfaces ( 1324 ) can include a modem, a network interface (such as an Ethernet card), a data transmission port or a PCMCIA slot and card, etc. Via the communication interface ( 1324 ) The transmitted software and data are in the form of signals, which can be, for example, electronic, electromagnetic, optical or other signals that are transmitted by the communication interface 1324 ) can be received. These signals are sent to the communication interface ( 1324 ) via a communication path (ie channel) ( 1326 ) provided. This communication path ( 1326 ) transmits signals and can be implemented using wiring or cables, fiber optic cables, a telephone line, a radiotelephone connection, a radio frequency (RF) connection and / or other communication channels.

Throughout this document, the terms "computer program medium", "medium usable by a computer" and "computer readable medium" are used to refer generally to media such as the main memory ( 1310 ) and secondary storage ( 1312 ), the drive for removable storage media ( 1316 ) and one in the hard drive ( 1314 ) installed hard disk.

Computer programs (also called control logic of the computer) are stored in the main memory ( 1310 ) and / or in secondary storage ( 1312 ) saved. Computer programs can also be operated via a communication interface ( 1324 ) are received. These computer programs, when executed, enable the computer system to carry out the features of the present invention as described in this document. In particular, the computer programs enable the processor to execute ( 1302 ) to perform the features of the computer system. Accordingly, these computer programs represent control units of the computer system.

The flowchart and block diagrams in the figures (FIG.) Illustrate the architecture, functionality, and operation of the possible implementations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or section of code that has one or more executable instructions for implementing the specified logic function (s). It should also be noted that in some alternative implementations, the functions specified in the block are not shown in the functions shown in FIG. specified order can occur. For example, depending on the functionality included, two consecutive blocks can be executed substantially simultaneously, or blocks can sometimes be executed in the reverse order. It should also be noted that each block in the block diagram and / or flowchart illustrations and the combinations of the blocks in the block diagram and / or flowchart illustrations are implemented by special hardware-based systems that perform the specified functions or operations, or combinations of special hardware and computer instructions can.

The terminology used here is for the purpose of describing only certain embodiments and is not intended to limit the invention. As used here, the singular forms "a", "a" and "the", "the" "that" are intended to include the plural forms, unless the context clearly indicates otherwise. It is understood that the term "comprises" or "having", when used in this description, is used to indicate the presence of specified features, integers, steps, operations, elements and / or components, but not the presence or excludes adding one or more characteristics, one or more integers, one or more steps, one or more operations, one or more elements, one or more components and / or groups thereof.

The associated structures, materials, processes, and correspondences of all means or steps plus functional elements in the following claims are intended to include any structure, material, or process to function in combination with other claimed elements, as particularly claimed to execute. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed form of the invention. Many changes and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application and to enable others of ordinary skill in the art to understand the invention in various embodiments with various changes as appropriate for the particular use contemplated . Accordingly, the improved cloud computing model supports flexibility in application processing and disaster recovery, including but not limited to, support for separating the location of the data from the location of the application and the Choosing an appropriate recovery location.

• Anordnung von Daten im Cluster, das die am häufigsten mit den Daten ausgeführten Jobs ausführt, Auswählen des für die physischen Knoten angemessenen Modus, um das einem Job zugewiesenen Set virtueller Maschinen anzuordnen, und Darstellen der Computing-, Speicher- und Netzwerktopologien für den Scheduler. CAM verwendet einen flussnetzwerkbasierten Algorithmus, der die Ressourcenzuweisung mit einer Vielzahl anderer im Wettbewerb stehenden Bedingungen in Einklang bringen kann, wie die Speichernutzung, Änderungen der Prozessorlast und die Netzwerkverbindungskapazitäten.

As described in this document, a platform is provided with a resource scheduler to counteract performance degradation of MapReduce jobs when executed in the cloud environment. MapReduce, which captures the cloud, uses a three-tier approach to avoid ordering anomalies due to inefficient resource allocation, including:

• Arrange data in the cluster that performs the jobs most frequently performed on the data, select the mode appropriate for the physical nodes to arrange the set of virtual machines assigned to a job, and present the computing, storage, and network topologies for the scheduler. CAM uses a flow network-based algorithm that can reconcile resource allocation with a variety of other competitive conditions such as memory usage, changes in processor load, and network connectivity.

Alternative embodiment

It is to be emphasized that although specific embodiments of the invention have been described here for purposes of illustration, various modifications can be made without departing from the spirit and scope of the invention. Accordingly, the scope of the invention is limited only by the following claims and their equivalents.

Claims (20)

A method that includes: Collecting a virtual machine status of one or more virtual machines in a shared pool of resources, including a physical host that is in communication with at least one physical machine that supports one or more virtual machines, Collecting local topology information from the shared pool of resources, including periodic communication with an integrated monitor of the physical machine; Organizing the collected local topology information, including a storage topology that underlies a virtual topology, and the associated resource usage information; Taking advantage of organized topology information, including assigning a job to a selected virtual machine in the shared pool in response, including job assignment that supports efficient performance of the job's I / O, a return of a virtual topological distance that is assigned to the job, the virtual topological distance being selected from the group consisting of: a distance between two virtual machines and a distance between a virtual machine and a data block, A distinction is made between three job types, namely a first job type for assignment and reduction-intensive tasks, a second job type for assignment-intensive tasks and a third job type for reduction-intensive tasks, being in an execution a machine with the lowest load is assigned to a job according to the second job type, of a job according to the third job type, a short distance between associated data is preferred, and a job according to the first job type and the job is executed on the machine with the lowest load, and wherein, in order to avoid hotspots, the data are arranged on the machine with the lowest load by measuring the computing resources currently allocated to the machine and so on add the expected virtual machine assignment request that would work with the data to be placed on the machine. The procedure after Claim 1 , further comprising creating a shared memory channel to support the job, the channel supporting data communication between virtual machines between a first virtual machine and a second virtual machine located on the same physical machine. The procedure after Claim 2 , whereby the shared memory channel supports efficient data transmission for both the first and the second virtual machine. The procedure after Claim 1 wherein the step of using the organized storage topology information includes returning a physical location of one or more data blocks in the shared pool supporting the job. The procedure after Claim 4 , further comprising a return of usage information of a processing unit and network usage information of the physical machine that is local to the virtual machine. The procedure after Claim 5 wherein the step of assigning the job to a virtual machine includes selecting a virtual machine with sufficient bandwidth and a close topological distance to the physical location of one or more data blocks supporting the job. A system comprising: a shared pool of resources, the shared pool comprising a physical host in communication with multiple physical machines, each physical machine supporting at least one virtual machine, each physical machine having an integrated monitor, around collect virtual machine status from a local virtual machine, and each virtual machine has a built-in agent; a functional unit in communication with the physical host, the functional unit having tools to support efficient processing of a job in response to a topological architecture of the shared pool, the tools comprising: a director in communication with the host, the director periodically communicating with each integrated monitor and collecting and organizing a topology that underlies a virtual topology, the topology having a data pool storage topology and associated resource usage information, and the director being the organized topology to assign the job to one or more of the shared resources, the job assignment supporting efficient performance of the I / O assigned to the job, and further comprising a topology manager in communication with the director, the topology m anager returns a virtual topological distance selected from the group consisting of: a distance between two virtual machines and a distance between a virtual machine and a data block. A distinction is made between three job types, namely a first job type for assignment-intensive and reduction-intensive tasks, a second job type for assignment-intensive tasks and a third job type for reduction-intensive tasks, whereby when executing a job according to the second job type, a machine with the is assigned to the lowest load, a job according to the third job type is preferred a short distance between associated data, and a job according to the first job type and the job is executed on the machine with the lowest load, and wherein, in order to avoid hotspots, the data be placed on the machine with the least load by measuring the computing resources currently assigned to the machine and adding the expected assignment request of the virtual machine that would work with the data to be placed on the machine. The system according to Claim 7 , further comprising a hook manager which is in communication with an integrated agent from one of the virtual machines and is in communication with the director, the hook manager being a shared memory channel for data communication between virtual machines between a first virtual machine and a second virtual machine that is on the same physical machine. The system according to Claim 8 , whereby the shared memory channel supports efficient data transmission for both the first and the second virtual machine. The system according to Claim 7 further comprising a storage topology manager in communication with the director, the storage topology manager returning a physical location of one or more data blocks supporting the job in the shared pool. The system according to Claim 10 further comprising a resource usage manager in communication with the director, the resource usage manager returning processing unit usage information and network usage information of the physical machine local to the virtual machine. The system according to Claim 11 , further comprising an application manager in communication with the director, the application manager assigning the job to a virtual machine having sufficient bandwidth to support the job and a close topological distance to the physical location of one or more data blocks supporting the job. A computer program product that has a computer readable, non-volatile storage medium with computer readable program code contained therein, the computer readable program code, when executed on a computer, causes the computer to: collect information from the resources in the shared pool, the information being a virtual machine status from a local virtual machine in the shared pool of resources and having at least one status of a physical machine that supports one or more virtual machines; Periodically collecting local topology information from the shared pool of resources and communicating the acquired information with an integrated monitor of the physical machine; Organizing the collected local topology information, including a storage topology based on a virtual topology and related resource usage information; and Utilizing, in response, organized storage topology information and assigning a job to a selected virtual machine in the shared pool, including assignment that supports efficient performance of the job's I / O that still has program code to return a virtual topological distance that selected from the group consisting of: a distance between two virtual machines and a distance between a virtual machine and a data block. A distinction is made between three job types, namely a first job type for assignment-intensive and reduction-intensive tasks, a second job type for assignment-intensive tasks and a third job type for reduction-intensive tasks, whereby when executing a job according to the second job type, a machine with the is assigned to the lowest load, a job according to the third job type is preferred a short distance between associated data, and a job according to the first job type and the job is executed on the machine with the lowest load, and wherein, in order to avoid hotspots, the data be placed on the machine with the least load by measuring the computing resources currently assigned to the machine and adding the expected assignment request of the virtual machine that would work with the data to be placed on the machine. The computer program product after Claim 13 , further comprising program code to create a shared memory channel for data communication between virtual machines between a first virtual machine and a second virtual machine, which are located on the same physical machine. The computer program product after Claim 14 , whereby the shared memory channel supports efficient data transmission for the first and also the second virtual machine. The computer program product after Claim 13 , further comprising program code to return a physical location of one or more blocks of data supporting the job in the shared pool. The computer program product after Claim 16 , further comprising program code to return usage information of a processing unit and network usage information of the physical machine that is local to the virtual machine. The computer program product after Claim 17 , further comprising program code to assign the job to a virtual machine having sufficient bandwidth to support the job and a close topological distance to the physical location of one or more blocks of data supporting the job. A method implemented in a computer, comprising: collecting a virtual machine status from one or more virtual machines in a shared pool of resources, including a physical host that is in communication with at least one physical machine that has one or more virtual machines supports; periodically collecting local topology information of a hierarchical organization of resources represented by the physical and virtual machines; Organizing the topology information; Access usage information of storage resources and virtual machines represented in the topology; and assigning a job to one or more selected virtual machines in the shared pool, the job assignment supporting efficient performance in response to the topology and resource utilization evaluation, further comprising determining a topological distance associated with the job, the topological distance from the group is selected, which consists of: a distance between two virtual machines and a distance between a virtual machine and a data block, whereby three job types are distinguished in the assignment, namely a first job type for assignment and reduction-intensive tasks, a second Job type for assignment-intensive tasks and a third job type for reduction-intensive tasks, a machine with the lowest load being assigned when executing a job according to the second job type, a short distance between associated data is preferred for a job according to the third job type, and a job according to the first job type and the job is executed on the machine with the lowest load, and wherein, in order to avoid hotspots, the data are arranged on the machine with the lowest load by measuring the computing resources currently allocated to the machine and so on add the expected virtual machine assignment request that would work with the data to be placed on the machine. The procedure after Claim 19 that further includes creating a shared memory channel to support the job between a first virtual machine and a second virtual machine that are on the same physical machine.

Images