US20170223117A1 - Multi-level cloud computing system - Google Patents
Multi-level cloud computing system Download PDFInfo
- Publication number
- US20170223117A1 US20170223117A1 US15/425,484 US201715425484A US2017223117A1 US 20170223117 A1 US20170223117 A1 US 20170223117A1 US 201715425484 A US201715425484 A US 201715425484A US 2017223117 A1 US2017223117 A1 US 2017223117A1
- Authority
- US
- United States
- Prior art keywords
- service
- storage
- cloud
- network
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- H04L67/16—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4641—Virtual LANs, VLANs, e.g. virtual private networks [VPN]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/78—Architectures of resource allocation
- H04L47/783—Distributed allocation of resources, e.g. bandwidth brokers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Definitions
- the present disclosure relates generally to cloud computing, and more particularly to a customizable multi-vendor, multi-tenant cloud computing system.
- Cloud computing services can provide computational capacity, data access, networking/routing and storage services via a large pool of shared resources operated by a cloud computing provider. Because the computing resources are delivered over a network, cloud computing is location-independent computing, with all resources being provided to end-users on demand with control of the physical resources separated from control of the computing resources.
- cloud computing is a model for enabling access to a shared collection of computing resources - networks for transfer, servers for storage, and applications or services for completing work. More specifically, the term “cloud computing” describes a consumption and delivery model for IT services based on the Internet, and it typically involves over-the-Internet provisioning of dynamically scalable and often virtualized resources. This frequently takes the form of web-based tools or applications that users can access and use through a web browser as if it was a program installed locally on their own computer.
- Clouds are abstracted from consumers, who no longer have need for expertise in, or control over, the technology infrastructure “in the cloud” that supports them.
- Most cloud computing infrastructures consist of services delivered through common centers and built on servers. Clouds often appear as single points of access for consumers' computing needs, and do not require end-user knowledge of the physical location and configuration of the system that delivers the services.
- the utility model of cloud computing is useful because many of the computers in place in data centers today are underutilized in computing power and networking bandwidth. People may briefly need a large amount of computing capacity to complete a computation for example, but may not need the computing power once the computation is done.
- the cloud computing utility model provides computing resources on an on-demand basis with the flexibility to bring it up or down through automation or with little intervention.
- clouds should enable self-service, so that users can provision servers and networks with little human intervention.
- network access because computational resources are delivered over the network, the individual service endpoints need to be network-addressable over standard protocols and through standardized mechanisms.
- clouds typically provide metered or measured service—like utilities that are paid for by the hour, clouds should optimize resource use and control it for the level of service or type of servers such as storage or processing.
- SaaS Software as a Service
- PaaS Platform as a Service
- IaaS-style clouds provide computer resources that mimic physical resources, such as computer instances, network connections, and storage devices. The actual scaling of the instances may be hidden from the developer, but users are required to control the scaling infrastructure.
- a public cloud has an infrastructure that is available to the general public or a large industry group and is likely owned by a cloud services company.
- a private cloud operates for a single organization, but can be managed on-premise or off-premise.
- a hybrid cloud can be a deployment model, as a composition of both public and private clouds, or a hybrid model for cloud computing may involve both virtual and physical servers.
- a multi-vendor cloud is a hybrid cloud that may involve multiple public clouds, multiple private clouds, or some mixture.
- cloud computing requires the rapid and dynamic creation and destruction of computational units, frequently realized as virtualized resources. Maintaining the reliable flow and delivery of dynamically changing computational resources on top of a pool of limited and less-reliable physical servers provides unique challenges. Accordingly, it is desirable to provide a better-functioning cloud computing system with superior operational capabilities.
- FIG. 1 is a schematic view illustrating an external view of a cloud computing system.
- FIG. 2 is a schematic view illustrating an information processing system as used in various embodiments.
- FIG. 3 is a virtual machine management system as used in various embodiments.
- FIG. 4 is an IaaS-style network service according to various embodiments.
- FIG. 5 a is a diagram showing types of network access available to virtual machines in a cloud computing system according to various embodiments.
- FIG. 5 b is a flow chart showing the establishment of a VLAN for a project according to various embodiments.
- FIG. 6 a shows a message service system according to various embodiments.
- FIG. 6 b is a diagram showing how a directed message is sent using the message service according to various embodiments.
- FIG. 6 c is a diagram showing how a broadcast message is sent using the message service according to various embodiments.
- FIG. 7 is a PaaS-style identity and authentication service according to various embodiments.
- FIG. 8 is a PaaS-style object storage service according to various embodiments.
- FIG. 9 is a schematic view illustrating an embodiment of a logical structure provided by the object storage service according to various embodiments.
- FIG. 10 is a schematic view of a user account storage structure according to one embodiment.
- FIG. 11 is a flow chart illustrating a method for storing an object according to one embodiment.
- FIG. 12 is a flow chart illustrating an embodiment of a method for creating a ring according to one embodiment.
- FIG. 13 is a flow chart illustrating an embodiment of a method for reassigning partitions in a ring according to one embodiment.
- FIG. 14 is a PaaS-style image service according to various embodiments.
- FIG. 15 is an IaaS-style computational cloud service according to various embodiments.
- FIG. 16 is an instantiating and launching process for virtual resources according to various embodiments.
- FIG. 17 is a multi-level cloud infrastructure, according to various embodiments.
- FIG. 18 is a flow chart showing a method for responding to changes in demand for cloud services, according to various embodiments.
- FIG. 19 is a flow chart showing a method for operating a multi-level cloud infrastructure, according to various embodiments.
- the following disclosure has reference to computing services delivered on top of a cloud architecture.
- the cloud computing system 110 includes a user device 102 connected to a network 104 such as, for example, a Transport Control Protocol/Internet Protocol (TCP/IP) network (e.g., the Internet.)
- the user device 102 is coupled to the cloud computing system 110 via one or more service endpoints 112 .
- service endpoints 112 Depending on the type of cloud service provided, these endpoints give varying amounts of control relative to the provisioning of resources within the cloud computing system 110 .
- SaaS endpoint 112 a will typically only give information and access relative to the application running on the cloud storage system, and the scaling and processing aspects of the cloud computing system will be obscured from the user.
- PaaS endpoint 112 b will typically give an abstract API that allows developers to declaratively request or command the backend storage, computation, and scaling resources provided by the cloud, without giving exact control to the user.
- IaaS endpoint 112 c will typically provide the ability to directly request the provisioning of resources, such as computation units (typically virtual machines), software-defined or software-controlled network elements like routers, switches, domain name servers, etc., file or object storage facilities, authorization services, database services, queue services and endpoints, etc.
- resources such as computation units (typically virtual machines), software-defined or software-controlled network elements like routers, switches, domain name servers, etc., file or object storage facilities, authorization services, database services, queue services and endpoints, etc.
- users interacting with an IaaS cloud are typically able to provide virtual machine images that have been customized for user-specific functions. This allows the cloud computing system 110 to be used for new, user-defined services without requiring specific support.
- the control allowed via an IaaS endpoint is not complete.
- the cloud computing system 110 are one more cloud controllers 120 (running what is sometimes called a “cloud operating system”) that work on an even lower level, interacting with physical machines, managing the contradictory demands of the multi-tenant cloud computing system 110 .
- the workings of the cloud controllers 120 are typically not exposed outside of the cloud computing system 110 , even in an IaaS context.
- the commands received through one of the service endpoints 112 are then routed via one or more internal networks 114 .
- the internal network 114 couples the different services to each other.
- the internal network 114 may encompass various protocols or services, including but not limited to electrical, optical, or wireless connections at the physical layer; Ethernet, Fibre channel, ATM (Asynchronous Transfer Mode), and SONET (Synchronous Optical Networking) at the MAC (Media Access Control) layer; TCP, UDP (User Datagram Protocol), ZeroMQ or other services at the connection layer; and XMPP (Extensible Messaging and Presence Protocol), HTTP (HyperText Transfer Protocol), AMQP (Advanced Message Queuing Protocol), STOMP (Streaming Text Oriented Messaging Protocol), SMS (Short Message Service), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), or other standards at the protocol layer.
- Ethernet Fibre channel
- ATM Asynchronous Transfer Mode
- SONET Synchronous Optical Networking
- TCP Transmission Control
- UDP User Datagram Protocol
- ZeroMQ or other services at the connection layer
- XMPP Extensible Messaging and Presence Protocol
- the internal network 114 is typically not exposed outside the cloud computing system, except to the extent that one or more virtual networks 116 may be exposed that control the internal routing according to various rules.
- the virtual networks 116 typically do not expose as much complexity as may exist in the actual internal network 114 ; but varying levels of granularity can be exposed to the control of the user, particularly in IaaS services.
- processing or routing nodes in the network layers 114 and 116 , such as proxy/gateway 118 .
- Other types of processing or routing nodes may include switches, routers, switch fabrics, caches, format modifiers, or correlators. These processing and routing nodes may or may not be visible to the outside. It is typical that one level of processing or routing nodes may be internal only, coupled to the internal network 114 , whereas other types of network services may be defined by or accessible to users, and show up in one or more virtual networks 116 . Either of the internal network 114 or the virtual networks 116 may be encrypted or authenticated according to the protocols and services described below.
- one or more parts of the cloud computing system 110 may be disposed on a single host. Accordingly, some of the “network” layers 114 and 116 may be composed of an internal call graph, inter-process communication (IPC), or a shared memory communication system.
- IPC inter-process communication
- the cloud controllers 120 are responsible for interpreting the message and coordinating the performance of the necessary corresponding services, returning a response if necessary.
- the cloud controllers 120 may provide services directly, more typically the cloud controllers 120 are in operative contact with the service resources 130 necessary to provide the corresponding services.
- a “compute” service 130 a may work at an IaaS level, allowing the creation and control of user-defined virtual computing resources.
- a PaaS-level object storage service 130 b may provide a declarative storage API
- a SaaS-level Queue service 130 c, DNS service 130 d, or Database service 130 e may provide application services without exposing any of the underlying scaling or computational resources.
- Other services are contemplated as discussed in detail below.
- various cloud computing services or the cloud computing system itself may require a message passing system.
- the message routing service 140 is available to address this need, but it is not a required part of the system architecture in at least one embodiment.
- the message routing service is used to transfer messages from one component to another without explicitly linking the state of the two components. Note that this message routing service 140 may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between storage for cloud service state and for user data, including user service state.
- various cloud computing services or the cloud computing system itself may require a persistent storage for system state.
- the data store 150 is available to address this need, but it is not a required part of the system architecture in at least one embodiment.
- various aspects of system state are saved in redundant databases on various hosts or as special files in an object storage service.
- a relational database service is used to store system state.
- a column, graph, or document-oriented database is used. Note that this persistent storage may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between storage for cloud service state and for user data, including user service state.
- the cloud computing system 110 may be useful for the cloud computing system 110 to have a system controller 160 .
- the system controller 160 is similar to the cloud computing controllers 120 , except that it is used to control or direct operations at the level of the cloud computing system 110 rather than at the level of an individual service.
- a plurality of user devices 102 may, and typically will, be connected to the cloud computing system 110 and that each element or set of elements within the cloud computing system is replicable as necessary.
- the cloud computing system 110 whether or not it has one endpoint or multiple endpoints, is expected to encompass embodiments including public clouds, private clouds, hybrid clouds, and multi-vendor clouds.
- Each of the user device 102 , the cloud computing system 110 , the endpoints 112 , the network switches and processing nodes 118 , the cloud controllers 120 and the cloud services 130 typically include a respective information processing system, a subsystem, or a part of a subsystem for executing processes and performing operations (e.g., processing or communicating information).
- An information processing system is an electronic device capable of processing, executing or otherwise handling information, such as a computer.
- FIG. 2 shows an information processing system 210 that is representative of one of, or a portion of, the information processing systems described above.
- diagram 200 shows an information processing system 210 configured to host one or more virtual machines, coupled to a network 205 .
- the network 205 could be one or both of the networks 114 and 116 described above.
- An information processing system is an electronic device capable of processing, executing or otherwise handling information. Examples of information processing systems include a server computer, a personal computer (e.g., a desktop computer or a portable computer such as, for example, a laptop computer), a handheld computer, and/or a variety of other information handling systems known in the art.
- the information processing system 210 shown is representative of, one of, or a portion of, the information processing systems described above.
- the information processing system 210 may include any or all of the following: (a) a processor 212 for executing and otherwise processing instructions, (b) one or more network interfaces 214 (e.g., circuitry) for communicating between the processor 212 and other devices, those other devices possibly located across the network 205 ; (c) a memory device 216 (e.g., FLASH memory, a random access memory (RAM) device or a read-only memory (ROM) device for storing information (e.g., instructions executed by processor 212 and data operated upon by processor 212 in response to such instructions)).
- the information processing system 210 may also include a separate computer-readable medium 218 operably coupled to the processor 212 for storing information and instructions as described further below.
- an information processing system has a “management” interface at 1 GB/s (Gigabits per second), a “production” interface at 10 GB/s, and may have additional interfaces for channel bonding, high availability, or performance.
- An information processing device configured as a processing or routing node may also have an additional interface dedicated to public Internet traffic, and specific circuitry or resources necessary to act as a VLAN (Virtual Local Area Network) trunk.
- VLAN Virtual Local Area Network
- the information processing system 210 may include a plurality of input/output devices 220 a - n which are operably coupled to the processor 212 , for inputting or outputting information, such as a display device 220 a, a print device 220 b, or other electronic circuitry 220 c - n for performing other operations of the information processing system 210 known in the art.
- the computer-readable media and the processor 212 are structurally and functionally interrelated with one another as described below in further detail, and information processing system of the illustrative embodiment is structurally and functionally interrelated with a respective computer-readable medium similar to the manner in which the processor 212 is structurally and functionally interrelated with the computer-readable media 216 and 218 .
- the computer-readable media may be implemented using a hard disk drive, a memory device, and/or a variety of other computer-readable media known in the art, and when including functional descriptive material, data structures are created that define structural and functional interrelationships between such data structures and the computer-readable media (and other aspects of the system 200 ). Such interrelationships permit the data structures' functionality to be realized.
- the processor 212 reads (e.g., accesses or copies) such functional descriptive material from the network interface 214 , the computer-readable media 218 onto the memory device 216 of the information processing system 210 , and the information processing system 210 (more particularly, the processor 212 ) performs its operations, as described elsewhere herein, in response to such material stored in the memory device of the information processing system 210 .
- the processor 212 is capable of reading such functional descriptive material from (or through) the network 105 .
- the information processing system 210 includes at least one type of computer-readable media that is non-transitory.
- the information processing system 210 includes a hypervisor 230 .
- the hypervisor 230 may be implemented in software, as a subsidiary information processing system, or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- software may include software that is stored on a computer-readable medium, including the computer-readable medium 218 .
- the hypervisor may be included logically “below” a host operating system, as a host itself, as part of a larger host operating system, or as a program or process running “above” or “on top of” a host operating system. Examples of hypervisors include Xenserver, KVM (Kernel-based Virtual Machine), VMware, Microsoft's Hyper-V, and emulation programs such as QEMU.
- the hypervisor 230 includes the functionality to add, remove, and modify a number of logical containers 232 a - n associated with the hypervisor. Zero, one, or many of the logical containers 232 a - n contain associated operating environments 234 a - n.
- the logical containers 232 a - n can implement various interfaces depending upon the desired characteristics of the operating environment. In one embodiment, a logical container 232 implements a hardware-like interface, such that the associated operating environment 234 appears to be running on or within an information processing system such as the information processing system 210 .
- a logical container 234 could implement an interface resembling an x86, x86-64, ARM, or other computer instruction set with appropriate RAM, busses, disks, and network devices.
- a corresponding operating environment 234 for this embodiment could be an operating system such as Microsoft Windows, Linux, Linux-Android, or Mac OS X.
- a logical container 232 implements an operating system-like interface, such that the associated operating environment 234 appears to be running on or within an operating system.
- this type of logical container 232 could appear to be a Microsoft Windows, Linux, or Mac OS X operating system.
- Another possible operating system includes an Android operating system, which includes significant runtime functionality on top of a lower-level kernel.
- a corresponding operating environment 234 could enforce separation between users and processes such that each process or group of processes appeared to have sole access to the resources of the operating system.
- a logical container 232 implements a software-defined interface, such a language runtime or logical process that the associated operating environment 234 can use to run and interact with its environment.
- a corresponding operating environment 234 would use the built-in threading, processing, and code loading capabilities to load and run code. Adding, removing, or modifying a logical container 232 may or may not also involve adding, removing, or modifying an associated operating environment 234 .
- these operating environments will be described in terms of an embodiment as “Virtual Machines,” or “VMs,” but this is simply one implementation among the options listed above.
- a VM has one or more virtual network interfaces 236 . How the virtual network interface is exposed to the operating environment depends upon the implementation of the operating environment. In an operating environment that mimics a hardware computer, the virtual network interface 236 appears as one or more virtual network interface cards. In an operating environment that appears as an operating system, the virtual network interface 236 appears as a virtual character device or socket. In an operating environment that appears as a language runtime, the virtual network interface appears as a socket, queue, message service, or other appropriate construct.
- the virtual network interfaces (VNIs) 236 may be associated with a virtual switch (Vswitch) at either the hypervisor or container level. The VNI 236 logically couples the operating environment 234 to the network, and allows the VMs to send and receive network traffic.
- the physical network interface card 214 is also coupled to one or more VMs through a Vswitch.
- each VM includes identification data for use naming, interacting, or referring to the VM. This can include the Media Access Control (MAC) address, the Internet Protocol (IP) address, and one or more unambiguous names or identifiers.
- MAC Media Access Control
- IP Internet Protocol
- a “volume” is a detachable block storage device.
- a particular volume can only be attached to one instance at a time, whereas in other embodiments a volume works like a Storage Area Network (SAN) so that it can be concurrently accessed by multiple devices.
- Volumes can be attached to either a particular information processing device or a particular virtual machine, so they are or appear to be local to that machine. Further, a volume attached to one information processing device or VM can be exported over the network to share access with other instances using common file sharing protocols.
- the network operating environment 300 includes multiple information processing systems 310 a - n, each of which correspond to a single information processing system 210 as described relative to FIG. 2 , including a hypervisor 230 , zero or more logical containers 232 and zero or more operating environments 234 .
- the information processing systems 310 a - n are connected via a communication medium 312 , typically implemented using a known network protocol such as Ethernet, Fibre Channel, Infiniband, or IEEE (Institute for Electrical and Electronics Engineers) 1394.
- the network operating environment 300 will be referred to as a “cluster,” “group,” or “zone” of operating environments.
- the cluster may also include a cluster monitor 314 and a network routing element 316 .
- the cluster monitor 314 and network routing element 316 may be implemented as hardware, as software running on hardware, or may be implemented completely as software.
- one or both of the cluster monitor 314 or network routing element 316 is implemented in a logical container 232 using an operating environment 234 as described above.
- one or both of the cluster monitor 314 or network routing element 316 is implemented so that the cluster corresponds to a group of physically co-located information processing systems, such as in a rack, row, or group of physical machines.
- the cluster monitor 314 provides an interface to the cluster in general, and provides a single point of contact allowing someone outside the system to query and control any one of the information processing systems 310 , the logical containers 232 and the operating environments 234 . In one embodiment, the cluster monitor also provides monitoring and reporting capabilities.
- the network routing element 316 allows the information processing systems 310 , the logical containers 232 and the operating environments 234 to be connected together in a network topology.
- the illustrated tree topology is only one possible topology; the information processing systems and operating environments can be logically arrayed in a ring, in a star, in a graph, or in multiple logical arrangements through the use of vLANs.
- the cluster also includes a cluster controller 318 .
- the cluster controller is outside the cluster, and is used to store or provide identifying information associated with the different addressable elements in the cluster—specifically the cluster generally (addressable as the cluster monitor 314 ), the cluster network router (addressable as the network routing element 316 ), each information processing system 310 , and with each information processing system the associated logical containers 232 and operating environments 234 .
- the cluster controller 318 is outside the cluster, and is used to store or provide identifying information associated with the different addressable elements in the cluster—specifically the cluster generally (addressable as the cluster monitor 314 ), the cluster network router (addressable as the network routing element 316 ), each information processing system 310 , and with each information processing system the associated logical containers 232 and operating environments 234 .
- the cluster controller 318 includes a registry of VM information 319 .
- the registry 319 is associated with but not included in the cluster controller 318 .
- the cluster also includes one or more instruction processors 320 .
- the instruction processor is located in the hypervisor, but it is also contemplated to locate an instruction processor within an active VM or at a cluster level, for example in a piece of machinery associated with a rack or cluster.
- the instruction processor 320 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium. Further, one or more embodiments have associated with them a buffer 322 .
- the buffer 322 can take the form of data structures, a memory, a computer-readable medium, or an off-script-processor facility.
- a language runtime as an instruction processor 320 .
- the language runtime can be run directly on top of the hypervisor, as a process in an active operating environment, or can be run from a low-power embedded processor.
- the instruction processor 320 takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs.
- an interoperating bash shell, gzip program, an rsync program, and a cryptographic accelerator chip are all components that may be used in an instruction processor 320 .
- the instruction processor 320 is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor.
- This hardware-based instruction processor can be embedded on a network interface card, built into the hardware of a rack, or provided as an add-on to the physical chips associated with an information processing system 310 . It is expected that in many embodiments, the instruction processor 320 will have an integrated battery and will be able to spend an extended period of time without drawing current.
- Various embodiments also contemplate the use of an embedded Linux or Linux-Android environment.
- the information processing devices as described relative to FIG. 2 and the clusters as described relative to FIG. 3 are used as underlying infrastructure to build and administer various cloud services. Except where noted specifically, either a single information processing device or a cluster can be used interchangeably to implement a single “node,” “service,” or “controller.” Where a plurality of resources are described, such as a plurality of storage nodes or a plurality of compute nodes, the plurality of resources can be implemented as a plurality of information processing devices, as a one-to-one relationship of information processing devices, logical containers, and operating environments, or in an M ⁇ N relationship of information processing devices to logical containers and operating environments.
- virtual machines or “virtual devices”; as described above, those refer to a particular logical container and operating environment, configured to perform the service described.
- the term “instance” is sometimes used to refer to a particular virtual machine running inside the cloud computing system.
- An “instance type” describes the compute, memory and storage capacity of particular VM instances.
- groups of resources may be organized into “zones.”
- a zone is defined as a group of one or more resources that share one or more common characteristics and are grouped together to achieve some function.
- one example of a zone is an availability zone, which is a group of resources subject to a correlated loss of access or data as a result of a particular event.
- a group of operating environments that use a common underlying network connection, power connection, or computer-readable medium is subject to loss of access to its stored objects as a result of a failure of one of these resources.
- a group of resources in the same cluster may suffer a loss of access if the cluster is contained within a single physical building.
- Zones may overlap and may be defined for different reasons.
- a group of resources that share a computer-readable medium may be defined to be in one zone.
- a plurality of information processing devices 210 (and their underlying hypervisors 230 , logical containers 232 and operating environments 234 ) in a given storage rack or cabinet may be defined to be in a zone
- a zone is defined by the availability of specialized hardware or by hardware of a certain class.
- a plurality of information processing devices 210 and their underlying hypervisors 230 , logical containers 232 and operating environments 234 ) that have access to high-throughput and low-latency storage, like a solid state disk, may be defined to be in a zone.
- Zones may be organized to reflect differing amounts of memory, processor speeds, age of installation, type of operating system or hypervisor, or any other underlying difference in platforms.
- the network has one or more physical edge routers 402 which connect to the broader Internet. These edge routers 402 are connected to an underlying physical distribution network 404 .
- the implementation of the physical distribution network 404 may encompass various protocols or services, including but not limited to electrical, optical, or wireless connections at the physical layer; Ethernet, Fibre channel, ATM, and SONET at the MAC layer; MPLS (Multi-protocol Label Switching) between layers 2 and 3, and TCP, UDP, ZeroMQ or other services at the connection layer.
- the physical distribution network 404 corresponds to the internal network 114 described relative to FIG. 1 .
- the physical distribution network 404 is connected to physical interfaces 423 at each physical information processing system 420 .
- the physical information processing systems 420 correspond to the information processing systems 210 as described relative to FIG. 2 , with a hypervisor 421 , and operating environments within logical containers 422 , each of which with one or more virtual network interfaces (VNIs) 424 .
- VNIs virtual network interfaces
- the virtual network service 400 corresponds to the virtual network 116 as describe relative to FIG. 1 .
- the virtual network service 400 defines packet flows between hosts in the network and is used to logically route packets independent of the underlying physical infrastructure.
- the virtual network service uses virtual routers 406 and 426 to define and manage these flows.
- Each virtual router both virtual routers 406 and 426 ) have an API 410 that allows the routing tables, packet filters, and underlying logical network fabric to be remotely reconfigured.
- the virtual routers may be located in the physical edge routers 402 , or within one of the information processing systems 420 , within the hypervisor 421 , within a particular operating environment 422 , or at an outside network routing controller 430 . These virtual network devices can work together to present multiple logical connections and networks that are independent from each other even though they are running on the same physical distribution network 404 . Information from a particular flow is tunneled or bridged from one physical network segment to another or is kept logically connected by associating it with an ATM-like circuit or an MPLS-style label.
- the virtual network service API 410 allows for creation and management of virtual networks each of which can have one or more ports.
- a port on a virtual network can be attached to a network interface, where a network interface is anything which can source traffic, such as physical interface 423 or one of the VNIs 424 .
- the API 410 allows users of the network as well as cloud administrators to declaratively define a network architecture including a series of virtual connections, routing rules, and filters.
- the VNIs 424 are then able to interact with the virtual network in the same way as a physical network without needing to also configure the underlying physical distribution network 404 .
- Adapters are used to bridge between the virtual network devices and the physical devices without exposing the way in which the physical devices are actually connected.
- a network device is instantiated within an operating system to communicate with physical interface 423 operating over a defined protocol, such as Ethernet or MPLS.
- a VNI 424 is also defined by the operating system. While the VNI 424 will often be associated with an operating environment, logical container, or virtual machine, those of skill in the art will recognize that VNIs 424 can be created as pure abstractions to help administer or logically control the network flows.
- one or more kernel bridges are defined routing the layer 2 packets from the physical network interface 423 to one or more VNIs 424 . The raw packets arriving on the physical interface 423 are routed according to the established bridges and then used by the operating environments, hypervisor, or physical device.
- the routing fabric associated with a particular physical network interface 423 can be reconfigured by changing the bridges associated with that interface.
- the routing between VNIs 424 is also software-defined, making the flows across the VNIs 424 similarly malleable to software manipulation via the API 410 .
- an entire flow coming off of a physical network interface 423 or a flow coming off of a VNI 424 can be filtered, diverted, or inspected independently.
- advanced networking functionality such as load balancing, firewalls, layer 7 routing, and geographic routing can be accomplished in a multi-tenant environment.
- One implementation of this uses standard networking utilities.
- the physical network device 423 and the VNIs 424 while different to the kernel, expose a standard network or socket API to upper-level layers and applications. Accordingly, a standard firewall is able to filter the packets and send them on without being aware that the packets being filtered are arriving over a virtual network interface instead of a physical one.
- the packet-routing logic within the cloud system as a whole can be centrally directed using network routing controller 430 .
- each router has an independent routing table and seeks to find the best route for each packet that arrives by examining its existing routing tables and probing the network around it using various protocols adapted for that purpose.
- the network service 400 can work identically to standard networks in this respect, but the ability to dynamically sense and control the flows using API 410 allows routing decisions to be made centrally when that makes more sense.
- Network routing controller 430 may, for example, control the routing across virtual routers 406 and 426 to direct traffic in a way that may not be locally optimal but provides for greater global throughput and higher overall utilization.
- one embodiment of the virtual network service 400 provides a plug-in architecture allowing entry routing (controlled by virtual router 406 ), overall network routing (controlled by the network routing controller 430 or by the combination of virtual routers) or any particular router 426 to have a tenant-defined flow routing and filtering policy.
- one plugin allows a tenant-specific security policy to be implemented on a flow.
- a virtual router 426 is instantiated for use by a particular user (the “user router”).
- the user router can be implemented in the hypervisor 421 or in a distinct operating environment 422 .
- the user router has an API 410 , just like the other virtual routers 426 .
- routing rules can be set using the API 410 .
- a series of rules can be provided and loaded by the user router 426 .
- This loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module.
- the virtual router 406 identifies it as being logically addressed to a particular operating environment associated with the user and routes it to flow to the defined user router 426 instantiated for the customer by way of the physical interface 423 and possibly other virtual routers 426 along the way.
- the tenant-defined rules and filters are applied to the packet and the flow is stopped, edited, or redirected accordingly.
- a second embodiment allows QoS (Quality of Service) policies to be implemented on a flow.
- QoS Quality of Service
- a centralized routing policy allows global maximization of network utilization.
- a user-defined network routing controller 430 (the “user controller”) is defined, or a plug-in or ruleset is loaded into a system-wide network routing controller using the same functionality described above.
- the user controller can route other packets on less-speedy routes and keep the most direct route free for high-priority packets. In this way the delay on best-effort packets is due mostly to a higher number of hops as opposed to rate limiting; each individual router (either physical or virtual) is operating at its peak capacity.
- the load is managed by virtually spreading out the number of routers handling packets rather than by limiting the numbers of packets being routed at a particular node.
- the network 500 is one embodiment of a virtual network 116 as discussed relative to FIG. 1 , and is implemented on top of the internal network layer 114 .
- a particular node is connected to the virtual network 500 through a virtual network interface 236 operating through physical network interface 214 .
- the VLANs, VSwitches, VPNs (Virtual Private Networks), and other pieces of network hardware (real or virtual) may be network routing elements 316 or may serve another function in the communications medium 312 .
- the cloud computing system 110 uses both “fixed” IP addresses and “floating” IP addresses to address virtual machines.
- Fixed IP addresses are assigned to an instance on creation and stay the same until the instance is explicitly terminated.
- Floating IP addresses are IP addresses that can be dynamically associated with an instance. A floating IP address can be disassociated and associated with another instance at any time.
- Different embodiments include various strategies for implementing and allocating fixed IP addresses, including “flat” mode, a “flat DHCP” (Dynamic Host Control Protocol) mode, and a “VLAN DHCP” mode.
- fixed IP addresses are managed using a flat Mode.
- an instance receives a fixed IP from a pool of available IP addresses. All instances are attached to the same bridge by default. Other networking configuration instructions are placed into the instance before it is booted or on boot.
- fixed IP addresses are managed using a flat DHCP mode.
- Flat DHCP mode is similar to the flat mode, in that all instances are attached to the same bridge. Instances will attempt to bridge using the default Ethernet device or socket. Instead of allocation from a fixed pool, a DHCP server listens on the bridge and instances receive their fixed IPs by doing a dhcpdiscover.
- the network 500 includes three nodes, network node 510 , private node 520 , and public node 530 .
- the nodes include one or more virtual machines or virtual devices, such as DNS (Domain Name System)/DHCP server 512 and virtual router 514 on network node 510 ,
- VPN 522 and private VM 524 on private node 520 and public VM 532 on public node 530 .
- VLAN DHCP mode requires a switch that supports host-managed VLAN tagging.
- DHCP server 512 is running on a VM that receives a static VLAN IP address at a known address, and virtual router 514 , VPN 522 , private VM 524 , and public VM 532 all receive private IP addresses upon request to the DHCP server running on the DHCP server VM.
- the DHCP server provides a public IP address to the virtual router 514 and optionally to the public VM 532 .
- the DHCP server 512 is running on or available from the virtual router 514 , and the public IP address of the virtual router 514 is used as the DHCP address.
- VLAN DHCP mode there is a private network segment for each project's or group's instances that can be accessed via a dedicated VPN connection from the Internet.
- each VLAN project or group gets its own VLAN, network bridge, and subnet.
- subnets are specified by the network administrator, and assigned dynamically to a project or group when required.
- a DHCP Server is started for each VLAN to pass out IP addresses to VM instances from the assigned subnet. All instances belonging to the VLAN project or group are bridged into the same VLAN. In this fashion, network traffic between VM instances belonging to the same VLAN is always open but the system can enforce isolation of network traffic between different projects by enforcing one VLAN per project.
- VLAN DHCP mode includes provisions for both private and public access.
- For private access (shown by the arrows to and from the private users cloud 502 ), users create an access keypair (as described further below) for access to the virtual private network through the gateway VPN 522 .
- the gateway VPN 522 From the VPN 522 , both the private VM 524 and the public VM 532 are accessible via the private IP addresses valid on the VLAN.
- Public access is shown by the arrows to and from the public users cloud 505 .
- Communications that come in from the public users cloud arrive at the virtual router VM 515 and are subject to network address translation (NAT) to access the public virtual machine via the bridge 516 .
- Communications out from the private VM 524 are source NATted by the bridge 516 so that the external source appears to be the virtual router VM 515 . If the public VM 532 does not have an externally routable address, communications out from the public VM 532 may be source NATted as well.
- the second IP in each private network is reserved for the VPN instance 522 .
- the network for each project is given a specific high-numbered port on the public IP of the network node 510 . This port is automatically forwarded to the appropriate VPN port on the VPN 522 .
- each group or project has its own certificate authority (CA) 523 .
- the CA 523 is used to sign the certificate for the VPN 522 , and is also passed to users on the private users cloud 502 .
- a certificate is revoked, a new Certificate Revocation List (CRL) is generated.
- the VPN 522 will block revoked users from connecting to the VPN if they attempt to connect using a revoked certificate.
- VLANs are segregated using 802.1q VLAN tagging in the switching layer, but other tagging schemes such as 802.1ad, MPLS, or frame tagging are also contemplated.
- the flows are defined by the virtual network 400 as described relative to FIG. 4 , so the underlying implementation can be chosen independent of the logical virtual network on top.
- Network hosts create VLAN-specific interfaces and bridges as required using a user router 426 .
- private VM 524 has per-VLAN interfaces and bridges created as required. These do not have IP addresses in the host to protect host access. Access is provided via routing table entries created per project and instance to protect against IP/MAC address spoofing and ARP (Address Resolution Protocol) poisoning.
- ARP Address Resolution Protocol
- FIG. 5 b is a flow chart showing the establishment of a VLAN for a project according to one embodiment.
- the process 550 starts at step 551 , when a VM instance for the project is requested.
- a user needs to specify a project for the instances, and the applicable security rules and security groups (as described herein) that the instance should join.
- a cloud controller determines if this is the first instance to be created for the project. If this is the first, then the process proceeds to step 553 . If the project already exists, then the process moves to step 559 .
- a user controller is identified to act as the network host for the project. This may involve creating a virtual network device and assigning it the role of network controller.
- this is a virtual router 514 .
- an unused VLAN id and unused subnet are identified.
- the VLAN id and subnet are assigned to the project.
- DHCP server 512 and bridge 516 are instantiated and registered.
- the VM instance request is examined to see if the request is for a private VM 524 or public VM 532 . If the request is for a private VM, the process moves to step 558 . Otherwise, the process moves to step 559 .
- the VPN 522 is instantiated and allocated the second IP in the assigned subnet.
- FIGS. 5 a and 5 b can appear to be configured and provisioned as in a legacy network, or can be completely implemented virtually using plugins, virtual routers, and centralized routing.
- the underlying implementation is transparent to the consumer of the network services.
- Other plugin implementations are defined similarly; load balancing can be dynamically adjusted based upon actual load; failover or service scaling can be defined “in the network,” and can occur transparently.
- Various services are envisioned, including basic network connectivity, network packet filtering, IP address management, load balancing, QoS, layer 7 routing, VLANs, L2-in-L3 (and other layer) tunneling, advanced security services and geo-routing.
- Multiple plugins can be combined to provide layered capabilities, with each plugin service being defined within a separate “segment” of the network for visibility and debuggability.
- a message queuing service is used for both local and remote communication so that there is no requirement that any of the services exist on the same physical machine.
- Various existing messaging infrastructures are contemplated, including AMQP, ZeroMQ, STOMP and XMPP. Note that this messaging system may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between internal messaging services and any messaging services associated with user data.
- the messaging service may run alongside or on top of the network service 400 described relative to FIGS. 4-5 b.
- the message service sits between various components and allows them to communicate in a loosely coupled fashion. This can be accomplished using Remote Procedure Calls (RPC hereinafter) to communicate between components, built atop either direct messages and/or an underlying publish/subscribe infrastructure. In a typical embodiment, it is expected that both direct and topic-based exchanges are used. This allows for decoupling of the components, full asynchronous communications, and transparent balancing between equivalent components.
- RPC Remote Procedure Calls
- calls between different APIs can be supported over the distributed system by providing an adapter class which takes care of marshalling and unmarshalling of messages into function calls.
- a cloud controller 120 (or the applicable cloud service 130 ) creates two queues at initialization time, one that accepts node-specific messages and another that accepts generic messages addressed to any node of a particular type. This allows both specific node control as well as orchestration of the cloud service without limiting the particular implementation of a node.
- the API can act as a consumer, server, or publisher.
- FIG. 6 a one implementation of a message service 140 is shown at reference number 600 .
- FIG. 6 a shows the message service 600 when a single instance 602 is deployed and shared in the cloud computing system 110 , but the message service 600 can be either centralized or fully distributed.
- the message service 600 keeps traffic associated with different queues or routing keys separate, so that disparate services can use the message service without interfering with each other. Accordingly, the message queue service may be used to communicate messages between network elements, between cloud services 130 , between cloud controllers 120 , between network elements, or between any group of sub-elements within the above. More than one message service 600 may be used, and a cloud service 130 may use its own message service as required.
- a message server 602 including one or more exchanges 610 .
- the message system is “brokerless,” and one or more exchanges are located at each client.
- the exchanges 610 act as internal message routing elements so that components interacting with the message service 600 can send and receive messages.
- these exchanges are subdivided further into a direct exchange 610 a and a topic exchange 610 b.
- An exchange 610 is a routing structure or system that exists in a particular context. In a currently preferred embodiment, multiple contexts can be included within a single message service with each one acting independently of the others.
- the type of exchange such as a direct exchange 610 a vs. topic exchange 610 b determines the routing policy.
- the routing policy is determined via a series of routing rules evaluated by the exchange 610 via a plugin service described further below.
- the direct exchange 610 a is a routing element created during or for RPC directed message operations. In one embodiment, there are many instances of a direct exchange 610 a that are created as needed for the message service 600 . In a further embodiment, there is one direct exchange 610 a created for each RPC directed message received by the system.
- the topic exchange 610 b is a routing element created during or for RPC directed broadcast operations.
- every message received by the topic exchange is received by every other connected component.
- the routing rule within a topic exchange is described as publish-subscribe, wherein different components can specify a discriminating function and only topics matching the discriminator are passed along.
- a queue 615 is a message stream; messages sent into the stream are kept in the queue 615 until a consuming component connects to the queue and fetches the message.
- a queue 615 can be shared or can be exclusive. In one embodiment, queues with the same topic are shared amongst Workers subscribed to that topic.
- element 620 is a topic publisher.
- a topic publisher 620 is created, instantiated, or awakened when an RPC directed message or an RPC broadcast operation is executed; this object is instantiated and used to push a message to the messages system. Every publisher connects always to the same topic-based exchange; its life-cycle is limited to the message delivery.
- element 630 is a direct consumer.
- a direct consumer 630 is created, instantiated, or awakened if an RPC directed message operation is executed; this component is instantiated and used to receive a response message from the queuing system.
- Every direct consumer 630 connects to a unique direct-based exchange via a unique exclusive queue, identified by a UUID (Universally Unique Identifier) or other unique name.
- UUID Universally Unique Identifier
- the life-cycle of the direct consumer 630 is limited to the message delivery.
- the exchange and queue identifiers are included the message sent by the topic publisher 620 for RPC directed message operations.
- element 640 is a topic consumer.
- a topic consumer 640 is created, instantiated, or awakened at system start.
- a topic consumer 640 is created, instantiated, or awakened when a topic is registered with the message system 600 .
- a topic consumer 640 is created, instantiated, or awakened at the same time that a Worker or Workers are instantiated and persists as long as the associated Worker or Workers have not been destroyed.
- the topic consumer 640 is used to receive messages from the queue and it invokes the appropriate action as defined by the Worker role.
- a topic consumer 640 connects to the topic-based exchange either via a shared queue or via a unique exclusive queue.
- every Worker has two associated topic consumers 640 , one that is addressed only during an RPC broadcast operations (and it connects to a shared queue whose exchange key is defined by the topic) and the other that is addressed only during an RPC directed message operations, connected to a unique queue whose with the exchange key is defined by the topic and the host.
- a direct publisher 660 is instantiated to send a response message via the message system 600 .
- the direct publisher 640 sends a message to an exchange 610 a.
- the response message is dispatched by the exchange 610 a.
- the response message is fetched by the direct consumer 630 instantiated to receive the response and dictated by the routing key.
- the message response is passed to the Invoker.
- FIG. 6 c one embodiment of the process of sending an RPC broadcast message is shown relative to the elements of the message system 600 as described relative to FIG. 6 a .
- a topic publisher 620 is instantiated.
- the topic publisher 620 sends a message to an exchange 610 b.
- the message is dispatched by the exchange 610 b.
- the message is fetched by a topic consumer 640 dictated by the routing key (either by topic or by topic and host).
- the message is passed to a Worker associated with the topic consumer 640 .
- a response to an RPC broadcast message can be requested.
- the process follows the steps outlined relative to FIG. 6 b to return a response to the Invoker.
- this loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module.
- the plugin interface is a messaging “device” that implements a particular messaging pattern.
- a messaging “device” that implements a particular messaging pattern.
- one embodiment uses a “forwarding” device. This is particularly useful in a cloud architecture where the endpoints of a message are relatively dynamic and there may not be a stable connection point.
- This plugin controls the routing of messages in a queue and passes messages from one (possibly dynamic) messaging endpoint to another.
- Another possible plugin “device” embodiment implements a pipeline-style messaging architecture. Messages arriving at one end of the device are sent to one or more downstream Workers, where the messages can be processed in parallel instead of serially.
- a third embodiment of a plugin device is a load balancer.
- Rule-based computing organizes statements into a data model that can be used for deduction, rewriting, and other inferential or transformational tasks. The data model can then be used to represent some problem domain and reason about the objects in that domain and the relations between them.
- one or more controllers or services have an associated rule processor that performs rule-based deduction, inference, and reasoning.
- Rule engines are particularly applicable as a plugin interface, where particular outcome can be presented either as the outcome of a series of rules provided by an end user or as the output of a “black box” that takes one or more defined inputs and returns an output.
- Rule Engines can be implemented similarly to script processors as described relative to FIG. 3 , and may be implemented as a sub-module of a script processor where needed. In other embodiments, Rule Engines can be implemented as discrete components, for example as a tailored electrical circuit or as software instructions to be used in conjunction with a hardware processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium. Further, one or more embodiments have associated with them a buffer. The buffer can take the form of data structures, a memory, a computer-readable medium, or an off-rule-engine facility.
- one embodiment uses a language runtime as a rule engine, running as a discrete operating environment, as a process in an active operating environment, or can be run from a low-power embedded processor.
- the rule engine takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs.
- the rule engine is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor.
- FIG. 7 one exemplary authentication (“authn”) and authorization (“authz”) system is shown at 700 .
- the authn/authz system shown at 700 is exemplary only and can be used in several different aspects within a cloud computing system.
- one embodiment uses a single authn/authz system for the entire cloud computing “platform,” including all subsidiary services.
- Another embodiment has separate authn/authz services for each subsidiary service.
- a third embodiment has a split authn/authz service, with authn being handled globally so that user, project, tenant, and account details are consistent, but allowing authz to be handled within each subsidiary service so that the privileges granted are logically and physically separable.
- a further embodiment has multiple authn/authz services, where there are one or more authn/authz services 600 that are used to mediate the authentication and authorization of users to interact with the cloud computing platform and its subsidiary components, and one or more authn/authz services 700 that are tenant-specific and used to control and expose internal virtual cloud resources in a controlled fashion. Rule-based access controls govern the use and interactions of these logical entities.
- a role-based computing system is a system in which identities and resources are managed by aggregating them into “roles” based on job functions, physical location, legal controls, and other criteria. These roles can be used to model organizational structures, manage assets, or organize data. By arranging roles and the associated rules into graphs or hierarchies, these roles can be used to reason about and manage various resources.
- RBAC Role-Based Access Control
- RBAC associates special rules, called “permissions,” with roles; each role is granted only the minimum permissions necessary for the performance of the functions associated with that role. Identities are assigned to roles, giving the users and other entities the permissions necessary to accomplish job functions.
- RBAC has been formalized mathematically by NIST (National Institute of Standards and Technology) and accepted as a standard by ANSI (American National Standards Institute).
- American National Standard 359-2004 is the information technology industry consensus standard for RBAC, and is incorporated herein by reference in its entirety.
- a user is defined as an entity that will act in one or more roles.
- a user is typically associated with an internal or external entity that will interact with the cloud computing system in some respect.
- a user can have multiple roles simultaneously.
- a user's roles define which API commands that user can perform.
- the “user” construct may sometimes represent a physical human being, there is no limitation that a user must be a person. Rather, a user may be a person, system, or service who interacts with the cloud computing system and is associable with a distinct identity.
- the authn/authz service 700 validates requests and API calls with reference to the declared user. No particular implementation of the user construct is required; users may be locally unique within a defined realm or globally unique within the cloud computing system. In most embodiments, a user will be identified using one or more “IDs.” These user IDs can be numbers, strings of characters, or compound multi-valued objects.
- this user ID is (or is associated with) a “login”—a sequence of characters presented to the authn/authz system 700 to identify the user.
- users can also have further non-security-related metadata associated with them, such as names, companies, email addresses, locations, etc.
- a resource is defined as some object to which access is restricted.
- resources can include network or user access to a virtual machine or virtual device, the ability to use the computational abilities of a device, access to storage, an amount of storage, API access, ability to configure a network, ability to access a network, network bandwidth, network speed, network latency, ability to access or set authentication rules, ability to access or set rules regarding resources, etc.
- any item which may be restricted or metered is modeled as a resource.
- resources may have quotas associated with them.
- a quota is a rule limiting the use or access to a resource.
- a quota can be placed on a per-project level, a per-role level, a per-user level, or a per-group level.
- quotas can be applied to the number of volumes which can be created, the total size of all volumes within a project or group, the number of instances which can be launched, both total and per instance type, the number of processor cores which can be allocated, and publicly accessible IP addresses. Other restrictions are also contemplated as described herein.
- a project is defined as a flexible association of users, acting in certain roles, which will define and access various resources.
- a project is typically defined by an administrative user according to varying demands. There may be templates for certain types of projects, but a project is a logical grouping created for administrative purposes and may or may not bear a necessary relation to anything outside the project.
- arbitrary roles can be defined relating to one or more particular projects only.
- a group is defined as a logical association of some other defined entity.
- a group “development” is defined.
- the development group may include a group of users with the tag “developers” and a group of virtual machine resources (“developer machines”). These may be connected to a developer-only virtual network (“devnet”).
- the development group may have a number of ongoing development projects, each with an associated “manager” role. There may be per-user quotas on storage and a group-wide quota on the total monthly bill associated with all development resources.
- the applicable set of rules, roles, and quotas is based upon context.
- a user's actual permissions in a particular project are the intersection of the global roles, user-specific roles, project-specific roles, and group-specific roles associated with that user, as well as any rules associated with project or group resources possibly affected by the user.
- Other embodiments may also use other grouping or organizational identifiers, such as domains, realms, locations, and organizations.
- a credential is an identifying token.
- This token can be either something that is uniquely known or controlled by the user, such as a password, key, certificate, token, or other piece of information, or it can be something associated with the user's identity, such as voice prints, iris or retinal scan data, fingerprints, etc.
- unique machine-identifying data can also be used as a credential, such as IP addresses, MAC addresses, embedded key, computer chip, or serial number information, or computationally secure data processing circuits.
- the identity-based credential is unique to the user being identified.
- credentials are treated probabilistically, and a number of credentials can be combined to increase the confidence that the user has been authenticated correctly.
- authentication is the act of confirming the identity of a user or the truth of a claim.
- the authn/authz service 700 confirms that incoming requests are being made by the user who claims to be making the call by validating a set of claims provided by the user.
- the claims are initially in the form of a set of credentials (username & password, or login and API key). After initial confirmation, The authn/authz service 700 issues a token that can serve as a credential.
- authorization is the act of confirming the capability of a user to perform some action.
- the authn/authz service 700 confirms that the user is authorized to perform the actions corresponding to the user's incoming requests.
- An individual authorization is called a “permission.”
- a “default” or “anonymous” profile is used as the default identity.
- a token can contain or refer to a set of authorization permissions.
- a token is a data (such as a string) that corresponds to an identity. Tokens are generally cryptographically generated to prevent forging and are associated with access controls. Each token has a scope and a timeframe that describe the resources that can be accessed using the token.
- an endpoint is a network-accessible address, usually described by URL (Uniform Resource Locator) or URI (Uniform Resource Identifier), where a service may be accessed.
- URL Uniform Resource Locator
- URI Uniform Resource Identifier
- authn/authz service will be described in terms of specific embodiments below, the logical constructs are more important than the particular implementations, and the representation of any particular part of the system may change from embodiment to embodiment.
- the authn/authz service 700 is organized as a group of internal services exposed on one or many endpoints 702 . Many of these services are used in a combined fashion by the frontend APIs (such as the service endpoints 112 described relative to FIG. 1 ).
- the authz/authn service 700 immediately authenticates and authorizes a request arriving to the cloud computing system and creates and return a token credential that can be used internally to authorize and authenticate the services without having to duplicate the original credential checks. This token may be returned to the user or may be a hidden internal implementation detail for operational efficiency within the cloud computing system as a whole.
- the authn/authz service 700 has a service-specific endpoint within each subsidiary service of the cloud computing system, such as endpoints 702 a, 702 b, and 702 c.
- endpoints 702 a, 702 b, and 702 c can be exposed over HTTP, OAUTH, XAUTH, XML-RPC (Extensible Markup Language-Remote Procedure Call), SOAP (Simple Object Access Protocol), CORBA (Common Object Request Broker Architecture), LDAP (Lightweight Directory Access Protocol), COM (Component Object Model), Kerberos, RADIUS (Remote Authentication Dial In User Service), OpenID, SAML (Security Assertion Markup Language), or other protocols known in the art.
- the authn/authz service 700 has four primary divisions, the authn provider 710 , the authz provider 720 , the storage provider 730 , and the credential service 740 .
- the authn provider 710 performs the calculations or comparisons needed to authenticate a user given the provided credentials.
- the authz provider 720 performs the necessary analysis to determine whether an identified user has the necessary permissions to perform a requested action.
- the storage provider 730 provides access to a durable storage that contains authentication and authorization rules, tokens, credentials, and other necessary data.
- the credential service 740 provides new credentials on demand.
- each of these four divisions presents a unified API for consumers, regardless of the underlying implementation.
- backend services that actually implement, store, or calculate the data needed to provide the authentication, authorization, user/credential storage and generation services exposed by the providers 710 , 720 , 730 , and 740 .
- These backend services may include but are not limited to a rule engine 750 , a disk store 760 , an LDAP database 770 , or an RDBMS (Relational Database Management System) 780 .
- FIG. 7 shows only one backend provider of each type and an architecture in which all backend providers are commonly accessible, this is for purposes of illustration only and should not be considered a limitation.
- a first embodiment of the rule engine 750 encompasses multiple rule engines working in parallel or a single rule engine 750 .
- the rule engine 750 may or may not have access to one or more disks 760 , LDAP stores 770 , RDBMS's 780 . In another embodiment, some of the backend services may be combined.
- one embodiment uses database logic, triggers, and stored procedures to implement the rule engine 750 .
- Another implementation uses an RBAC analysis engine associated with an LDAP datastore like Active Directory or OpenLDAP.
- a third embodiment of the rule engine 750 is implemented as described elsewhere in this document. Other embodiments may use other implementations as known in the art.
- the disk store 760 is implemented using structured files on a computer-readable medium.
- a disk store 760 comprises /etc/passwd and /etc/grp files.
- a second embodiment of a disk store 760 is a structured data store, such as a CSV (Cluster Shared Volumes), XML, SQLite or BerkelyDB-formatted and structured file on a disk.
- a third embodiment uses an object storage service. Other embodiments may use other implementations as known in the art.
- the LDAP store 770 is implemented using commercially available tree-structured directory such as Active Directory, OpenLDAP, the Red Hat directory service or Novell eDirectory. In another embodiment, the LDAP store 770 is implemented using a graph database or a key-value database, a document database, or a tree-structured filesystem with disk-based storage and attributes. Other embodiments may use other implementations as known in the art.
- the RDBMS 780 is implemented using commercially available relational database such as MySQL, PostgreSQL, Oracle database server, Microsoft SQL Server, SQLite, DB2, or other relational databases as known in the art.
- relational database such as MySQL, PostgreSQL, Oracle database server, Microsoft SQL Server, SQLite, DB2, or other relational databases as known in the art.
- Each of the four main divisions of the authn/authz service 700 can call out to other portions of the service, either using the API bus 704 or the direct access bus 706 .
- the API bus 704 is a communications medium, such as a message service or network connection, that allows one provider to call another provider's API as part of its functionality.
- the credential service 740 uses the API bus 704 to retrieve a representation of the permissions associated with a particular user so that the permissions can be encoded into a generated token.
- the direct access bus 706 is a communications medium, such as a message service or network connection, which allows one provider to call out to one or more of the backend services to provide part or all of its functionality.
- the authz provider 720 uses organizational information stored in the LDAP store 770 to evaluate whether a particular action is authorized.
- the service proxy 790 transparently connect a remote implementation of one or more of the providers ( 710 , 720 , 730 , 740 ) or services ( 750 , 760 , 770 , 780 ) for execution in a remote location.
- the service proxy supports federation, caching, and user-specific business logic in the authn/authz system 700 .
- two services capable of providing authn/authz service exist, authn/authz system 1 and authn/authz system 2 .
- Each system has a set of local backend services providing authn, authz, storage, and credential access via the endpoints 702 for a set of users, where the two sets of users are disjoint.
- a secure communication channel exists between the service proxy 790 of each service and the remote service.
- “fallback” logic is provided for each of the authn provider 710 , authz provider 720 , and storage provider 730 . The fallback logic first looks to the local implementation for authentication and authorization of users and permissions.
- the service proxy 790 is used to present the same user request to the remote system. Only if both the local and remote calculation of authentication, authorization, or storage fails is a rejection returned.
- connection from the service proxy it is preferred for the connection from the service proxy to go to an endpoint 702 or API bus 704 of a remote system so as to reduce the coupling between systems, but it is equally possible to use the service proxy to connect directly to a remote service implementation.
- the remote service implementation does not need to be controlled by the cloud service provider, and in fact it may be more common to have a secure connection to a customer-internal authentication and authorization service so that sensitive authentication and authorization information can stay inside user-controlled systems and never be transmitted outside of a protected area.
- the embodiment described above is bidirectional, but it would be equally easy to have a unidirectional federation by only connecting one service proxy 790 to the remote system without having the second reciprocal link.
- the credential service 740 can also be proxied using the service proxy 790 , but it may be more efficient in some circumstances to generate a new temporary credential at the local system so that the cross-service proxied call does not need to be made for subsequent accesses.
- the service proxy 790 is used to support caching of remote information. This is related to the federation embodiment described above, except that a copy of the credentials and necessary information needed to identify the correct authentication and authorization responses can be saved in a local data store for use in subsequent connections.
- a “TTL” (time-to-live) value can be imposed on the cached values so as to force periodic reauthorization, and the connection between the service proxy 790 and the remote system can be used to revoke particular tokens or credentials for reasons other than the expiration of time.
- TTL time-to-live
- An advantage of particular embodiments is that the response to certain API calls can be cached without revealing the underlying mechanism used to compute the response.
- the service proxy 790 is used to implement user-specific business logic via a plugin system.
- the service proxy 790 can be used to replace or proxy any provider or backend service to an arbitrary point, including a local service implementing user-specific logic.
- the service proxy 790 provides a plugin API or RPC API that allows user-defined authentication, authorization, storage, and credential services to be used transparently without any change in the outer API endpoints 702 . In one implementation, this is performed similarly to the federation or caching examples above, but the “remote” service is actually provided at a local node running the user-provided plugin.
- each provider or service is configurable to use a backend so that the authn/authz service 700 can fit a variety of environments and needs.
- the backend for each service is defined in a configuration file and the proper backend is loaded at runtime.
- a key-value store backend is implemented that can support primary key lookups, the most trivial implementation being an in-memory dictionary lookup.
- a relational database mapper can be used to provide many services on top of a traditional SQL (Sequence Query Language) backend, outside of simple RDBMS service.
- PAM Pluggable Authentication Modules
- This provides a one-to-one relationship between Users and Groups with the root User also having the ‘admin’ role.
- authentication of a user is performed through public/private encryption, with keys used to authenticate particular users, or in some cases, particular resources such as particular machines.
- a user or machine may have multiple keypairs associated with different roles, projects, groups, or permissions. For example, a different key may be needed for general authentication and for project access.
- a user is identified within the system by the possession and use of one or more cryptographic keys, such as an access and secret key.
- a user's access key needs to be included in a request, and the request must be signed with the secret key.
- the rules engine verifies the signature and executes commands on behalf of the user.
- Some resources can be shared by many users. Accordingly, it can be impractical or insecure to include private cryptographic information in association with a shared resource.
- the system supports providing public keys to resources dynamically.
- a public key such as an SSH (Secure Shell) key, is injected into a VM instance before it is booted. This allows a user to login to the instances securely, without sharing private key information and compromising security.
- SSH Secure Shell
- a rule processor is also used to attach and evaluate rule-based restrictions on non-user entities within the system.
- a “Cloud Security Group” (or just “security group”) is a named collection of access rules that apply to one or more non-user entities. Typically these will include network access rules, such as firewall policies, applicable to a resource, but the rules may apply to any resource, project, or group.
- a security group specifies which incoming network traffic should be delivered to all VM instances in the group, all other incoming traffic being discarded. Users with the appropriate permissions (as defined by their roles) can modify rules for a group. New rules are automatically enforced for all running instances and instances launched from then on.
- a project or group administrator specifies which security groups it wants the VM to join. If the directive to join the groups has been given by an administrator with sufficient permissions, newly launched VMs will become a member of the specified security groups when they are launched.
- an instance is assigned to a “default” group if no groups are specified.
- the default group allows all network traffic from other members of this group and discards traffic from other IP addresses and groups. The rules associated with the default group can be modified by users with roles having the appropriate permissions.
- a security group is similar to a role for a non-user, extending RBAC to projects, groups, and resources.
- one rule in a security group can stipulate that servers with the “webapp” role must be able to connect to servers with the “database” role on port 3306.
- an instance can be launched with membership of multiple security groups—similar to a server with multiple roles.
- Security groups are not necessarily limited, and can be equally expressive as any other type of RBAC security.
- all rules in security groups are ACCEPT rules, making them easily composible.
- each rule in a security group must specify the source of packets to be allowed. This can be specified using CIDR (Classless Inter-Domain Routing) notation (such as 10.22.0.0/16, representing a private subnet in the 10.22 IP space, or 0.0.0.0/0 representing the entire Internet) or another security group.
- CIDR Classless Inter-Domain Routing
- the creation of rules with other security groups specified as sources helps deal with the elastic nature of cloud computing; instances are impermanent and IP addresses frequently change.
- security groups can be maintained dynamically without having to adjust actual IP addresses.
- the APIs, RBAC-based authentication system, and various specific roles are used to provide a US eAuthentication-compatible federated authentication system to achieve access controls and limits based on traditional operational roles.
- the implementation of auditing APIs provides the necessary environment to receive a certification under FIPS 199 Moderate classification for a hybrid cloud environment.
- Typical implementations of US eAuth authentication systems are structured as a Federated LDAP user store, back-ending to a SAML Policy Controller.
- the SAML Policy Controller maps access requests or access paths, such as requests to particular URLs, to a Policy Agent in front of an eAuth-secured application.
- the application-specific account information is stored either in extended schema on the LDAP server itself, via the use of a translucent LDAP proxy, or in an independent datastore keyed off of the UID provided via SAML assertion.
- API calls are secured via access and secret keys, which are used to sign API calls, along with traditional timestamps to prevent replay attacks.
- the APIs can be logically grouped into sets that align with the following typical roles:
- System Administrators and Developers have the same permissions, Project and Group Administrators have the same permissions, and Cloud Administrators and Security have the same permissions.
- the End-user or Third-party User is optional and external, and may not have access to protected resources, including APIs. Additional granularity of permissions is possible by separating these roles.
- the RBAC security system described above is extended with SAML Token passing.
- the SAML token is added to the API calls, and the SAML UID is added to the instance metadata, providing end-to-end auditability of ownership and responsibility.
- APIs can be grouped according to role. Any authenticated user may:
- Network Administrators may:
- Cloud Administrators and Security personnel would have all permissions.
- access to the audit subsystem would be restricted.
- Audit queries may spawn long-running processes, consuming resources.
- detailed system information is a system vulnerability, so proper restriction of audit resources and results would be restricted by role.
- APIs are extended with three additional type declarations, mapping to the “Confidentiality, Integrity, Availability” (“C.I.A.”) classifications of FIPS 199 .
- C.I.A. Constantiality, Integrity, Availability
- Additional parameters would also apply to creation of block storage volumes and creation of object storage “buckets.”
- C.I.A. classifications on a bucket would be inherited by the keys within the bucket.
- Establishing declarative semantics for individual API calls allows the cloud environment to seamlessly proxy API calls to external, third-party vendors when the requested C.I.A. levels match.
- a hybrid or multi-vendor cloud uses the networking architecture described relative to FIG. 4 and the RBAC controls to manage and secure inter-cluster networking.
- the hybrid cloud environment provides dedicated, potentially co-located physical hardware with a network interconnect to the project or users' cloud virtual network.
- User-specific security policies are implemented as plugins on the network without necessarily disclosing the underlying implementation or specific business rules implemented by the plugin.
- the interconnect is a bridged VPN connection.
- a security group is created specifying the access at each end of the bridged connection.
- the interconnect VPN implements audit controls so that the connections between each side of the bridged connection can be queried and controlled.
- Network discovery protocols ARP, CDP
- ARP Network discovery protocols
- CDP can be used to provide information directly, and existing protocols (SNMP location data, DNS LOC records) overloaded to provide audit information.
- FIG. 8 a diagram showing the logical structure of an object storage service is shown at reference 800 .
- the structure connects a user 802 through a proxy 804 , where the proxy 804 provides an API endpoint for interacting with the storage service 800 .
- the proxy 804 is connected to one or more rings 806 such as an object ring 806 a, a container ring 806 b, and an account ring 806 c, described in further detail below, that are connected to an object service 808 , container service 810 , and an account service 812 , respectively, described in further detail below.
- rings such as a structured data ring, a graph storage ring, or another type of ring (not pictured).
- each ring would be connected to an appropriate service, such as a structured data service, a graph service, or another service (not pictured).
- these rings and services are implemented as services integrated with or running on top of an information processing system 210 as described relative to FIG. 2 . These could be running alone, or configured to host one or more virtual machines; the services and rings can be implemented on “bare metal,” or within one or more of the virtual machines (logical containers and operating environments).
- the services and functionalities described relative to the object storage service may be centralized onto a smaller number of physical devices or may be expanded out to a large number of independent physical machines, including machines in different zones.
- a storage management server 816 centralizes the proxy 804 and the rings 806
- a storage pool server 818 centralizes the object service 808 , the container service, 810 , the account service 812 , and the storage pools 814 .
- This grouping is for convenience in explanation only, however, and is not intended to be limiting.
- each of these groupings may be logically present in a particular embodiment but may be implemented using a plurality of physical devices in a configuration that is different from the logical configuration presented.
- each storage pool 814 is provided by a separate storage server 818 or includes a virtual server that is included in a portion of one of the storage servers 818 or across a plurality of the storage servers 818 .
- the storage servers 818 may be physically located in one or more data centers, and the resources of the storage servers 818 may be virtualized according to the requirements of a plurality of users (e.g., the user 802 ) such that the plurality of storage pools 814 are provided to the plurality of users in order to store files and/or data objects.
- resources for a particular virtual server or storage pool may span across multiple storage servers 818 .
- the multi-zone object storage service 840 encompasses multiple zones 850 a - 850 n. These zones may be included in separate regions, such as the exemplary regions 852 a and 852 b. Each zone may also be under the control of a separate organization. Each zone includes a object storage service, such as the object storage service described relative to FIG. 8 , possibly including in each object storage service a proxy 854 a - n, one or more rings 856 a - n, object, container, account, or other services 858 a - n, and a storage pool 860 a - n.
- object storage service such as the object storage service described relative to FIG. 8 , possibly including in each object storage service a proxy 854 a - n, one or more rings 856 a - n, object, container, account, or other services 858 a - n, and a storage pool 860 a - n.
- the user 802 interacts with each zone independently, addressing any requests directly to the proxies 854 a - n.
- the multi-zone proxy 862 is used to provide a single entry point to the zones 850 a - n
- the multi-zone ring 864 is used to balance requests across the zones 850 a - n.
- An embodiment may use either a multi-zone proxy 862 , or a multi-zone ring 864 , or both, or neither.
- the individual zones 850 a - n can optionally forego the use of the zone-specific proxy 854 a - n or zone-specific ring 856 a - n.
- a particular user 802 interacts with the storage service 800 via the API to store and receive data objects.
- User 802 may create a plurality of containers 1002 in the user account 1000 and store a plurality of data objects 1004 in each of the containers 1002 for retrieval.
- a user account is referred to as an “account”
- a container is referred to as a “container”
- a data object is referred to as an “object” for clarity of discussion.
- the terms “account,” “container” and “object” are generic forms of data naming that are used to direct the object storage service 800 to a specific data object.
- rings and services When other types of rings and services are used, an appropriate name may be substituted.
- discussion of alternative rings and services will be limited to the “account,” “container” and “object” rings and services.
- the account as shown in FIG. 10 are consistent whether or not they are deployed on a logical structure within a single cluster, such as the structure illustrated in FIG. 10 , or are arrayed across a multi-zone system, such as the structure illustrated in FIG. 9 .
- each zone 850 a - n (specifically the proxies 854 a - n, the rings 856 a - n, the object/container services 858 a - n, and the storage pools 860 a - n ).
- equivalent structures to the proxy 804 and the rings 806 are contemplated relative to the multi-zone proxy 862 and the multi-cluster ring 864 .
- the rings 806 are implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- software may include software that is stored on a computer-readable medium location in the storage management server 816 and/or the storage servers 818 .
- the rings 806 include semiconductor circuits and/or computer-executable instructions that, when executed by a processor, provide subsystems of the object storage service 800 that provide a mapping between the entities stored in the object storage service 800 and the locations of those entities in the storage pools 814 .
- the object storage service 800 includes a separate object ring 806 a, container ring 806 b, and account ring 806 c, and when components of the object storage service 800 need to perform any operation on an object, container, or account, those components interact with the object ring 806 a, container ring 806 b, and account ring 806 c, respectively, to determine the location of that stored entity in the storage pools 814 .
- ring structures may be provided (e.g., a single ring for the objects, containers, and accounts, more than one ring for each of the objects, containers, and account, etc.) without departing from the scope of the present disclosure.
- the rings 806 maintain the availability and safety of data in the object storage service 800 through the use of zones, partitions, replicas, and the storage pools 814 , as described below.
- zones are defined across one or more of the storage pools 814 that are subject to a correlated loss of access or data as a result of a particular event.
- zones can be used for identification and management of other aspects of the system, such as the use of specific storage technology.
- a partition is an abstract storage bucket.
- the object storage service 800 maps each partition to a plurality of storage pools 814 that are in different zones, and stores data using those partitions.
- the mapping of a given partition to a plurality of storage pools 814 creates a plurality of partition replicas of that partition (e.g., equal to the number of storage pools 814 the partition is mapped to.) For example, when a given partition is mapped to 3 storage pools 814 that are in different zones, 3 partition replicas of that partition are created.
- the object ring 806 a for the management of objects will be described in detail below. However, one of skill in the art will recognize how the discussion may be applied to the container ring 806 b, the account ring 806 c, and/or a ring for any other stored entity, without departing from the scope of the present disclosure.
- an object from a user is received by a proxy.
- some attribute of the object or the object data itself is hashed. If necessary, some attribute of the object is modified so that three different results are returned from the hashing function.
- the object is then replicated and stored in the storage pool corresponding to the number returned by the hash function.
- a consistent hashing function is used as the hash function.
- the use of the consistent hashing function ensures that there will be minimal changes to the assigned storage pools given a change in membership due to adding or removing new storage pools.
- the consistent hashing function results in minimal changes to the storage location, sometimes the assignments made by the consistent hashing function or the rearrangements needed due to a change in membership may have undesirable storage characteristics. For example, such methods have been found to result in multiple object replicas for the same object being stored in one or more storage pools that are in the same zone. As discussed above, this is undesirable because then multiple (and possibly all) object replicas for the same object are subject to being lost as a result of a particular event. Alternatively, rebalancing the replicas due to a change in membership has been found to require the movement to two of the replicas 4% of the time, and the movement of all three replicas 1% of the time. It is desirable to never have to move more than one replica at a time.
- the object storage service 800 solves the problem of multiple object replicas for the same object being stored in storage pools that are in the same zone through the use of the rings 806 .
- a method 400 for storing stored entities is illustrated.
- an object is received by a user.
- an object is received from the user 202 by the proxy 804 .
- the method 1100 then proceeds to block 1104 where a partition identification is generated.
- a consistent hash function is applied to the object received in block 1102 , and the hash function returns a partition identification that corresponds to a partition.
- the method 1100 then proceeds to block 1106 where the partition associated with the partition identification is mapped to storage pools that are in different zones. This mapping function is constrained so that the physical location of the storage pools is required to have one or more desirable properties, such as having each partition replica in a separate zone.
- the constrained mapping function is the output of a constraint satisfaction solver, in which the desired storage characteristics (such as the requirement that each replica of a partition be in a different availability zone) are inputs to the solving function.
- the solver uses one or more search methodologies within the solution space to find a storage layout that maps partitions to storage pools in a desirable manner.
- a constrained mapping function is applied to portions of the partition identification (e.g., the portions of the partition identification that the constrained mapping function is applied to) may be bits of the output of the original hashing function is applied to the object.
- the number of bits to which the constrained mapping function is applied may be known as the partition power, and 2 to the partition power may indicate the partition count.
- the constrained mapping function is designed to return a storage pool location for each portion of the partition identification to which it is applied, and the storage pool locations returned for a given partition identification will each correspond to storage pools 814 in different zones. These storage pool locations are then associated with the partition identification.
- the partition corresponding to the partition identification is replicated multiple times in the object storage service 800 (i.e., a partition replica is included in each storage pool corresponding to the storage pool locations determined from the constrained mapping function.)
- the method 1100 then proceeds to block 1108 where the object is stored according to the partition.
- the object received by the user 802 in block 1102 of the method 1100 may then be stored according to the partition corresponding to the partition identification, which results in multiple object replicas for the object being stored in storage pools that are in different zones in the object storage service 800 .
- the constrained mapping function is used to determined storage pool locations that are in different zones for each partition prior to the object being received by the user 802 , discussed in further detail below.
- the output of the constrained mapping function signifies a particular storage pool where a replica of the partition should be stored.
- An example of this output is as follows:
- a hash function is applied to the object.
- the user 802 provides data including an account/container/object name to the proxy 804 , and a hash function is applied to the account/container/object name as follows:
- the partition mapping number may then be divided into 3 parts (e.g., the first three digits, the second three digits, and the third three digits of the partition identification,) and the constrained mapping function is applied to each of those parts: Constrained mapping function (123) storage pool location (zone 1) Constrained mapping function (456) storage pool location (zone 7) Constrained mapping function (789) storage pool location (zone 3)
- the constrained mapping function is designed to return the storage pool location (zone 1), storage pool location (zone 7), and storage pool location (zone 3) that correspond to storage pools that are in different zones (e.g., zones 1, 3, and 7).
- Partition identification (storage pool location (zone 1), storage pool location (zone 7), storage pool location (zone 3)
- the partition corresponding to the partition identification is replicated across storage pools that are in different zones (here, zones 1, 3, and 7.)
- the object received from the user 802 is then stored, using the partition corresponding to the partition identification, in each of the storage pools corresponding to the storage pool locations returned by the application of the constrained mapping function to portions of the partition identification.
- each of the storage pool locations are IP addresses, i.e., when each of the storage pools are separate storage servers.
- the constrained mapping function is a hash function.
- one of skill in the art will recognize that a variety of functions may be used to ensure that each partition is mapped to storage pools that are in different zones without departing from the scope of the present disclosure.
- the constrained mapping function is applied to the object storage service 800 before the object is received by the user 802 at block 1102 in order to accomplish the mapping of the partitions to storage pools described above with reference to block 1106 of the method 1100 .
- the total number of partitions and the total number of storage servers/storage pools in the object storage service 800 may (and typically will) be known.
- the constrained mapping function is used to map each partition in the object storage service 800 to a plurality of storage pools that are in different zones, and that information is stored in a constrained mapping database.
- a constrained mapping database may include partitions mapped to storage pools such as:
- Partition 1 storage pool location (zone 1), storage pool location (zone 2), storage pool location (zone 3)
- Partition 2 storage pool location (zone 4), storage pool location (zone 5), storage pool location (zone 6)
- Partition 3 storage pool location (zone 7), storage pool location (zone 8), storage pool location (zone 9)
- the output of the constrained mapping function can be saved for optimized lookup.
- the saved output may be embodied in a file provided to each of the storage pools 814 , or stored in a database that is available for the appropriate systems to query. If the saved output is contained within a file, the storage pools 814 may then periodically check the modification time of this file and reload their in-memory copies of the ring structure as needed.
- the hash function is applied to that object to get the partition identification (e.g., partition 1, 2, or 3 in the example above) at block 1104 , and then at block 1106 , the partition identification may then be used with the constrained mapping database to determine the corresponding partition and its associated storage pool locations.
- the partition identification e.g., partition 1, 2, or 3 in the example above
- the partition identification may then be used with the constrained mapping database to determine the corresponding partition and its associated storage pool locations.
- an ideal number of partitions for each storage pool in the object storage service is determined.
- the number of partitions that should ideally be assigned to each storage pool 814 is calculated based the weight (e.g., storage capacity) of each storage pool 814 . For example, if the partition power is 20, the ring 806 will have 1,048,576 (2 20 ) partitions. If there are 1,000 storage pools 814 of equal weight, each storage pool 814 will ideally be assigned 1,048.576 partitions. This may be referred to as an ideal partition count, and in the example, each storage pool 814 starts off empty with a 1,048.576 ideal partition count.
- the method 1200 then proceeds to block 1204 where the storage pools are placed in a sorting order.
- the storage pools 814 are placed in a sorting order based on their ideal partition count (e.g., highest to lowest), and this sorting order is maintained throughout the method 1200 as partitions are assigned storage pools 814 .
- the method 1200 then proceeds to block 1206 where partitions are assigned to storage pools based on their sorting order but with a zone restriction.
- the partitions are assigned to the storage pool 814 with the highest ideal partition count, but subject to the restriction that the storage pool 814 to which a partition is being assigned is not in the same zone as any other storage pool 814 that includes a partition replica for that same partition.
- the method 1200 then proceeds to block 1208 where the sorting order of the storage pools is adjusted.
- that storage pool 814 will have its ideal partition count decremented and thus that storage pool 814 is moved to a lower position in the sorting order, and the method 1200 then returns to block 1206 to continue to assign partitions to storage pools based on their sorting order but with the zone restriction.
- each partition is assigned multiple storage pools 814 in different zones, and thus objects received from users may have multiple object replicas stored in storage pools in different zones simply by associating those objects with the partitions.
- Another problem relates to the rebalancing of object replicas stored in the object storage service due to changing membership (i.e., adding or subtracting storage servers or storage pools from the object storage service.)
- changing membership i.e., adding or subtracting storage servers or storage pools from the object storage service.
- the mapping of partitions to multiple storage pools in different zones in the object storage service 800 described above solves these problems.
- the use of the constrained mapping function to ensure that each partition is mapped to storage pools in different zones ensures that object replicas for the same object are never located in storage pools 814 that are in the same zone (i.e., because any given object received from a user is stored in a partition that is replicated in storage pools that are in different zones.)
- each storage server 816 defined as a separate zone
- the addition or subtraction of a given storage server 816 from the object storage service 800 thus can only effect one partition replica, and hence one object replica of a given object (i.e., because only one of the partition replica will ever be located on a storage server that is defined as a separate zone.)
- the rebalancing associated with changing the zone membership can be accomplished without affecting more than one replica because each zone is guaranteed to only contain one replica of a given partition.
- partitions may need to be reassigned to different storage pools 814 , and the reassignment of partitions will result in the building of a new ring from an old ring. Such an event may occur due to the removal and/or addition of a storage pool 814 from the object storage service 800 (e.g., a membership change.)
- FIG. 13 a method 1300 for reassigning partitions in response to the removal of a storage pool is illustrated.
- the method 1300 begins at block 1302 where the ideal number of partitions for each storage pool is recalculated. In one embodiment, the ideal partition count for the storage pools 814 remaining in the object storage service 800 (subsequent to the removal of a storage pool) is recalculated.
- the method 1300 then proceeds to block 1304 where the storage pools are placed in a sorting order as described above with reference to block 504 of the method 500 .
- the method then proceeds to block 1306 where partitions to be reassigned are grouped. In one embodiment, a partition list for the partitions to be reassigned is created.
- any storage pools 814 that have been removed from the filesystem 800 may have all their assigned partitions unassigned and added to the partition list, and any storage pools 814 that have more partitions than their ideal partition count may have random partitions unassigned from them and added to the partition list (i.e., such that those storage pools have a number of partitions that is within a predetermined amount of their ideal partition count.)
- the partitions on the partition list may then be reassigned to the storage pool 814 in blocks 1308 and 610 of the method 1300 substantially as discussed above with reference to blocks 506 and 508 of the method 500 .
- reassignment times may be used when gathering partitions to reassign to storage pools 814 , such that no partition replica for a given partition is moved twice in a predetermined amount of time.
- reassignment restrictions based on reassignment times may be ignored for partition replicas on storage pools 814 that have been removed from the object storage service 800 , as removing a storage pool 814 only happens upon storage pool 814 /storage server 816 failure and thus requires the reassignment of the partitions.
- the method 1300 is conducted periodically to help balance the amount of data stored by storage pools 814 in the object storage service 800 .
- the partition reassignment method 1300 discussed above may repeated until each storage pool 814 is within a predetermined threshold of a predetermined storage capacity (e.g., within 1% of 130% storage capacity for that storage pool) or when it is determined that partition reassignment will not improve the balance of data stored by the object storage service 800 by more than a predetermined amount.
- step 1310 adjusts the sorting order for efficiency, load sharing, or other reasons.
- the constrained mapping within the rings 806 is modifiable to deal with more than issues of availability. Even though the embodiment above has been described in terms of availability zones, any type of zone can be used for the mapping from partition to storage device.
- the constrained mapping is extensible via a plugin mechanism.
- the constrained mapping function is a rule-based hash, and so it can be extended similarly to other rule-based plugin mechanisms described herein.
- the ring 806 has an API by which mapping rules can be set by user 802 .
- a series of rules can be provided and loaded by the ring 806 .
- This loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module.
- the constrained mapping function When set of partitions is provided to the plugin, the constrained mapping function logically associates it with a particular resource and assigns the virtual partition to the applicable physical resource. When a request arrives addressed to that resource, the tenant-defined rules and filters are applied to the request and the appropriate object is uploaded, downloaded, edited, or redirected accordingly.
- the API provided to the user 802 is sufficient to for the user to direct the creation of the ring structure via interaction with a remote, user-controlled entity on the network.
- the user can direct the allocation of underlying resources via the plugin mechanism.
- the user can present weights, correlations, or affinities between different groups of resources so that an underlying cloud-provider-specific ring generation function can take those into account.
- the user can have direct control over the allocation of storage resources and those can be mapped onto available physical resources provided by the cloud provider. Plugging into the constraint mapping portion of the ring generation function is sufficient for both levels of control, but it is anticipated that different cloud providers will favor or provide one level of control over another.
- the object service 808 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- those instructions may include software that is stored on a computer-readable medium located in the storage management server 816 and/or the storage servers 818 .
- the object service 808 may include instructions that, when executed by a processor, provide object storage and objection manipulation functionality such that the object service 808 is operable to, for example, store, retrieve and delete stored objects in the storage pools 814 .
- an object service 808 is provided for each storage pool that holds object data.
- an object service 808 may be included on a server that further includes one or more storage drives that provide a storage pool for objects.
- the objects are stored as binary files with metadata stored as extended attributes of the file in the filesystem used by the object storage service.
- the object service 808 will uses the extended attributes of the filesystem to manage the metadata.
- the metadata is stored in a machine-readable format next to the data itself.
- the metadata for a file is stored in a text file or single file database.
- subsequent processing based on file metadata is also pluggable by the end user.
- one embodiment stores different sets of files and associates them with the same URL.
- An ordinary operation to read or modify the object could also have a “processor” function associated with it that allows for user-directed modification or logging of the request.
- objects are stored by the object service 808 using a path derived by applying a hash function to the name of the object along with a timestamp. For example, an incoming object for a user account to be written to a container will have a hash applied to its account/container/object name and the path generated for the object is:
- objects/ ⁇ partition>/ ⁇ storage pool location>/objectname_hash.15673.data where “objects” indicate that the object data is stored in an object storage pool 814 , ⁇ partition>is the partition identification that maps the object to a partition, ⁇ storage pool location>is the storage pool location that maps the partition replica to a storage pool 814 in a different zone than its related partition replicas, objectname_hash is the hash of the account/container/object name, and 15672 is the timestamp.
- the object storage service 800 When there is a request for an object, the object storage service 800 will find all the object replicas in the object storage service 800 that include the objectname_hash and return the object data that has the most recent timestamp value. Special care is needed to record updates that should be persisted as the new canonical value.
- a modification sentinel e.g., a 0 byte “tombstone” file or “.ts” file
- a modification sentinel e.g., a 0 byte “tombstone” file or “.ts” file
- that tombstone file stays in the object storage service 800 for a predetermined amount of time (e.g., 7 days.)
- a predetermined amount of time e.g. 7 days.
- the object storage service 800 searches for and deletes any object replicas that it finds related to that tombstone file (e.g., replicas that same objectname_hash as the tombstone file) to ensure that objects that were meant to be deleted from the object storage service 800 are removed and older versions of object replicas of a given object do not appear in the object storage service 800 due to, for example, the temporary failure of a storage server 816 or storage pool 814 that might have prevented the deletion of that object replica previously. If the object storage service 800 determines that a tombstone file has been in the object storage service 800 for longer than the predetermined time, that tombstone file is deleted.
- that tombstone file is deleted.
- the mechanism used for recording file deletion is also used to record other types of updates.
- a “purge” marker indicates that the system should overwrite all copies of the object and set the space to free
- a “version” marker indicates that the system should create a copy and mark the copy with a version number
- a “ttl” (time-to-live) marker indicates that the system should check an authoritative source for updates after the expiry of a set time period.
- Other types of out-of-band changes to the file are also contemplated.
- the container service 810 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- those instructions may include software that is stored on a computer-readable medium located in the storage management server 816 and/or the storage servers 818 .
- the container service 810 may include instructions that, when executed by a processor, provide container storage and container manipulation functionality such that the container service 810 is operable to store, retrieve and delete stored containers in the storage pools 814 .
- a container service 810 is provided for each storage pool that holds container data.
- a container service 810 may be included on a server that further includes one or more storage drives that provide a storage pool for containers, and the container service 810 may include the names of containers and objects in those containers.
- the container service 810 handles the listing of containers, and does not hold the location where the objects are stored (e.g., the storage pool where a given object replica resides), but rather the locations of containers that hold the objects.
- the listings for the container locations may be stored as database files, and those listings may be replicated across the storage pools 814 in a manner that is similar to the replication of objects (e.g., through their association with partitions) as discussed above.
- Container storage statistics for the container service(s) 810 may be tracked by the object storage service 800 and may include total number of objects stored by one or more containers, the total storage provided by any given container, and/or a variety of other statistics known in the art.
- the account service 812 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- those instructions may include software that is stored on a computer-readable medium located in the storage management server 816 and/or the storage servers 818 .
- the account service 812 may include instructions that, when executed by a processor, provide account storage and account manipulation functionality such that the account service 812 is operable to store, retrieve and delete stored accounts in the storage pools 814 .
- an account service 812 is provided for each storage pool that holds account data.
- an account service 812 may be implemented by a server that includes storage drives that provide a storage pool for accounts, and the account service 812 may include the names of accounts and containers in those accounts.
- the account service 812 is very similar to the container service 810 , discussed above, with the exception that account storage 812 handles the listings of accounts.
- an authorization service may include instructions that, when executed by a processor, handle the storage and manipulation of authorization metadata so that the authorization service is operable to store, retrieve, delete, and query stored credentials from in the storage pools 814 .
- an authorization service provides an ACL-based authorization.
- the authorization service provides posix-compatible authorization.
- the authorization service provides tree or graph-based authorization, such as would be provided with an LDAP-based authorization service.
- a second implementation includes a structured data service.
- the structured data service may include instructions that, when executed by a processor, provide the storage and manipulation of structured data such that the structured data service is operable to store, retrieve, delete, and query tabular, graph, or tree-based data from in the storage pools 814 .
- a structured data service provides a JSON (JavaScript Object Notation)-based output.
- the structured data service provides XML-based output.
- the structured data service provides HTML output.
- the proxy 804 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- the proxy 804 is responsible for tying together the object storage service 800 .
- the proxy 804 determines the location of the account, container, or object in the appropriate ring 806 (e.g., the object ring 806 a, the container ring 806 b, or the account ring 806 c ,) and routes the request accordingly.
- a public Application Programming Interface (API) may be exposed to users through the proxy 804 .
- a large number of failures may be handled by the proxy 804 . For example, if a storage server 816 and/or storage pool 814 is unavailable for an object PUT, the proxy 804 may use the rings 806 to determine an appropriate storage server 816 and/or storage pool 814 for that object and route the object there instead.
- API Application Programming Interface
- proxies there are multiple proxies associated with an object storage service.
- the existence of multiple proxies may be ascertainable from outside the object storage service, or it may be transparent.
- Different proxies may be used for different purposes. For example, in one embodiment different proxies are used for different types of files. In another embodiment, different proxies are used for different types of requests.
- an appropriate proxy is chosen to minimize latency, geographic, or network distance between the proxy and the system making the request.
- FIG. 14 is a functional block diagram of a virtual machine (VM) image service 1400 according to various aspects of the current disclosure.
- the VM image service 1400 is an IaaS-style cloud computing system for registering, storing, and retrieving virtual machine images and associated metadata.
- the VM image service 1400 is deployed as a service resource 130 in the cloud computing system 110 ( FIG. 1 ).
- the service 1400 presents to clients of the cloud computing system 110 an endpoint from which the clients may store, lookup, and retrieve virtual machine images on demand.
- the VM image service 1400 comprises a component-based architecture that includes an API server 1402 , an image store 1404 , and a registry server 1406 .
- the API server 1402 is a communication hub that routes VM image requests and data between clients 1408 a - n and the image store 1404 and registry server 1406 .
- the image store 1404 is an autonomous and extensible storage resource that stores VM images managed by the service 1400
- the registry server 1406 is a componentized service that stores metadata associated with the managed VM images.
- the API server 1402 includes an “external” API endpoint 1410 through which the clients 1408 a - n may programmatically access VM images managed by the service 1400 .
- the API endpoint 1410 exposes both metadata about managed VM images and the image data itself to requesting clients.
- the API endpoint 1410 is implemented with an RPC-style system, such as CORBA, DCE/COM, SOAP, or XML-RPC, and adheres to the calling structure and conventions defined by these respective standards.
- the external API endpoint 1410 is a basic HTTP web service adhering to a representational state transfer (REST) style and may be identifiable via a URL. Specific functionality of the API endpoint 1410 will be described in greater detail below.
- the API server 1402 may be implemented in software or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that software is used to implement the API server 1402 , it may include software that is stored on a non-transitory computer-readable medium in an information processing system, such as the information processing system 210 of FIG. 2 .
- the VM image service 1400 stores VM images in the image store 1404 .
- the image store 1404 is any local or remote storage resource that is programmatically accessible by an “internal” API endpoint within the API server 1402 .
- the image store 1404 may simply be a file system storage 1412 a that is physically associated with the API server 1402 .
- the API server 1402 includes a file system API endpoint 1414 a that communicates natively with the file system storage 1412 a.
- the file system API endpoint 1414 a conforms to a standardized storage API for reading, writing, and deleting VM image data.
- the API server 1402 makes an internal API call to the file system API endpoint 1414 a, which, in turn, sends a read command to the file system storage.
- the image store 1404 may be implemented with an outside cloud storage 1412 b, object storage service storage 1412 c, and/or HTTP storage 1412 n that are respectively associated with an outside cloud service endpoint 1414 b, object storage service endpoint 1414 c, and HTTP endpoint 1414 n on the API server 1402 .
- the HTTP storage 1412 n may comprise a URL that points to a virtual machine image hosted somewhere on the Internet, and thus may be read-only.
- any number of additional storage resources such as Sheepdog, a RADOS block device (RBD), a storage area network (SAN), and any other programmatically accessible storage solutions, may be provisioned as the image store 1404 .
- additional storage resources such as Sheepdog, a RADOS block device (RBD), a storage area network (SAN), and any other programmatically accessible storage solutions.
- multiple storage resources may be simultaneously available as image stores within service 1400 such that the API server 1402 may select a specific storage option based on the size, availability requirements, etc of a VM image.
- the image store 1404 provides the image service 1400 with redundant, scalable, and/or distributed storage for VM images.
- the API server 1402 may include a local image cache that temporarily stores VM image data.
- the API server can distribute the VM image to the client without having to retrieve the image from the image store 1404 .
- Locally caching VM images on the API server not only decreases response time but it also enhances the scalability of the VM image service 1400 .
- the image service 1400 may include a plurality of API servers, where each may cache the same VM image and simultaneously distribute portions of the image to a client.
- the registry server 1406 stores and publishes VM image metadata corresponding to VM images stored by the system 1400 in the image store 1404 .
- each VM image managed by the service 1400 includes at least the following metadata properties stored in the registry server 1406 : UUID, name, status of the image, disk format, container format, size, public availability, and user-defined properties. Additional and/or different metadata may be associated with VM images in alternative embodiments.
- the registry server 1406 includes a registry database 1416 in which the metadata is stored.
- the registry database 1416 is a relational database such as MySQL, but, in other embodiments, it may be a non-relational structured data storage system like MongoDB, Apache Cassandra, or Redis.
- the registry server 1406 includes a registry API endpoint 1418 .
- the registry API endpoint 1418 is a RESTful API that programmatically exposes the database functions to the API server 1402 so that the API server may query, insert, and delete VM image metadata upon receiving requests from clients.
- the registry server 1406 may be any public or private web service that exposes the RESTful API to the API server 1402 .
- the registry server 1402 may be implemented on a dedicated information processing system of may be a software component stored on a non-transitory computer-readable medium in the same information processing system as the API server 1402 .
- clients 1408 a - n exploit the external API endpoint 1410 exposed by the API server 1402 to lookup, store, and retrieve VM images managed by the VM image service 1400 .
- clients may issue HTTP GETs, PUTs, POSTs, and HEADs to communicate with the API server 1402 .
- a client may issue a GET request to ⁇ API_server_URL>/images/ to retrieve the list of available public images managed by the image service 1400 .
- the API server Upon receiving the GET request from the client, the API server sends a corresponding HTTP GET request to the registry server 1406 .
- the registry server 1406 queries the registry database 1416 for all images with metadata indicating that they are public.
- the registry server 1406 returns the image list to the API server 1402 which forwards it on to the client.
- the client may receive a JSON-encoded mapping containing the following information: URI, name, disk_format, container format, and size.
- a client may retrieve a virtual machine image from the service 1400 by sending a GET request to ⁇ API_server_URL>/imageskimage_URI>.
- the API server 1404 retrieves the VM image data from the image store 1404 by making an internal API call to one of the storage API endpoints 1414 a - n and also requests the metadata associated with the image from the registry server 1406 .
- the API server 1402 returns the metadata to the client as a set of HTTP headers and the VM image as data encoded into the response body. Further, to store a VM image and metadata in the service 1400 , a client may issue a POST request to ⁇ API_server_URL>/images/ with the metadata in the HTTP header and the VM image data in the body of the request. Upon receiving the POST request, the API server 1402 issues a corresponding POST request to the registry API endpoint 1418 to store the metadata in the registry database 1416 and makes an internal API call to one of the storage API endpoints 1414 a - n to store the VM image in the image store 1404 .
- VM image service 1400 may be implemented in various other manners, such as through non-RESTful HTTP interactions, RPC-style communications, internal function calls, shared memory communication, or other communication mechanisms.
- the API server 1402 includes a plugin mechanism that allows for a user-configurable response to image requests.
- sensitive security information is held by a user outside the scope of the cloud computing system.
- the plugin creates a secure connection to a user-controlled endpoint and receives the necessary information from the user's system. The plugin then injects the custom authentication information into the image prior to the image being instantiated.
- Other embodiments can modify the image or the image request in arbitrary user-defined ways.
- the VM image service 1400 may include security features such as an authentication manager to authenticate and manage user, account, role, project, group, quota, and security group information associated with the managed VM images.
- an authentication manager may filter every request received by the API server 1402 to determine if the requesting client has permission to access specific VM images.
- Role-Based Access Control RBAC may be implemented in the context of the VM image service 1400 , whereby a user's roles defines the API commands that user may invoke. For example, certain API calls to the API server 1402 , such as POST requests, may be only associated with a specific subset of roles.
- VM image service 1400 may be shared between the cloud computing system and the VM image service, or they may be completely separate.
- controllers “nodes,” “servers,” “managers,” “VMs,” or similar terms are described relative to the VM image service 1400 , those can be understood to comprise any of a single information processing device 210 as described relative to FIG. 2 , multiple information processing devices 210 , a single VM as described relative to FIG. 2 , a group or cluster of VMs or information processing devices as described relative to FIG. 3 . These may run on a single machine or a group of machines, but logically work together to provide the described function within the system.
- FIG. 15 an IaaS-style computational cloud service (a “compute” service) is shown at 1500 according to one embodiment.
- This is one embodiment of a cloud controller 120 with associated cloud service 130 as described relative to FIG. 1 .
- the existence of a compute service does not require or prohibit the existence of other portions of the cloud computing system 110 nor does it require or prohibit the existence of other cloud controllers 120 with other respective services 130 .
- controllers that are similar to components of the larger cloud computing system 110 , those components may be shared between the cloud computing system 110 and the compute service 1500 , or they may be completely separate.
- controllers that are described relative to the compute service 1500 , those can be understood to comprise any of a single information processing device 210 as described relative to FIG. 2 , multiple information processing devices 210 , a single VM as described relative to FIG. 2 , a group or cluster of VMs or information processing devices as described relative to FIG. 3 . These may run on a single machine or a group of machines, but logically work together to provide the described function within the system.
- compute service 1500 includes an API Server 1510 , a Compute Controller 1520 , an Auth Manager 1530 , an Object Store 1540 , a Volume Controller 1550 , a Network Controller 1560 , and a Compute Manager 1570 . These components are coupled by a communications network of the type previously described. In one embodiment, communications between various components are message-oriented, using HTTP or a messaging protocol such as AMQP, ZeroMQ, or STOMP.
- compute service 1500 further includes distributed data store 1590 .
- Global state for compute service 1500 is written into this store using atomic transactions when required. Requests for system state are read out of this store.
- results are cached within controllers for short periods of time to improve performance.
- the distributed data store 1590 can be the same as, or share the same implementation as Object Store 800 discussed relative to FIG. 8 .
- the API server 1510 includes external API endpoints 1512 .
- the external API endpoints 1512 are provided over an RPC-style system, such as CORBA, DCE/COM, SOAP, or XML-RPC. These follow the calling structure and conventions defined in their respective standards.
- the external API endpoints 1512 are basic HTTP web services following a REST pattern and identifiable via URL. Requests to read a value from a resource are mapped to HTTP GETs, requests to create resources are mapped to HTTP PUTs, requests to update values associated with a resource are mapped to HTTP POSTs, and requests to delete resources are mapped to HTTP DELETEs.
- the API endpoints 1512 are provided via internal function calls, IPC (Inter-process Communication), or a shared memory mechanism. Regardless of how the API is presented, the external API endpoints 1512 are used to handle authentication, authorization, and basic command and control functions using various API interfaces. In one embodiment, the same functionality is available via multiple APIs, including APIs associated with other cloud computing systems. This enables API compatibility with multiple existing tool sets created for interaction with offerings from other vendors.
- the Compute Controller 1520 coordinates the interaction of the various parts of the compute service 1500 .
- the various internal services that work together to provide the compute service 1500 are internally decoupled by adopting a service-oriented architecture (SOA).
- SOA service-oriented architecture
- the Compute Controller 1520 serves as an internal API server, allowing the various internal controllers, managers, and other components to request and consume services from the other components.
- all messages pass through the Compute Controller 1520 .
- the Compute Controller 1520 brings up services and advertises service availability, but requests and responses go directly between the components making and serving the request.
- there is a hybrid model in which some services are requested through the Compute Controller 1520 , but the responses are provided directly from one component to another.
- communication to and from the Compute Controller 1520 is mediated via one or more internal API endpoints 1522 , provided in a similar fashion to those discussed above.
- the internal API endpoints 1522 differ from the external API endpoints 1512 in that the internal API endpoints 1522 advertise services only available within the overall compute service 1500 , whereas the external API endpoints 1512 advertise services available outside the compute service 1500 .
- the Compute Controller 1520 includes an instruction processor 1524 for receiving and processing instructions associated with directing the compute service 1500 . For example, in one embodiment, responding to an API call involves making a series of coordinated internal API calls to the various services available within the compute service 1500 , and conditioning later API calls on the outcome or results of earlier API calls.
- the instruction processor 1524 is the component within the Compute Controller 1520 responsible for marshalling arguments, calling services, and making conditional decisions to respond appropriately to API calls.
- the instruction processor 1524 is implemented as described above relative to FIG. 3 , specifically as a tailored electrical circuit or as software instructions to be used in conjunction with a hardware processor to create a hardware-software combination that implements the specific functionality described herein.
- those instructions may include software that is stored on a computer-readable medium.
- one or more embodiments have associated with them a buffer.
- the buffer can take the form of data structures, a memory, a computer-readable medium, or an off-script-processor facility.
- one embodiment uses a language runtime as an instruction processor 1524 , running as a discrete operating environment, as a process in an active operating environment, or can be run from a low-power embedded processor.
- the instruction processor 1524 takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs.
- the instruction processor 1524 is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor.
- the instruction processor includes a rule engine as a submodule as described herein.
- the Compute Controller 1520 includes a message queue as provided by message service 1526 .
- the various functions within the compute service 1500 are isolated into discrete internal services that communicate with each other by passing data in a well-defined, shared format, or by coordinating an activity between two or more services. In one embodiment, this is done using a message queue as provided by message service 1526 .
- the message service 1526 brokers the interactions between the various services inside and outside the Compute Service 1500 .
- the message service 1526 is implemented similarly to the message service described relative to FIGS. 6 a - 6 c.
- the message service 1526 may use the message service 140 directly, with a set of unique exchanges, or may use a similarly configured but separate service.
- the Auth Manager 1530 provides services for authenticating and managing user, account, role, project, group, quota, and security group information for the compute service 1500 .
- every call is necessarily associated with an authenticated and authorized entity within the system, and so is or can be checked before any action is taken.
- internal messages are assumed to be authorized, but all messages originating from outside the service are suspect.
- the Auth Manager checks the keys provided associated with each call received over external API endpoints 1512 and terminates and/or logs any call that appears to come from an unauthenticated or unauthorized source.
- the Auth Manager 1530 is also used for providing resource-specific information such as security groups, but the internal API calls for that information are assumed to be authorized.
- Auth Manager 1530 is implemented similarly to or as a proxy to authn/authz service 700 as described relative to FIG. 7 .
- external communication to and from the Auth Manager 1530 is mediated via one or more authentication and authorization API endpoints 1532 , provided in a similar fashion to those discussed above.
- the authentication and authorization API endpoints 1532 differ from the external API endpoints 1512 in that the authentication and authorization API endpoints 1532 are only used for managing users, resources, projects, groups, and rules associated with those entities, such as security groups, RBAC roles, etc.
- the authentication and authorization API endpoints 1532 are provided as a subset of external API endpoints 1512 .
- the Auth Manager 1530 includes a rules processor 1534 for processing the rules associated with the different portions of the compute service 1500 . In one embodiment, this is implemented in a similar fashion to the instruction processor 1524 described above.
- the Object Store 1540 provides redundant, scalable object storage capacity for arbitrary data used by other portions of the compute service 1500 .
- the Object Store 1540 can be implemented one or more block devices exported over the network.
- the Object Store 1540 is implemented as a structured and possibly distributed data organization system. Examples include relational database systems—both standalone and clustered—as well as non-relational structured data storage systems like MongoDB, Apache Cassandra, or Redis.
- the Object Store 1540 is implemented as a redundant, eventually consistent, fully distributed data storage service, such as the object storage service 800 described relative to FIG. 8 .
- external communication to and from the Object Store 1540 is mediated via one or more object storage API endpoints 1542 , provided in a similar fashion to those discussed above.
- the object storage API endpoints 1542 are internal APIs only.
- the Object Store 1540 is provided by a separate cloud service 130 , so the “internal” API used for compute service 1500 is the same as the external API provided by the object storage service itself.
- the Object Store 1540 includes an Image Service 1544 .
- the Image Service 1544 is a lookup and retrieval system for virtual machine images.
- various virtual machine images can be associated with a unique project, group, user, or name and stored in the Object Store 1540 under an appropriate key. In this fashion multiple different virtual machine image files can be provided and programmatically loaded by the compute service 1500 .
- the image service 1544 is implemented as image service 1400 , as described relative to FIG. 14 .
- the Volume Controller 1550 coordinates the provision of block devices for use and attachment to virtual machines.
- the Volume Controller 1550 includes Volume Workers 1552 .
- the Volume Workers 1552 are implemented as unique virtual machines, processes, or threads of control that interact with one or more backend volume providers 1554 to create, update, delete, manage, and attach one or more volumes 1556 to a requesting VM.
- the Volume Controller 1550 is implemented using a SAN that provides a sharable, network-exported block device that is available to one or more VMs, using a network block protocol such as iSCSI.
- the Volume Workers 1552 interact with the SAN to manage and iSCSI (Internet Small Computer System Interface) storage to manage LVM-based (Logical Volume Manager-based) instance volumes, stored on one or more smart disks or independent processing devices that act as volume providers 1554 using their embedded storage 1556 .
- disk volumes 1556 are stored in the Object Store 1540 as image files under appropriate keys.
- the Volume Controller 1550 interacts with the Object Store 1540 to retrieve a disk volume 1556 and place it within an appropriate logical container on the same information processing system 240 that contains the requesting VM.
- An instruction processing module acting in concert with the instruction processor and hypervisor on the information processing system 240 acts as the volume provider 1554 , managing, mounting, and unmounting the volume 1556 on the requesting VM.
- the same volume 1556 may be mounted on two or more VMs, and a block-level replication facility may be used to synchronize changes that occur in multiple places.
- the Volume Controller 1550 acts as a block-device proxy for the Object Store 1540 , and directly exports a view of one or more portions of the Object Store 1540 as a volume.
- the volumes are simply views onto portions of the Object Store 1540
- the Volume Workers 1554 are part of the internal implementation of the Object Store 1540 .
- the Network Controller 1560 manages the networking resources for VM hosts managed by the compute manager 1570 . Messages received by Network Controller 1560 are interpreted and acted upon to create, update, and manage network resources for compute nodes within the compute service, such as allocating fixed IP addresses, configuring VLANs for projects or groups, or configuring networks for compute nodes.
- the Network Controller 1560 is implemented similarly to the network service 400 described relative to FIG. 4 .
- the network controller 1560 may use a shared cloud controller directly, with a set of unique addresses, identifiers, and routing rules, or may use a similarly configured but separate service.
- a shared cloud controller directly, with a set of unique addresses, identifiers, and routing rules, or may use a similarly configured but separate service.
- the Compute Manager 1570 manages computing instances for use by API users using the compute service 1500 .
- the Compute Manager 1570 is coupled to a plurality of resource pools 1572 , each of which includes one or more compute nodes 1574 .
- Each compute node 1574 is a virtual machine management system as described relative to FIG. 3 and includes a compute worker 1576 , a module working in conjunction with the hypervisor and instruction processor to create, administer, and destroy multiple user- or system-defined logical containers and operating environments—VMs—according to requests received through the API.
- the compute nodes may be organized into clusters, such as clusters 1572 a and 1572 b.
- each resource pool 1572 is physically located in one or more data centers in one or more different locations.
- resource pools have different physical or software resources, such as different available hardware, higher-throughput network connections, or lower latency to a particular location.
- the Compute Manager 1570 allocates VM images to particular compute nodes 1574 via a Scheduler 1578 .
- the Scheduler 1578 is a matching service; requests for the creation of new VM instances come in and the most applicable Compute nodes 1574 are selected from the pool of potential candidates.
- the Scheduler 1578 selects a compute node 1574 using a random algorithm. Because the node is chosen randomly, the load on any particular node tends to be non-coupled and the load across all resource pools tends to stay relatively even.
- a smart scheduler 1578 is used.
- a smart scheduler analyzes the capabilities associated with a particular resource pool 1572 and its component services to make informed decisions on where a new instance should be created. When making this decision it consults not only all the Compute nodes across the resource pools 1572 until the ideal host is found.
- a distributed scheduler 1578 is used.
- a distributed scheduler is designed to coordinate the creation of instances across multiple compute services 1500 . Not only does the distributed scheduler 1578 analyze the capabilities associated with the resource pools 1572 available to the current compute service 1500 , it also recursively consults the schedulers of any linked compute services until the ideal host is found.
- either the smart scheduler or the distributed scheduler is implemented using rules engine 1579 and a series of associated rules regarding costs and weights associated with desired compute node characteristics.
- rules engine 1579 compares a Weighted Cost for each node.
- the Weighting is just the sum of the total Costs.
- a Weighting is calculated using an exponential or polynomial algorithm.
- costs are nothing more than integers along a fixed scale, although costs can also be represented by floating point numbers, vectors, or matrices. Costs are computed by looking at the various Capabilities of the available node relative to the specifications of the Instance being requested. The costs are calculated so that a “good” match has lower cost than a “bad” match, where the relative goodness of a match is determined by how closely the available resources match the requested specifications.
- specifications can be hierarchical, and can include both hard and soft constraints.
- a hard constraint is a constraint that cannot be violated and have an acceptable response. This can be implemented by having hard constraints be modeled as infinite-cost requirements.
- a soft constraint is a constraint that is preferable, but not required. Different soft constraints can have different weights, so that fulfilling one soft constraint may be more cost-effective than another. Further, constraints can take on a range of values, where a good match can be found where the available resource is close, but not identical, to the requested specification. Constraints may also be conditional, such that constraint A is a hard constraint or high-cost constraint if Constraint B is also fulfilled, but can be low-cost if Constraint C is fulfilled.
- the constraints are implemented as a series of rules with associated cost functions. These rules can be abstract, such as preferring nodes that don't already have an existing instance from the same project or group.
- Other constraints may include: a node with available GPU hardware; a node with an available network connection over 100 Mbps; a node that can run Windows instances; a node in a particular geographic location, etc.
- the constraints are computed to select the group of possible nodes, and then a weight is computed for each available node and for each requested instance. This allows large requests to have dynamic weighting; if 1000 instances are requested, the consumed resources on each node are “virtually” depleted so the Cost can change accordingly.
- the scheduler can be directly accessed or modified using an API and/or a plugin interface.
- a plugin is implemented as a series of loadable and executable rules that can be provided to the compute service 1500 and run by the scheduler as part of its weighting or allocation process.
- this loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module.
- FIG. 16 a diagram showing one embodiment of the process of instantiating and launching a VM instance is shown as diagram 1600 .
- the implementation of the image instantiating and launching process will be shown in a manner consistent with the embodiment of the compute service 1500 as shown relative to FIG. 15 , the process is not limited to the specific functions or elements shown in FIG. 15 .
- internal details not relevant to diagram 1600 have been removed from the diagram relative to FIG. 15 .
- the messages are sent via a message service, such as message service 1526 as described relative to FIG. 15 .
- the API Server 1510 receives a request to create and run an instance with the appropriate arguments. In one embodiment, this is done by using a command-line tool that issues arguments to the API server 1510 . In a second embodiment, this is done by sending a message to the API Server 1510 .
- the API to create and run the instance includes arguments specifying a resource type, a resource image, and control arguments. A further embodiment includes requester information and is signed and/or encrypted for security and privacy.
- API server 1510 accepts the message, examines it for API compliance, and relays a message to Compute Controller 1520 , including the information needed to service the request.
- the Compute Controller 1520 sends a message to Auth Manager 1530 to authenticate and authorize the request at time 1606 and Auth Manager 1530 sends back a response to Compute Controller 1520 indicating whether the request is allowable at time 1608 . If the request is allowable, a message is sent to the Compute Manager 1570 to instantiate the requested resource at time 1610 . At time 1612 , the Compute Manager selects a Compute Worker 1576 and sends a message to the selected Worker to instantiate the requested resource.
- Compute Worker identifies and interacts with Network Controller 1560 to get a proper VLAN and IP address as described in steps 451 - 457 relative to FIG. 4 .
- the selected Worker 1576 interacts with the Object Store 1540 and/or the Image Service 1544 to locate and retrieve an image corresponding to the requested resource. If requested via the API, or used in an embodiment in which configuration information is included on a mountable volume, the selected Worker interacts with the Volume Controller 1550 at time 1618 to locate and retrieve a volume for the to-be-instantiated resource.
- the selected Worker 1576 uses the available virtualization infrastructure as described relative to FIG. 2 to instantiate the resource, mount any volumes, and perform appropriate configuration.
- selected Worker 1576 interacts with Network Controller 1560 to configure routing as described relative to step 460 as discussed relative to FIG. 4 .
- a message is sent back to the Compute Controller 1520 via the Compute Manager 1570 indicating success and providing necessary operational details relating to the new resource.
- a message is sent back to the API Server 1510 with the results of the operation as a whole.
- the API-specified response to the original command is provided from the API Server 1510 back to the originally requesting entity. If at any time a requested operation cannot be performed, then an error is returned to the API Server at time 1690 and the API-specified response to the original command is provided from the API server at time 1692 . For example, an error can be returned if a request is not allowable at time 1608 , if a VLAN cannot be created or an IP allocated at time 1614 , if an image cannot be found or transferred at time 1616 , etc.
- the foregoing text describes the workings of a cloud infrastructure that utilizes a physical resource pool to run virtualized resources for use within the cloud computing system.
- the present specification discloses a multi-level cloud computing system wherein a first cloud infrastructure includes a first set of virtualized resources run by an underlying physical resource pool, and a second cloud infrastructure utilizes the first set of virtualized resources to run a second set of virtualized resources.
- a multi-level cloud computing system will now be described in more detail.
- FIG. 17 is a diagram showing a multi-level cloud computing system 1700 .
- the multi-level cloud computing system 1700 includes a physical resource pool 1702 , a first cloud infrastructure 1706 , a first set of virtualized resources 1712 , a second cloud infrastructure 1716 , and a second set of virtualized resources 1722 .
- the physical resource pool 1702 includes a number of information processing devices 1704 a, 1704 b. These information processing devices 1704 may be similar to the information processing device illustrated in FIG. 2 . Thus, the information processing devices 1704 are configured to host one or more virtual machines. In one embodiment, the information processing devices 1704 comprise a number of computing systems within a data center. The number of information processing devices 1704 within the physical resource pool 1702 may vary depending on the expected purposes of the multi-level cloud computing system 1700 . For various purposes, including energy efficiency and security, the information processing devices 1704 not currently being utilized are either shut down or in a sleep/standby mode. The information processing systems 1704 that are not shut down or on standby are used to provide physical resources to the first cloud infrastructure 1706 .
- the first cloud infrastructure 1706 is a cloud infrastructure similar to that illustrated in FIG. 1 .
- the first cloud infrastructure 1706 includes at least one cloud controller, which will be referred to for illustrative purposes as the first cloud controller 1708 .
- the first cloud infrastructure 1706 includes a number of cloud services 1710 a, 1710 b.
- these cloud services may include, for example, an IaaS-level compute service, a PaaS-level object storage service, a SaaS-level queue service, a DNS service, and a database service.
- the first cloud infrastructure 1706 provides a first set of virtualized resources 1712 with access to the underlying physical resource pool 1702 .
- resources within the first set of virtualized resources 1712 will be referred to as primary virtualized resources 1714 .
- a virtualized resource 1714 may include a virtual information processing device, a logical container, an operating environment.
- a virtualized resource may also include network services and storage services.
- a virtual resource may also include an emulator for any type of physical computing device.
- a virtualized resource is provisioned using a process similar or identical to the process discussed relative to FIG. 16 .
- a virtualized resource can vary based upon need.
- an underlying information processing device with 16 GB of RAM and 16 processor cores can be divided into sixteen, eight, four, two, or one single virtualized computing device, each with a proportionate share of the underlying resources.
- a virtualized computing device can be instantiated that has a higher or lower number of VNICs, access (shared or exclusive) to specialized processing hardware, access to high-performance storage, etc. In this way a virtualized resource can be customized to meet various usage scenarios with better or more predictable performance.
- the first set of virtualized resources 1712 is used to support a second cloud infrastructure 1716 .
- the second cloud infrastructure 1716 utilizes the primary virtualized resources 1714 of the first cloud infrastructure 1706 as support.
- the second cloud infrastructure 1716 has access to the physical resource pool 1702 through the first cloud infrastructure 1706 .
- the first cloud infrastructure 1706 is used to bootstrap the second cloud infrastructure 1716 .
- the bootstrap process for a cloud computing system can be complicated as various physical machines need to be started up in particular sequences. This process may typically require manual coordination of the initial cloud infrastructure. But, by having the first cloud infrastructure 1706 bootstrap the second cloud infrastructure 1716 , the bootstrap process is simpler and does not require manual startup or provisioning of additional physical machines. Virtual resources can be efficiently started up as necessary in order to provide the appropriate support for the second cloud infrastructure 1716 .
- the second cloud infrastructure 1716 is also similar to the cloud infrastructure illustrated in FIG. 1 .
- the second cloud infrastructure 1716 includes at least one cloud controller, which will be referred to for illustrative purposes as the second cloud controller 1718 .
- the second cloud infrastructure 1716 includes a number of cloud services 1720 a, 1720 b.
- these cloud services may include, for example, an IaaS-level compute service, a PaaS-level object storage service, a SaaS-level queue service, a DNS service, and a database service.
- the second cloud infrastructure 1716 provides a second set of virtualized resources 1722 with access to the first cloud infrastructure 1706 , which in turn provides access to the underlying physical resource pool 1702 .
- resources within the second set of virtualized resources 1722 will be referred to as secondary virtualized resources 1724 .
- a virtualized resource 1722 may include a virtual information processing device, a logical container, an operating environment.
- a virtualized resource may also include network services and storage services.
- the first cloud infrastructure 1706 may include a cluster as described above in the text accompanying FIG. 3 .
- the first cloud infrastructure 1706 may be referred to as a parent cluster while the second cloud infrastructure 1716 may be referred to as a child cluster.
- the secondary virtualized resources 1724 may support further cloud infrastructures.
- Both the first cloud infrastructure 1706 and the second cloud infrastructure 1716 may exhibit characteristics described and explained above and illustrated in FIGS. 1-16 .
- the second cloud structure 1716 utilizes virtual resources while the first cloud infrastructure 1706 utilizes the physical resources of the physical resource pool 1702 .
- the first cloud infrastructure may also utilize various types of virtual resources.
- the second cloud infrastructure 1716 ultimately accesses the underlying physical resources through the first cloud infrastructure 1706 .
- This two-level infrastructure is able to virtualize itself and boot a second cloud service (or set of cloud services) on top of the underlying first cloud infrastructure.
- An exemplary process for performing this function will be described relative to an IaaS-style compute service as described above, but any of the services described herein may be bootstrapped in a similar fashion.
- this is performed by creating a first set of virtual machine images that include the software needed to run a compute controller (like compute controller 1520 ), an auth manager (like auth manager 1530 ), a network controller (like network controller 1580 ), a volume controller (like volume controller 1550 ), a compute manager (like compute manager 1570 ), and an API server (like API server 1510 ).
- These images have configuration information that in most cases only refers to the second cloud infrastructure 1716 ; the configuration information is independent of the first cloud infrastructure 1706 .
- the compute manager configuration would include information about the same underlying physical resource pool 1702 .
- a portion of the physical resource pool 1702 is reserved for the exclusive use of the first cloud infrastructure 1706 .
- a network controller virtualized resource 1714 is instantiated using the network controller image from the image service (or object service) associated with the first cloud infrastructure 1706 within the first set of virtualized resources 1712 .
- the network controller creates a virtual network for the provision of the new services and sets itself up to provide configuration information as a boot server, DHCP server, DNS server, etc.
- the remaining essential services (queues, controllers, servers) 1718 are instantiated as virtualized resources 1714 from the first set of virtualized resources 1712 using the images described above.
- the API server starts accepting requests from outside to create user-directed virtualized resources 1724 within the second set of virtualized resources 1722 .
- the services of either the first cloud infrastructure 1706 or the second cloud infrastructure 1716 include an API server, a compute controller, a compute manager, a network controller, and a volume controller, and an image or object store may be shared between the first and second cloud infrastructures 1706 and 1716 . These services are sufficient to allow for the creation of new virtual machine instances. In some cases, the services of either cloud infrastructure may include additional services such as an authorization service. Thus it is not necessary for the second cloud infrastructure 1716 to be completely separate from the first cloud infrastructure 1706 . For example, a volume controller (one type of cloud controller 1708 ) could provide volume services to both the first cloud infrastructure 1706 , the second cloud infrastructure 1716 , and to user-directed resources 1724 .
- a second-post-bootstrap cloud service 1720 b is created in the second cloud infrastructure 1716 to serve user requests.
- cloud service 1720 b could be an image or object service provided only for user requests, separate from the image or object service used to store the virtual machine images used to instantiate the second cloud infrastructure 1716 .
- the first cloud infrastructure 1706 is designed to support multiple secondary cloud infrastructures. Particularly, a number of secondary cloud infrastructures similar to the second cloud infrastructure 1716 may utilize the primary virtualized resources 1714 . In one example, each different secondary cloud infrastructure supported by the primary virtualized resources 1714 may have a different operator. Particularly, one entity may control the first cloud infrastructure 1706 and the primary virtualized resources 1714 associated therewith. Additionally, different entities may control the various secondary cloud infrastructures that access the physical resource pool 1702 through the first cloud infrastructure 1706 .
- a multi-level cloud computing system allows for more efficient scaling. Based on the demand for resources made by a user, a cloud computing system provisions a number of resources. In response to an increased demand made by the user, the cloud computing system has to provision additional resources. Typically, in a standard single level cloud, the cloud infrastructure has to spin up more virtualized resources which often requires provisioning more physical information processing devices. As physical information processing devices are relatively slow to start up, the process of scaling up or down can be complicated and time-consuming.
- an increased demand for resources made by a user on the second cloud infrastructure 1716 can be handled by having the first cloud infrastructure 1706 start up additional primary virtualized resources 1714 for use by the second cloud infrastructure 1724 .
- the virtualized resources can be provisioned or torn down much faster than a physical resource can be booted up or shut down.
- the multi-level cloud system provides more efficient response to changes in demand.
- FIG. 18 is a flow chart showing a process 1800 performed by a monitoring service 1802 .
- the monitoring service 1802 may be implemented in software, as a subsidiary information processing system, or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein.
- the monitoring service 1802 monitors 1804 the demand for services made by a user on the second cloud infrastructure 1716 .
- the monitoring service 1802 also monitors 1804 the capacity of the primary virtualized resources 1714 that support that second cloud infrastructure 1716 .
- This capacity may be, for example, a computational capacity, a storage capacity, or a network capacity.
- the monitoring service determines 1806 whether there is a discrepancy between the demand for services and capacity for services. If there is no discrepancy ( 1806 , NO), then no action is taken. If, however, there is a discrepancy ( 1806 , YES), then the monitoring service 1802 then determines 1808 whether the demand is sufficiently greater than or sufficiently less than the capacity. If the demand for services is sufficiently greater than the capacity ( 1808 , YES), then the monitoring system reports this to the first cloud controller 1708 . The first cloud controller 1708 of the first cloud infrastructure 1706 can then create 1810 an additional primary virtualized resource 1714 .
- the discrepancy is that the demand is sufficiently less than the capacity. This will be reported to the first cloud controller 1708 .
- the first cloud controller 1708 can then tear down 1812 any primary virtual resources 1714 not being used.
- the demand is sufficiently greater or less than the capacity if a change in the number or virtual resources is required under a particular policy.
- the monitoring service 1802 may determine that the demand for computational capacity is far less than the computational capacity currently being provided to the second cloud infrastructure 1716 by the primary virtual resources 1714 . In such a case, the resources used to support that virtual computational capacity may be better utilized elsewhere. Thus, upon being notified by the monitoring service 1802 of such a situation, the first cloud controller 1708 can tear down a number of virtual computational services not currently being used.
- the monitoring service 1802 may determine that the demand for storage capacity on the second cloud infrastructure 1716 is greater than the storage capacity currently being provided by the underlying primary virtual resources 1714 of the first cloud infrastructure 1706 . The monitoring service 1802 then reports this to the first cloud controller 1708 . The first cloud controller 1708 can then cause one or more additional virtual storage services to be started up. As these additional resources are virtual, they can be started up relatively quickly.
- the monitoring service 1802 may also be used to detect any virtual resources that are no longer being provided. For example, if the second cloud infrastructure 1716 is utilizing a particular primary virtual resource 1714 , and for whatever reason that primary virtual resource 1714 gets shut down, the monitoring service 1802 can report that to the first cloud controller 1708 . The first cloud controller 1708 can then quickly start up a replacement primary virtual resource 1714 relatively quickly.
- FIG. 19 is a flow chart showing an illustrative method 1900 for operating a multi-level cloud computing system.
- the method includes, with a first cloud controller for a first cloud infrastructure, providing 1902 a number of primary virtualized resources access to a physical resource pool, the physical resource pool comprising a number of information processing devices, each information processing device comprising a processor, a computer-readable medium, and a network interface.
- the method further includes, with a second cloud controller for a second cloud infrastructure comprising the primary virtualized resources, providing 1904 a number of secondary virtualized resources access to the physical resource pool through the primary virtualized resources.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Mathematical Physics (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Stored Programmes (AREA)
- Hardware Redundancy (AREA)
Abstract
A cloud computing system includes a physical resource pool that includes a number of information processing devices. Each information processing device includes a processor, a computer-readable medium, and a network interface. The system further includes a first cloud controller to manage a first cloud infrastructure, the first cloud infrastructure operating a first set of virtualized resources, the first set of virtualized resources having access to the physical resource pool through the first cloud controller. The system further includes a second cloud controller to manage a second cloud infrastructure, the second cloud infrastructure utilizing the first set of virtual resources to operate a second set of virtual resources, the second set of virtual resources being provided access to the physical resource pool through the second cloud controller and the first cloud controller.
Description
- The present application is a continuation of U.S. patent application Ser. No. 13/590,944, filed on Aug. 21, 2012, the entirety of which is incorporated herein by reference.
- The present disclosure relates generally to cloud computing, and more particularly to a customizable multi-vendor, multi-tenant cloud computing system.
- Cloud computing services can provide computational capacity, data access, networking/routing and storage services via a large pool of shared resources operated by a cloud computing provider. Because the computing resources are delivered over a network, cloud computing is location-independent computing, with all resources being provided to end-users on demand with control of the physical resources separated from control of the computing resources.
- Originally the term cloud came from a diagram that contained a cloud-like shape to contain the services that afforded computing power that was harnessed to get work done. Much like the electrical power we receive each day, cloud computing is a model for enabling access to a shared collection of computing resources - networks for transfer, servers for storage, and applications or services for completing work. More specifically, the term “cloud computing” describes a consumption and delivery model for IT services based on the Internet, and it typically involves over-the-Internet provisioning of dynamically scalable and often virtualized resources. This frequently takes the form of web-based tools or applications that users can access and use through a web browser as if it was a program installed locally on their own computer. Details are abstracted from consumers, who no longer have need for expertise in, or control over, the technology infrastructure “in the cloud” that supports them. Most cloud computing infrastructures consist of services delivered through common centers and built on servers. Clouds often appear as single points of access for consumers' computing needs, and do not require end-user knowledge of the physical location and configuration of the system that delivers the services.
- The utility model of cloud computing is useful because many of the computers in place in data centers today are underutilized in computing power and networking bandwidth. People may briefly need a large amount of computing capacity to complete a computation for example, but may not need the computing power once the computation is done. The cloud computing utility model provides computing resources on an on-demand basis with the flexibility to bring it up or down through automation or with little intervention.
- As a result of the utility model of cloud computing, there are a number of aspects of cloud-based systems that can present challenges to existing application infrastructure. First, clouds should enable self-service, so that users can provision servers and networks with little human intervention. Second, network access; because computational resources are delivered over the network, the individual service endpoints need to be network-addressable over standard protocols and through standardized mechanisms. Third, multi-tenancy. Clouds are designed to serve multiple consumers according to demand, and it is important that resources be shared fairly and that individual users not suffer performance degradation. Fourth, elasticity. Clouds are designed for rapid creation and destruction of computing resources, typically based upon virtual containers. Provisioning these different types of resources must be rapid and scale up or down based on need. Further, the cloud itself as well as applications that use cloud computing resources must be prepared for impermanent, fungible resources; application or cloud state must be explicitly managed because there is no guaranteed permanence of the infrastructure. Fifth, clouds typically provide metered or measured service—like utilities that are paid for by the hour, clouds should optimize resource use and control it for the level of service or type of servers such as storage or processing.
- Cloud computing offers different service models depending on the capabilities a consumer may require, including SaaS, PaaS, and IaaS-style clouds. SaaS (Software as a Service) clouds provide the users the ability to use software over the network and on a distributed basis. SaaS clouds typically do not expose any of the underlying cloud infrastructure to the user. PaaS (Platform as a Service) clouds provide users the ability to deploy applications through a programming language or tools supported by the cloud platform provider. Users interact with the cloud through standardized APIs (Application Programming Interfaces), but the actual cloud mechanisms are abstracted away. Finally, IaaS (Infrastructure as a Service) clouds provide computer resources that mimic physical resources, such as computer instances, network connections, and storage devices. The actual scaling of the instances may be hidden from the developer, but users are required to control the scaling infrastructure.
- One way in which different cloud computing systems may differ from each other is in how they deal with control of the underlying hardware and privacy of data. The different approaches are sometimes referred to a “public clouds,” “private clouds,” “hybrid clouds,” and “multi-vendor clouds.” A public cloud has an infrastructure that is available to the general public or a large industry group and is likely owned by a cloud services company. A private cloud operates for a single organization, but can be managed on-premise or off-premise. A hybrid cloud can be a deployment model, as a composition of both public and private clouds, or a hybrid model for cloud computing may involve both virtual and physical servers. A multi-vendor cloud is a hybrid cloud that may involve multiple public clouds, multiple private clouds, or some mixture.
- Because the flow of services provided by the cloud is not directly under the control of the cloud computing provider, cloud computing requires the rapid and dynamic creation and destruction of computational units, frequently realized as virtualized resources. Maintaining the reliable flow and delivery of dynamically changing computational resources on top of a pool of limited and less-reliable physical servers provides unique challenges. Accordingly, it is desirable to provide a better-functioning cloud computing system with superior operational capabilities.
-
FIG. 1 is a schematic view illustrating an external view of a cloud computing system. -
FIG. 2 is a schematic view illustrating an information processing system as used in various embodiments. -
FIG. 3 is a virtual machine management system as used in various embodiments. -
FIG. 4 is an IaaS-style network service according to various embodiments. -
FIG. 5a is a diagram showing types of network access available to virtual machines in a cloud computing system according to various embodiments. -
FIG. 5b is a flow chart showing the establishment of a VLAN for a project according to various embodiments. -
FIG. 6a shows a message service system according to various embodiments. -
FIG. 6b is a diagram showing how a directed message is sent using the message service according to various embodiments. -
FIG. 6c is a diagram showing how a broadcast message is sent using the message service according to various embodiments. -
FIG. 7 is a PaaS-style identity and authentication service according to various embodiments. -
FIG. 8 is a PaaS-style object storage service according to various embodiments. -
FIG. 9 is a schematic view illustrating an embodiment of a logical structure provided by the object storage service according to various embodiments. -
FIG. 10 is a schematic view of a user account storage structure according to one embodiment. -
FIG. 11 is a flow chart illustrating a method for storing an object according to one embodiment. -
FIG. 12 is a flow chart illustrating an embodiment of a method for creating a ring according to one embodiment. -
FIG. 13 is a flow chart illustrating an embodiment of a method for reassigning partitions in a ring according to one embodiment. -
FIG. 14 is a PaaS-style image service according to various embodiments. -
FIG. 15 is an IaaS-style computational cloud service according to various embodiments. -
FIG. 16 is an instantiating and launching process for virtual resources according to various embodiments. -
FIG. 17 is a multi-level cloud infrastructure, according to various embodiments. -
FIG. 18 is a flow chart showing a method for responding to changes in demand for cloud services, according to various embodiments. -
FIG. 19 is a flow chart showing a method for operating a multi-level cloud infrastructure, according to various embodiments. - The following disclosure has reference to computing services delivered on top of a cloud architecture.
- Referring now to
FIG. 1 , an external view of one embodiment of acloud computing system 110 is illustrated. Thecloud computing system 110 includes auser device 102 connected to anetwork 104 such as, for example, a Transport Control Protocol/Internet Protocol (TCP/IP) network (e.g., the Internet.) Theuser device 102 is coupled to thecloud computing system 110 via one ormore service endpoints 112. Depending on the type of cloud service provided, these endpoints give varying amounts of control relative to the provisioning of resources within thecloud computing system 110. For example,SaaS endpoint 112 a will typically only give information and access relative to the application running on the cloud storage system, and the scaling and processing aspects of the cloud computing system will be obscured from the user. PaaS endpoint 112 b will typically give an abstract API that allows developers to declaratively request or command the backend storage, computation, and scaling resources provided by the cloud, without giving exact control to the user. IaaS endpoint 112 c will typically provide the ability to directly request the provisioning of resources, such as computation units (typically virtual machines), software-defined or software-controlled network elements like routers, switches, domain name servers, etc., file or object storage facilities, authorization services, database services, queue services and endpoints, etc. In addition, users interacting with an IaaS cloud are typically able to provide virtual machine images that have been customized for user-specific functions. This allows thecloud computing system 110 to be used for new, user-defined services without requiring specific support. - It is important to recognize that the control allowed via an IaaS endpoint is not complete. Within the
cloud computing system 110 are one more cloud controllers 120 (running what is sometimes called a “cloud operating system”) that work on an even lower level, interacting with physical machines, managing the contradictory demands of the multi-tenantcloud computing system 110. The workings of the cloud controllers 120 are typically not exposed outside of thecloud computing system 110, even in an IaaS context. In one embodiment, the commands received through one of theservice endpoints 112 are then routed via one or moreinternal networks 114. Theinternal network 114 couples the different services to each other. Theinternal network 114 may encompass various protocols or services, including but not limited to electrical, optical, or wireless connections at the physical layer; Ethernet, Fibre channel, ATM (Asynchronous Transfer Mode), and SONET (Synchronous Optical Networking) at the MAC (Media Access Control) layer; TCP, UDP (User Datagram Protocol), ZeroMQ or other services at the connection layer; and XMPP (Extensible Messaging and Presence Protocol), HTTP (HyperText Transfer Protocol), AMQP (Advanced Message Queuing Protocol), STOMP (Streaming Text Oriented Messaging Protocol), SMS (Short Message Service), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), or other standards at the protocol layer. Theinternal network 114 is typically not exposed outside the cloud computing system, except to the extent that one or morevirtual networks 116 may be exposed that control the internal routing according to various rules. Thevirtual networks 116 typically do not expose as much complexity as may exist in the actualinternal network 114; but varying levels of granularity can be exposed to the control of the user, particularly in IaaS services. - In one or more embodiments, it may be useful to include various processing or routing nodes in the network layers 114 and 116, such as proxy/
gateway 118. Other types of processing or routing nodes may include switches, routers, switch fabrics, caches, format modifiers, or correlators. These processing and routing nodes may or may not be visible to the outside. It is typical that one level of processing or routing nodes may be internal only, coupled to theinternal network 114, whereas other types of network services may be defined by or accessible to users, and show up in one or morevirtual networks 116. Either of theinternal network 114 or thevirtual networks 116 may be encrypted or authenticated according to the protocols and services described below. - In various embodiments, one or more parts of the
cloud computing system 110 may be disposed on a single host. Accordingly, some of the “network” layers 114 and 116 may be composed of an internal call graph, inter-process communication (IPC), or a shared memory communication system. - Once a communication passes from the endpoints via a
network layer processing devices 118, it is received by one or more applicable cloud controllers 120. The cloud controllers 120 are responsible for interpreting the message and coordinating the performance of the necessary corresponding services, returning a response if necessary. Although the cloud controllers 120 may provide services directly, more typically the cloud controllers 120 are in operative contact with the service resources 130 necessary to provide the corresponding services. For example, it is possible for different services to be provided at different levels of abstraction. For example, a “compute”service 130 a may work at an IaaS level, allowing the creation and control of user-defined virtual computing resources. In the samecloud computing system 110, a PaaS-levelobject storage service 130 b may provide a declarative storage API, and a SaaS-level Queue service 130 c,DNS service 130 d, orDatabase service 130 e may provide application services without exposing any of the underlying scaling or computational resources. Other services are contemplated as discussed in detail below. - In various embodiments, various cloud computing services or the cloud computing system itself may require a message passing system. The
message routing service 140 is available to address this need, but it is not a required part of the system architecture in at least one embodiment. In one embodiment, the message routing service is used to transfer messages from one component to another without explicitly linking the state of the two components. Note that thismessage routing service 140 may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between storage for cloud service state and for user data, including user service state. - In various embodiments, various cloud computing services or the cloud computing system itself may require a persistent storage for system state. The
data store 150 is available to address this need, but it is not a required part of the system architecture in at least one embodiment. In one embodiment, various aspects of system state are saved in redundant databases on various hosts or as special files in an object storage service. In a second embodiment, a relational database service is used to store system state. In a third embodiment, a column, graph, or document-oriented database is used. Note that this persistent storage may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between storage for cloud service state and for user data, including user service state. - In various embodiments, it may be useful for the
cloud computing system 110 to have asystem controller 160. In one embodiment, thesystem controller 160 is similar to the cloud computing controllers 120, except that it is used to control or direct operations at the level of thecloud computing system 110 rather than at the level of an individual service. - For clarity of discussion above, only one
user device 102 has been illustrated as connected to thecloud computing system 110, and the discussion generally referred to receiving a communication from outside the cloud computing system, routing it to a cloud controller 120, and coordinating processing of the message via a service 130, the infrastructure described is also equally available for sending out messages. These messages may be sent out as replies to previous communications, or they may be internally sourced. Routing messages from a particular service 130 to auser device 102 is accomplished in the same manner as receiving a message fromuser device 102 to a service 130, just in reverse. The precise manner of receiving, processing, responding, and sending messages is described below with reference to the various discussed service embodiments. One of skill in the art will recognize, however, that a plurality ofuser devices 102 may, and typically will, be connected to thecloud computing system 110 and that each element or set of elements within the cloud computing system is replicable as necessary. Further, thecloud computing system 110, whether or not it has one endpoint or multiple endpoints, is expected to encompass embodiments including public clouds, private clouds, hybrid clouds, and multi-vendor clouds. - Each of the
user device 102, thecloud computing system 110, theendpoints 112, the network switches andprocessing nodes 118, the cloud controllers 120 and the cloud services 130 typically include a respective information processing system, a subsystem, or a part of a subsystem for executing processes and performing operations (e.g., processing or communicating information). An information processing system is an electronic device capable of processing, executing or otherwise handling information, such as a computer.FIG. 2 shows aninformation processing system 210 that is representative of one of, or a portion of, the information processing systems described above. - Referring now to
FIG. 2 , diagram 200 shows aninformation processing system 210 configured to host one or more virtual machines, coupled to anetwork 205. Thenetwork 205 could be one or both of thenetworks information processing system 210 shown is representative of, one of, or a portion of, the information processing systems described above. - The
information processing system 210 may include any or all of the following: (a) aprocessor 212 for executing and otherwise processing instructions, (b) one or more network interfaces 214 (e.g., circuitry) for communicating between theprocessor 212 and other devices, those other devices possibly located across thenetwork 205; (c) a memory device 216 (e.g., FLASH memory, a random access memory (RAM) device or a read-only memory (ROM) device for storing information (e.g., instructions executed byprocessor 212 and data operated upon byprocessor 212 in response to such instructions)). In some embodiments, theinformation processing system 210 may also include a separate computer-readable medium 218 operably coupled to theprocessor 212 for storing information and instructions as described further below. - In one embodiment, there is more than one
network interface 214, so that the multiple network interfaces can be used to separately route management, production, and other traffic. In one exemplary embodiment, an information processing system has a “management” interface at 1 GB/s (Gigabits per second), a “production” interface at 10 GB/s, and may have additional interfaces for channel bonding, high availability, or performance. An information processing device configured as a processing or routing node may also have an additional interface dedicated to public Internet traffic, and specific circuitry or resources necessary to act as a VLAN (Virtual Local Area Network) trunk. - In some embodiments, the
information processing system 210 may include a plurality of input/output devices 220 a-n which are operably coupled to theprocessor 212, for inputting or outputting information, such as adisplay device 220 a, a print device 220 b, or other electronic circuitry 220 c-n for performing other operations of theinformation processing system 210 known in the art. - With reference to the computer-readable media, including both
memory device 216 and secondary computer-readable medium 218, the computer-readable media and theprocessor 212 are structurally and functionally interrelated with one another as described below in further detail, and information processing system of the illustrative embodiment is structurally and functionally interrelated with a respective computer-readable medium similar to the manner in which theprocessor 212 is structurally and functionally interrelated with the computer-readable media processor 212 reads (e.g., accesses or copies) such functional descriptive material from thenetwork interface 214, the computer-readable media 218 onto thememory device 216 of theinformation processing system 210, and the information processing system 210 (more particularly, the processor 212) performs its operations, as described elsewhere herein, in response to such material stored in the memory device of theinformation processing system 210. In addition to reading such functional descriptive material from the computer-readable medium 218, theprocessor 212 is capable of reading such functional descriptive material from (or through) the network 105. In one embodiment, theinformation processing system 210 includes at least one type of computer-readable media that is non-transitory. For explanatory purposes below, singular forms such as “computer-readable medium,” “memory,” and “disk” are used, but it is intended that these may refer to all or any portion of the computer-readable media available in or to a particularinformation processing system 210, without limiting them to a specific location or implementation. - The
information processing system 210 includes ahypervisor 230. Thehypervisor 230 may be implemented in software, as a subsidiary information processing system, or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that software is used to implement the hypervisor, it may include software that is stored on a computer-readable medium, including the computer-readable medium 218. The hypervisor may be included logically “below” a host operating system, as a host itself, as part of a larger host operating system, or as a program or process running “above” or “on top of” a host operating system. Examples of hypervisors include Xenserver, KVM (Kernel-based Virtual Machine), VMware, Microsoft's Hyper-V, and emulation programs such as QEMU. - The
hypervisor 230 includes the functionality to add, remove, and modify a number oflogical containers 232 a-n associated with the hypervisor. Zero, one, or many of thelogical containers 232 a-n contain associated operatingenvironments 234 a-n. Thelogical containers 232 a-n can implement various interfaces depending upon the desired characteristics of the operating environment. In one embodiment, alogical container 232 implements a hardware-like interface, such that the associatedoperating environment 234 appears to be running on or within an information processing system such as theinformation processing system 210. For example, one embodiment of alogical container 234 could implement an interface resembling an x86, x86-64, ARM, or other computer instruction set with appropriate RAM, busses, disks, and network devices. Acorresponding operating environment 234 for this embodiment could be an operating system such as Microsoft Windows, Linux, Linux-Android, or Mac OS X. In another embodiment, alogical container 232 implements an operating system-like interface, such that the associatedoperating environment 234 appears to be running on or within an operating system. For example one embodiment of this type oflogical container 232 could appear to be a Microsoft Windows, Linux, or Mac OS X operating system. Another possible operating system includes an Android operating system, which includes significant runtime functionality on top of a lower-level kernel. Acorresponding operating environment 234 could enforce separation between users and processes such that each process or group of processes appeared to have sole access to the resources of the operating system. In a third environment, alogical container 232 implements a software-defined interface, such a language runtime or logical process that the associatedoperating environment 234 can use to run and interact with its environment. For example one embodiment of this type oflogical container 232 could appear to be a Java, Dalvik, Lua, Python, or other language virtual machine. Acorresponding operating environment 234 would use the built-in threading, processing, and code loading capabilities to load and run code. Adding, removing, or modifying alogical container 232 may or may not also involve adding, removing, or modifying an associatedoperating environment 234. For ease of explanation below, these operating environments will be described in terms of an embodiment as “Virtual Machines,” or “VMs,” but this is simply one implementation among the options listed above. - In one or more embodiments, a VM has one or more virtual network interfaces 236. How the virtual network interface is exposed to the operating environment depends upon the implementation of the operating environment. In an operating environment that mimics a hardware computer, the
virtual network interface 236 appears as one or more virtual network interface cards. In an operating environment that appears as an operating system, thevirtual network interface 236 appears as a virtual character device or socket. In an operating environment that appears as a language runtime, the virtual network interface appears as a socket, queue, message service, or other appropriate construct. The virtual network interfaces (VNIs) 236 may be associated with a virtual switch (Vswitch) at either the hypervisor or container level. TheVNI 236 logically couples the operatingenvironment 234 to the network, and allows the VMs to send and receive network traffic. In one embodiment, the physicalnetwork interface card 214 is also coupled to one or more VMs through a Vswitch. - In one or more embodiments, each VM includes identification data for use naming, interacting, or referring to the VM. This can include the Media Access Control (MAC) address, the Internet Protocol (IP) address, and one or more unambiguous names or identifiers.
- In one or more embodiments, a “volume” is a detachable block storage device. In some embodiments, a particular volume can only be attached to one instance at a time, whereas in other embodiments a volume works like a Storage Area Network (SAN) so that it can be concurrently accessed by multiple devices. Volumes can be attached to either a particular information processing device or a particular virtual machine, so they are or appear to be local to that machine. Further, a volume attached to one information processing device or VM can be exported over the network to share access with other instances using common file sharing protocols. In other embodiments, there are areas of storage declared to be “local storage.” Typically a local storage volume will be storage from the information processing device shared with or exposed to one or more operating environments on the information processing device. Local storage is guaranteed to exist only for the duration of the operating environment; recreating the operating environment may or may not remove or erase any local storage associated with that operating environment.
- Turning now to
FIG. 3 , a simplenetwork operating environment 300 for a cloud controller or cloud service is shown. Thenetwork operating environment 300 includes multiple information processing systems 310 a-n, each of which correspond to a singleinformation processing system 210 as described relative toFIG. 2 , including ahypervisor 230, zero or morelogical containers 232 and zero ormore operating environments 234. The information processing systems 310 a-n are connected via acommunication medium 312, typically implemented using a known network protocol such as Ethernet, Fibre Channel, Infiniband, or IEEE (Institute for Electrical and Electronics Engineers) 1394. For ease of explanation, thenetwork operating environment 300 will be referred to as a “cluster,” “group,” or “zone” of operating environments. The cluster may also include acluster monitor 314 and anetwork routing element 316. Thecluster monitor 314 andnetwork routing element 316 may be implemented as hardware, as software running on hardware, or may be implemented completely as software. In one implementation, one or both of the cluster monitor 314 ornetwork routing element 316 is implemented in alogical container 232 using anoperating environment 234 as described above. In another embodiment, one or both of the cluster monitor 314 ornetwork routing element 316 is implemented so that the cluster corresponds to a group of physically co-located information processing systems, such as in a rack, row, or group of physical machines. - The cluster monitor 314 provides an interface to the cluster in general, and provides a single point of contact allowing someone outside the system to query and control any one of the information processing systems 310, the
logical containers 232 and the operatingenvironments 234. In one embodiment, the cluster monitor also provides monitoring and reporting capabilities. - The
network routing element 316 allows the information processing systems 310, thelogical containers 232 and the operatingenvironments 234 to be connected together in a network topology. The illustrated tree topology is only one possible topology; the information processing systems and operating environments can be logically arrayed in a ring, in a star, in a graph, or in multiple logical arrangements through the use of vLANs. - In one embodiment, the cluster also includes a
cluster controller 318. The cluster controller is outside the cluster, and is used to store or provide identifying information associated with the different addressable elements in the cluster—specifically the cluster generally (addressable as the cluster monitor 314), the cluster network router (addressable as the network routing element 316), each information processing system 310, and with each information processing system the associatedlogical containers 232 and operatingenvironments 234. - The
cluster controller 318 is outside the cluster, and is used to store or provide identifying information associated with the different addressable elements in the cluster—specifically the cluster generally (addressable as the cluster monitor 314), the cluster network router (addressable as the network routing element 316), each information processing system 310, and with each information processing system the associatedlogical containers 232 and operatingenvironments 234. In one embodiment, thecluster controller 318 includes a registry ofVM information 319. In a second embodiment, theregistry 319 is associated with but not included in thecluster controller 318. - In one embodiment, the cluster also includes one or
more instruction processors 320. In the embodiment shown, the instruction processor is located in the hypervisor, but it is also contemplated to locate an instruction processor within an active VM or at a cluster level, for example in a piece of machinery associated with a rack or cluster. In one embodiment, theinstruction processor 320 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium. Further, one or more embodiments have associated with them abuffer 322. Thebuffer 322 can take the form of data structures, a memory, a computer-readable medium, or an off-script-processor facility. For example, one embodiment uses a language runtime as aninstruction processor 320. The language runtime can be run directly on top of the hypervisor, as a process in an active operating environment, or can be run from a low-power embedded processor. In a second embodiment, theinstruction processor 320 takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs. For example, in this embodiment, an interoperating bash shell, gzip program, an rsync program, and a cryptographic accelerator chip are all components that may be used in aninstruction processor 320. In another embodiment, theinstruction processor 320 is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor. This hardware-based instruction processor can be embedded on a network interface card, built into the hardware of a rack, or provided as an add-on to the physical chips associated with an information processing system 310. It is expected that in many embodiments, theinstruction processor 320 will have an integrated battery and will be able to spend an extended period of time without drawing current. Various embodiments also contemplate the use of an embedded Linux or Linux-Android environment. - In the disclosure that follows, the information processing devices as described relative to
FIG. 2 and the clusters as described relative toFIG. 3 are used as underlying infrastructure to build and administer various cloud services. Except where noted specifically, either a single information processing device or a cluster can be used interchangeably to implement a single “node,” “service,” or “controller.” Where a plurality of resources are described, such as a plurality of storage nodes or a plurality of compute nodes, the plurality of resources can be implemented as a plurality of information processing devices, as a one-to-one relationship of information processing devices, logical containers, and operating environments, or in an M×N relationship of information processing devices to logical containers and operating environments. - Various aspects of the services implemented in the cloud computing system may be referred to as “virtual machines” or “virtual devices”; as described above, those refer to a particular logical container and operating environment, configured to perform the service described. The term “instance” is sometimes used to refer to a particular virtual machine running inside the cloud computing system. An “instance type” describes the compute, memory and storage capacity of particular VM instances.
- In various embodiments, groups of resources (information processing systems, logical containers, operating environments, users, accounts, etc.) may be organized into “zones.” A zone is defined as a group of one or more resources that share one or more common characteristics and are grouped together to achieve some function. For example, one example of a zone is an availability zone, which is a group of resources subject to a correlated loss of access or data as a result of a particular event. For example, a group of operating environments that use a common underlying network connection, power connection, or computer-readable medium is subject to loss of access to its stored objects as a result of a failure of one of these resources. A group of resources in the same cluster may suffer a loss of access if the cluster is contained within a single physical building.
- Zones may overlap and may be defined for different reasons. For example, a group of resources that share a computer-readable medium may be defined to be in one zone. A plurality of information processing devices 210 (and their
underlying hypervisors 230,logical containers 232 and operating environments 234) in a given storage rack or cabinet may be defined to be in a zone, A plurality of information processing devices 210 (and theirunderlying hypervisors 230,logical containers 232 and operating environments 234) coupled to the same networking switch may be defined to be in a zone; and a plurality of information processing devices 210 (and theirunderlying hypervisors 230,logical containers 232 and operating environments 234), in a given datacenter may be defined to be in a zone. - In another embodiment, a zone is defined by the availability of specialized hardware or by hardware of a certain class. For example, a plurality of information processing devices 210 (and their
underlying hypervisors 230,logical containers 232 and operating environments 234) that have access to high-throughput and low-latency storage, like a solid state disk, may be defined to be in a zone. Zones may be organized to reflect differing amounts of memory, processor speeds, age of installation, type of operating system or hypervisor, or any other underlying difference in platforms. - One of skill in the art will recognize that the examples of zones provided above are not limiting, and a variety of possible organizational schemes fall into the scope of the present disclosure.
- Within the architecture described above, various services are provided, and different capabilities can be included through a plug-in architecture. Although specific services and plugins are detailed below, these disclosures are intended to be representative of the services and plugins available for integration across the entire
cloud computing system 110. - Referring now to
FIG. 4 , a diagram of an IaaS-style network service 400 is shown according to one embodiment. The network has one or morephysical edge routers 402 which connect to the broader Internet. Theseedge routers 402 are connected to an underlyingphysical distribution network 404. The implementation of thephysical distribution network 404 may encompass various protocols or services, including but not limited to electrical, optical, or wireless connections at the physical layer; Ethernet, Fibre channel, ATM, and SONET at the MAC layer; MPLS (Multi-protocol Label Switching) betweenlayers 2 and 3, and TCP, UDP, ZeroMQ or other services at the connection layer. Thephysical distribution network 404 corresponds to theinternal network 114 described relative toFIG. 1 . Thephysical distribution network 404 is connected tophysical interfaces 423 at each physicalinformation processing system 420. The physicalinformation processing systems 420 correspond to theinformation processing systems 210 as described relative toFIG. 2 , with ahypervisor 421, and operating environments within logical containers 422, each of which with one or more virtual network interfaces (VNIs) 424. - On top of the physical distribution network is a virtual network service defining one or more virtual networks and virtual network devices. The
virtual network service 400 corresponds to thevirtual network 116 as describe relative toFIG. 1 . Thevirtual network service 400 defines packet flows between hosts in the network and is used to logically route packets independent of the underlying physical infrastructure. In one embodiment, the virtual network service usesvirtual routers virtual routers 406 and 426) have anAPI 410 that allows the routing tables, packet filters, and underlying logical network fabric to be remotely reconfigured. The virtual routers may be located in thephysical edge routers 402, or within one of theinformation processing systems 420, within thehypervisor 421, within a particular operating environment 422, or at an outside network routing controller 430. These virtual network devices can work together to present multiple logical connections and networks that are independent from each other even though they are running on the samephysical distribution network 404. Information from a particular flow is tunneled or bridged from one physical network segment to another or is kept logically connected by associating it with an ATM-like circuit or an MPLS-style label. - The virtual
network service API 410 allows for creation and management of virtual networks each of which can have one or more ports. A port on a virtual network can be attached to a network interface, where a network interface is anything which can source traffic, such asphysical interface 423 or one of theVNIs 424. - The
API 410 allows users of the network as well as cloud administrators to declaratively define a network architecture including a series of virtual connections, routing rules, and filters. TheVNIs 424 are then able to interact with the virtual network in the same way as a physical network without needing to also configure the underlyingphysical distribution network 404. Adapters are used to bridge between the virtual network devices and the physical devices without exposing the way in which the physical devices are actually connected. - In one embodiment, this is done by using programmable kernel bridges. A network device is instantiated within an operating system to communicate with
physical interface 423 operating over a defined protocol, such as Ethernet or MPLS. AVNI 424 is also defined by the operating system. While theVNI 424 will often be associated with an operating environment, logical container, or virtual machine, those of skill in the art will recognize thatVNIs 424 can be created as pure abstractions to help administer or logically control the network flows. Finally, one or more kernel bridges are defined routing thelayer 2 packets from thephysical network interface 423 to one ormore VNIs 424. The raw packets arriving on thephysical interface 423 are routed according to the established bridges and then used by the operating environments, hypervisor, or physical device. - Because the connections across the kernel bridges are software-defined and completely arbitrary, the routing fabric associated with a particular
physical network interface 423 can be reconfigured by changing the bridges associated with that interface. On a more fine-grained level, the routing betweenVNIs 424 is also software-defined, making the flows across theVNIs 424 similarly malleable to software manipulation via theAPI 410. - In a further embodiment, an entire flow coming off of a
physical network interface 423 or a flow coming off of aVNI 424 can be filtered, diverted, or inspected independently. In this way, advanced networking functionality such as load balancing, firewalls, layer 7 routing, and geographic routing can be accomplished in a multi-tenant environment. One implementation of this uses standard networking utilities. Thephysical network device 423 and theVNIs 424, while different to the kernel, expose a standard network or socket API to upper-level layers and applications. Accordingly, a standard firewall is able to filter the packets and send them on without being aware that the packets being filtered are arriving over a virtual network interface instead of a physical one. - In another embodiment, the packet-routing logic within the cloud system as a whole can be centrally directed using network routing controller 430. In standard networks, each router has an independent routing table and seeks to find the best route for each packet that arrives by examining its existing routing tables and probing the network around it using various protocols adapted for that purpose. The
network service 400 can work identically to standard networks in this respect, but the ability to dynamically sense and control theflows using API 410 allows routing decisions to be made centrally when that makes more sense. Network routing controller 430 may, for example, control the routing acrossvirtual routers - To accommodate these use cases, one embodiment of the
virtual network service 400 provides a plug-in architecture allowing entry routing (controlled by virtual router 406), overall network routing (controlled by the network routing controller 430 or by the combination of virtual routers) or anyparticular router 426 to have a tenant-defined flow routing and filtering policy. For example, one plugin allows a tenant-specific security policy to be implemented on a flow. In this embodiment, avirtual router 426 is instantiated for use by a particular user (the “user router”). The user router can be implemented in thehypervisor 421 or in a distinct operating environment 422. The user router has anAPI 410, just like the othervirtual routers 426. In one embodiment, routing rules can be set using theAPI 410. In a second embodiment, a series of rules can be provided and loaded by theuser router 426. This loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module. When a packet arrives atedge router 402, thevirtual router 406 identifies it as being logically addressed to a particular operating environment associated with the user and routes it to flow to the defineduser router 426 instantiated for the customer by way of thephysical interface 423 and possibly othervirtual routers 426 along the way. When the packet arrives atuser router 426, the tenant-defined rules and filters are applied to the packet and the flow is stopped, edited, or redirected accordingly. - A second embodiment allows QoS (Quality of Service) policies to be implemented on a flow. As noted above, a centralized routing policy allows global maximization of network utilization. In this embodiment, a user-defined network routing controller 430 (the “user controller”) is defined, or a plug-in or ruleset is loaded into a system-wide network routing controller using the same functionality described above. When a high-priority packet arrives or is sent, the user controller can route other packets on less-speedy routes and keep the most direct route free for high-priority packets. In this way the delay on best-effort packets is due mostly to a higher number of hops as opposed to rate limiting; each individual router (either physical or virtual) is operating at its peak capacity. The load is managed by virtually spreading out the number of routers handling packets rather than by limiting the numbers of packets being routed at a particular node.
- Referring now to
FIG. 5a , a network service-implemented VLAN available to one embodiment of the system is shown. Thenetwork 500 is one embodiment of avirtual network 116 as discussed relative toFIG. 1 , and is implemented on top of theinternal network layer 114. A particular node is connected to thevirtual network 500 through avirtual network interface 236 operating throughphysical network interface 214. The VLANs, VSwitches, VPNs (Virtual Private Networks), and other pieces of network hardware (real or virtual) may benetwork routing elements 316 or may serve another function in thecommunications medium 312. - In one embodiment, the
cloud computing system 110 uses both “fixed” IP addresses and “floating” IP addresses to address virtual machines. Fixed IP addresses are assigned to an instance on creation and stay the same until the instance is explicitly terminated. Floating IP addresses are IP addresses that can be dynamically associated with an instance. A floating IP address can be disassociated and associated with another instance at any time. - Different embodiments include various strategies for implementing and allocating fixed IP addresses, including “flat” mode, a “flat DHCP” (Dynamic Host Control Protocol) mode, and a “VLAN DHCP” mode.
- In one embodiment, fixed IP addresses are managed using a flat Mode. In this embodiment, an instance receives a fixed IP from a pool of available IP addresses. All instances are attached to the same bridge by default. Other networking configuration instructions are placed into the instance before it is booted or on boot.
- In another embodiment, fixed IP addresses are managed using a flat DHCP mode. Flat DHCP mode is similar to the flat mode, in that all instances are attached to the same bridge. Instances will attempt to bridge using the default Ethernet device or socket. Instead of allocation from a fixed pool, a DHCP server listens on the bridge and instances receive their fixed IPs by doing a dhcpdiscover.
- Turning now to a preferred embodiment using VLAN DHCP mode, there are two groups of off-local-network users, the
private users 502 and the public internet users 505. To respond to communications from theprivate users 502 and the public users 505, thenetwork 500 includes three nodes,network node 510,private node 520, andpublic node 530. The nodes include one or more virtual machines or virtual devices, such as DNS (Domain Name System)/DHCP server 512 andvirtual router 514 onnetwork node 510, -
VPN 522 andprivate VM 524 onprivate node 520, andpublic VM 532 onpublic node 530. - In one embodiment, VLAN DHCP mode requires a switch that supports host-managed VLAN tagging. In one embodiment, there is a
VLAN 506 and bridge 516 for each project or group. In the illustrated embodiment, there is a VLAN associated with a particular project. The project receives a range of private IP addresses that are only accessible from inside the VLAN, and assigns an IP address from this range toprivate node 520, as well as to a VNI in the virtual devices in the VLAN. In one embodiment,DHCP server 512 is running on a VM that receives a static VLAN IP address at a known address, andvirtual router 514,VPN 522,private VM 524, andpublic VM 532 all receive private IP addresses upon request to the DHCP server running on the DHCP server VM. In addition, the DHCP server provides a public IP address to thevirtual router 514 and optionally to thepublic VM 532. In a second embodiment, theDHCP server 512 is running on or available from thevirtual router 514, and the public IP address of thevirtual router 514 is used as the DHCP address. - In an embodiment using VLAN DHCP mode, there is a private network segment for each project's or group's instances that can be accessed via a dedicated VPN connection from the Internet. As described below, each VLAN project or group gets its own VLAN, network bridge, and subnet. In one embodiment, subnets are specified by the network administrator, and assigned dynamically to a project or group when required. A DHCP Server is started for each VLAN to pass out IP addresses to VM instances from the assigned subnet. All instances belonging to the VLAN project or group are bridged into the same VLAN. In this fashion, network traffic between VM instances belonging to the same VLAN is always open but the system can enforce isolation of network traffic between different projects by enforcing one VLAN per project.
- As shown in
FIG. 5a , VLAN DHCP mode includes provisions for both private and public access. For private access (shown by the arrows to and from the private users cloud 502), users create an access keypair (as described further below) for access to the virtual private network through thegateway VPN 522. From theVPN 522, both theprivate VM 524 and thepublic VM 532 are accessible via the private IP addresses valid on the VLAN. - Public access is shown by the arrows to and from the public users cloud 505. Communications that come in from the public users cloud arrive at the virtual router VM 515 and are subject to network address translation (NAT) to access the public virtual machine via the
bridge 516. Communications out from theprivate VM 524 are source NATted by thebridge 516 so that the external source appears to be the virtual router VM 515. If thepublic VM 532 does not have an externally routable address, communications out from thepublic VM 532 may be source NATted as well. - In one embodiment of VLAN DHCP mode, the second IP in each private network is reserved for the
VPN instance 522. This gives a consistent IP to the instance so that forwarding rules can be more easily created. The network for each project is given a specific high-numbered port on the public IP of thenetwork node 510. This port is automatically forwarded to the appropriate VPN port on theVPN 522. - In one embodiment, each group or project has its own certificate authority (CA) 523. The
CA 523 is used to sign the certificate for theVPN 522, and is also passed to users on the private users cloud 502. When a certificate is revoked, a new Certificate Revocation List (CRL) is generated. TheVPN 522 will block revoked users from connecting to the VPN if they attempt to connect using a revoked certificate. - In a project VLAN organized similarly to the embodiment described above, the project has an independent RFC 1918 IP space; public IP via NAT; has no default inbound network access without public NAT; has limited, controllable outbound network access; limited, controllable access to other project segments; and VPN access to instance and cloud APIs. Further, there is a DMZ segment for support services, allowing project metadata and reporting to be provided in a secure manner.
- In one embodiment, VLANs are segregated using 802.1q VLAN tagging in the switching layer, but other tagging schemes such as 802.1ad, MPLS, or frame tagging are also contemplated. The flows are defined by the
virtual network 400 as described relative toFIG. 4 , so the underlying implementation can be chosen independent of the logical virtual network on top. Network hosts create VLAN-specific interfaces and bridges as required using auser router 426. - In one embodiment,
private VM 524 has per-VLAN interfaces and bridges created as required. These do not have IP addresses in the host to protect host access. Access is provided via routing table entries created per project and instance to protect against IP/MAC address spoofing and ARP (Address Resolution Protocol) poisoning. -
FIG. 5b is a flow chart showing the establishment of a VLAN for a project according to one embodiment. Theprocess 550 starts at step 551, when a VM instance for the project is requested. When running a VM instance, a user needs to specify a project for the instances, and the applicable security rules and security groups (as described herein) that the instance should join. Atstep 552, a cloud controller determines if this is the first instance to be created for the project. If this is the first, then the process proceeds to step 553. If the project already exists, then the process moves to step 559. Atstep 553, a user controller is identified to act as the network host for the project. This may involve creating a virtual network device and assigning it the role of network controller. In one embodiment, this is avirtual router 514. Atstep 554, an unused VLAN id and unused subnet are identified. Atstep 555, the VLAN id and subnet are assigned to the project. Atstep 556,DHCP server 512 andbridge 516 are instantiated and registered. Atstep 557, the VM instance request is examined to see if the request is for aprivate VM 524 orpublic VM 532. If the request is for a private VM, the process moves to step 558. Otherwise, the process moves to step 559. Atstep 558, theVPN 522 is instantiated and allocated the second IP in the assigned subnet. Atstep 559, the subnet and a VLAN have already been assigned to the project. Accordingly, the requested VM is created and assigned and assigned a private IP within the project's subnet. Atstep 560, the routing rules inbridge 516 are updated to properly NAT traffic to or from the requested VM. - Those of skill in the art will note that the VPN and VLAN functionality described relative to
FIGS. 5a and 5b can appear to be configured and provisioned as in a legacy network, or can be completely implemented virtually using plugins, virtual routers, and centralized routing. The underlying implementation is transparent to the consumer of the network services. Other plugin implementations are defined similarly; load balancing can be dynamically adjusted based upon actual load; failover or service scaling can be defined “in the network,” and can occur transparently. Various services are envisioned, including basic network connectivity, network packet filtering, IP address management, load balancing, QoS, layer 7 routing, VLANs, L2-in-L3 (and other layer) tunneling, advanced security services and geo-routing. Multiple plugins can be combined to provide layered capabilities, with each plugin service being defined within a separate “segment” of the network for visibility and debuggability. - Between the various virtual machines and virtual devices, it may be necessary to have a reliable messaging infrastructure. In various embodiments, a message queuing service is used for both local and remote communication so that there is no requirement that any of the services exist on the same physical machine. Various existing messaging infrastructures are contemplated, including AMQP, ZeroMQ, STOMP and XMPP. Note that this messaging system may or may not be available for user-addressable systems; in one preferred embodiment, there is a separation between internal messaging services and any messaging services associated with user data. The messaging service may run alongside or on top of the
network service 400 described relative toFIGS. 4-5 b. - In one embodiment, the message service sits between various components and allows them to communicate in a loosely coupled fashion. This can be accomplished using Remote Procedure Calls (RPC hereinafter) to communicate between components, built atop either direct messages and/or an underlying publish/subscribe infrastructure. In a typical embodiment, it is expected that both direct and topic-based exchanges are used. This allows for decoupling of the components, full asynchronous communications, and transparent balancing between equivalent components. In some embodiments, calls between different APIs can be supported over the distributed system by providing an adapter class which takes care of marshalling and unmarshalling of messages into function calls.
- In one embodiment, a cloud controller 120 (or the applicable cloud service 130) creates two queues at initialization time, one that accepts node-specific messages and another that accepts generic messages addressed to any node of a particular type. This allows both specific node control as well as orchestration of the cloud service without limiting the particular implementation of a node. In an embodiment in which these message queues are bridged to an API, the API can act as a consumer, server, or publisher.
- Turning now to
FIG. 6a , one implementation of amessage service 140 is shown atreference number 600. For simplicity of description,FIG. 6a shows themessage service 600 when asingle instance 602 is deployed and shared in thecloud computing system 110, but themessage service 600 can be either centralized or fully distributed. - In one embodiment, the
message service 600 keeps traffic associated with different queues or routing keys separate, so that disparate services can use the message service without interfering with each other. Accordingly, the message queue service may be used to communicate messages between network elements, between cloud services 130, between cloud controllers 120, between network elements, or between any group of sub-elements within the above. More than onemessage service 600 may be used, and a cloud service 130 may use its own message service as required. - For clarity of exposition, access to the
message service 600 will be described in terms of “Invokers” and “Workers,” but these labels are purely expository and are not intended to convey a limitation on purpose; in some embodiments, a single component (such as a VM) may act first as an Invoker, then as a Worker, the other way around, or simultaneously in each role. An Invoker is a component that sends messages in the system via two operations: 1) an RPC directed message and ii) an RPC broadcast. A Worker is a component that receives messages from the message system and replies accordingly. - In one embodiment, there is a
message server 602 including one or more exchanges 610. In a second embodiment, the message system is “brokerless,” and one or more exchanges are located at each client. The exchanges 610 act as internal message routing elements so that components interacting with themessage service 600 can send and receive messages. In one embodiment, these exchanges are subdivided further into adirect exchange 610 a and atopic exchange 610 b. An exchange 610 is a routing structure or system that exists in a particular context. In a currently preferred embodiment, multiple contexts can be included within a single message service with each one acting independently of the others. In one embodiment, the type of exchange, such as adirect exchange 610 avs. topic exchange 610 b determines the routing policy. In a second embodiment, the routing policy is determined via a series of routing rules evaluated by the exchange 610 via a plugin service described further below. - The
direct exchange 610 a is a routing element created during or for RPC directed message operations. In one embodiment, there are many instances of adirect exchange 610 a that are created as needed for themessage service 600. In a further embodiment, there is onedirect exchange 610 a created for each RPC directed message received by the system. - The
topic exchange 610 b is a routing element created during or for RPC directed broadcast operations. In one simple embodiment, every message received by the topic exchange is received by every other connected component. In a second embodiment, the routing rule within a topic exchange is described as publish-subscribe, wherein different components can specify a discriminating function and only topics matching the discriminator are passed along. In one embodiment, there are many instances of atopic exchange 610 b that are created as needed for themessage service 600. In one embodiment, there is one topic-based exchange for every topic created in the cloud computing system. In a second embodiment, there are a set number of topics that have pre-created andpersistent topic exchanges 610 b. - Within one or more of the exchanges 610, it may be useful to have a
queue element 615. Aqueue 615 is a message stream; messages sent into the stream are kept in thequeue 615 until a consuming component connects to the queue and fetches the message. Aqueue 615 can be shared or can be exclusive. In one embodiment, queues with the same topic are shared amongst Workers subscribed to that topic. - In a typical embodiment, a
queue 615 will implement a FIFO policy for messages and ensure that they are delivered in the same order that they are received. In other embodiments, however, aqueue 615 may implement other policies, such as LIFO, a priority queue (highest-priority messages are delivered first), or age (oldest objects in the queue are delivered first), or other configurable delivery policies. In other embodiments, aqueue 615 may or may not make any guarantees related to message delivery or message persistence. - In one embodiment,
element 620 is a topic publisher. Atopic publisher 620 is created, instantiated, or awakened when an RPC directed message or an RPC broadcast operation is executed; this object is instantiated and used to push a message to the messages system. Every publisher connects always to the same topic-based exchange; its life-cycle is limited to the message delivery. - In one embodiment,
element 630 is a direct consumer. Adirect consumer 630 is created, instantiated, or awakened if an RPC directed message operation is executed; this component is instantiated and used to receive a response message from the queuing system. Everydirect consumer 630 connects to a unique direct-based exchange via a unique exclusive queue, identified by a UUID (Universally Unique Identifier) or other unique name. The life-cycle of thedirect consumer 630 is limited to the message delivery. In one embodiment, the exchange and queue identifiers are included the message sent by thetopic publisher 620 for RPC directed message operations. - In one embodiment,
element 640 is a topic consumer. In one embodiment, atopic consumer 640 is created, instantiated, or awakened at system start. In a second embodiment, atopic consumer 640 is created, instantiated, or awakened when a topic is registered with themessage system 600. In a third embodiment, atopic consumer 640 is created, instantiated, or awakened at the same time that a Worker or Workers are instantiated and persists as long as the associated Worker or Workers have not been destroyed. In this embodiment, thetopic consumer 640 is used to receive messages from the queue and it invokes the appropriate action as defined by the Worker role. Atopic consumer 640 connects to the topic-based exchange either via a shared queue or via a unique exclusive queue. In one embodiment, every Worker has two associatedtopic consumers 640, one that is addressed only during an RPC broadcast operations (and it connects to a shared queue whose exchange key is defined by the topic) and the other that is addressed only during an RPC directed message operations, connected to a unique queue whose with the exchange key is defined by the topic and the host. - In one embodiment,
element 660 is a direct publisher. In one embodiment, adirect publisher 660 is created, instantiated, or awakened for RPC directed message operations and it is instantiated to return the message required by the request/response operation. The object connects to a direct-based exchange whose identity is dictated by the incoming message. - Turning now to
FIG. 6b , one embodiment of the process of sending an RPC directed message is shown relative to the elements of themessage system 600 as described relative toFIG. 6a . Atstep 660, atopic publisher 620 is instantiated. Atstep 661, thetopic publisher 620 sends a message to anexchange 610 b. Atstep 662, adirect consumer 630 is instantiated to wait for the response message. Atstep 663, the message is dispatched by theexchange 610 b. Atstep 664, the message is fetched by thetopic consumer 640 dictated by the routing key (either by topic or by topic and host). Atstep 666, the message is passed to a Worker associated with thetopic consumer 640. If needed, atstep 666, adirect publisher 660 is instantiated to send a response message via themessage system 600. Atstep 667, thedirect publisher 640 sends a message to anexchange 610 a. Atstep 668, the response message is dispatched by theexchange 610 a. Atstep 669, the response message is fetched by thedirect consumer 630 instantiated to receive the response and dictated by the routing key. Atstep 670, the message response is passed to the Invoker. - Turning now to
FIG. 6c , one embodiment of the process of sending an RPC broadcast message is shown relative to the elements of themessage system 600 as described relative toFIG. 6a . Atstep 680, atopic publisher 620 is instantiated. Atstep 681, thetopic publisher 620 sends a message to anexchange 610 b. Atstep 682, the message is dispatched by theexchange 610 b. Atstep 683, the message is fetched by atopic consumer 640 dictated by the routing key (either by topic or by topic and host). Atstep 684, the message is passed to a Worker associated with thetopic consumer 640. - In some embodiments, a response to an RPC broadcast message can be requested. In that case, the process follows the steps outlined relative to
FIG. 6b to return a response to the Invoker. - As noted above, in one embodiment the routing policy is determined via a series of routing rules evaluated by the exchange 610 via a plugin service. Unlike the
network service 400, themessage service 600 is already software-defined, so it is not necessary to create pseudo-hardware ports to bridge the physical-virtual gap. Instead, the routing functions of the message service can be directly accessed or modified using an API and/or a plugin interface. In one embodiment, a plugin is implemented as a series of loadable and executable rules that can be provided to themessage service 600 and run by thedirect exchange 610 a or thetopic exchange 610 b as part of its routing. In one embodiment, this loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module. - In one embodiment, the plugin interface is a messaging “device” that implements a particular messaging pattern. For example, one embodiment uses a “forwarding” device. This is particularly useful in a cloud architecture where the endpoints of a message are relatively dynamic and there may not be a stable connection point. This plugin controls the routing of messages in a queue and passes messages from one (possibly dynamic) messaging endpoint to another. Another possible plugin “device” embodiment implements a pipeline-style messaging architecture. Messages arriving at one end of the device are sent to one or more downstream Workers, where the messages can be processed in parallel instead of serially. A third embodiment of a plugin device is a load balancer.
- In the image below we can see such a device being used, in this situation both the client and the server initialize a connection to the forwarder, which binds to two different ports. Using such a device will remove the need of extra application logic, as you will not need to maintain a list of connected peers.
- Because many aspects of the cloud computing system do not allow direct access to the underlying hardware or services, many aspects of the cloud computing system are handled declaratively, through rule-based computing. Rule-based computing organizes statements into a data model that can be used for deduction, rewriting, and other inferential or transformational tasks. The data model can then be used to represent some problem domain and reason about the objects in that domain and the relations between them. In one embodiment, one or more controllers or services have an associated rule processor that performs rule-based deduction, inference, and reasoning. Rule engines are particularly applicable as a plugin interface, where particular outcome can be presented either as the outcome of a series of rules provided by an end user or as the output of a “black box” that takes one or more defined inputs and returns an output.
- Rule Engines can be implemented similarly to script processors as described relative to
FIG. 3 , and may be implemented as a sub-module of a script processor where needed. In other embodiments, Rule Engines can be implemented as discrete components, for example as a tailored electrical circuit or as software instructions to be used in conjunction with a hardware processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium. Further, one or more embodiments have associated with them a buffer. The buffer can take the form of data structures, a memory, a computer-readable medium, or an off-rule-engine facility. For example, one embodiment uses a language runtime as a rule engine, running as a discrete operating environment, as a process in an active operating environment, or can be run from a low-power embedded processor. In a second embodiment, the rule engine takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs. In another embodiment, the rule engine is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor. - One common need for cloud computing systems is an authentication and authorization system. Turning now to
FIG. 7 , one exemplary authentication (“authn”) and authorization (“authz”) system is shown at 700. The authn/authz system shown at 700 is exemplary only and can be used in several different aspects within a cloud computing system. For example, one embodiment uses a single authn/authz system for the entire cloud computing “platform,” including all subsidiary services. Another embodiment has separate authn/authz services for each subsidiary service. A third embodiment has a split authn/authz service, with authn being handled globally so that user, project, tenant, and account details are consistent, but allowing authz to be handled within each subsidiary service so that the privileges granted are logically and physically separable. A further embodiment has multiple authn/authz services, where there are one or more authn/authz services 600 that are used to mediate the authentication and authorization of users to interact with the cloud computing platform and its subsidiary components, and one or more authn/authz services 700 that are tenant-specific and used to control and expose internal virtual cloud resources in a controlled fashion. Rule-based access controls govern the use and interactions of these logical entities. - In a preferred embodiment, a role-based computing system is a system in which identities and resources are managed by aggregating them into “roles” based on job functions, physical location, legal controls, and other criteria. These roles can be used to model organizational structures, manage assets, or organize data. By arranging roles and the associated rules into graphs or hierarchies, these roles can be used to reason about and manage various resources.
- In one application, role-based strategies have been used to form a security model called Role-Based Access Control (RBAC). RBAC associates special rules, called “permissions,” with roles; each role is granted only the minimum permissions necessary for the performance of the functions associated with that role. Identities are assigned to roles, giving the users and other entities the permissions necessary to accomplish job functions. RBAC has been formalized mathematically by NIST (National Institute of Standards and Technology) and accepted as a standard by ANSI (American National Standards Institute). American National Standard 359-2004 is the information technology industry consensus standard for RBAC, and is incorporated herein by reference in its entirety.
- In a preferred embodiment, a user is defined as an entity that will act in one or more roles. A user is typically associated with an internal or external entity that will interact with the cloud computing system in some respect. A user can have multiple roles simultaneously. In one embodiment of the system, a user's roles define which API commands that user can perform.
- Although the “user” construct may sometimes represent a physical human being, there is no limitation that a user must be a person. Rather, a user may be a person, system, or service who interacts with the cloud computing system and is associable with a distinct identity. The authn/
authz service 700 validates requests and API calls with reference to the declared user. No particular implementation of the user construct is required; users may be locally unique within a defined realm or globally unique within the cloud computing system. In most embodiments, a user will be identified using one or more “IDs.” These user IDs can be numbers, strings of characters, or compound multi-valued objects. In some embodiments, this user ID is (or is associated with) a “login”—a sequence of characters presented to the authn/authz system 700 to identify the user. In some embodiments, users can also have further non-security-related metadata associated with them, such as names, companies, email addresses, locations, etc. - In a preferred embodiment, a resource is defined as some object to which access is restricted. In various embodiments, resources can include network or user access to a virtual machine or virtual device, the ability to use the computational abilities of a device, access to storage, an amount of storage, API access, ability to configure a network, ability to access a network, network bandwidth, network speed, network latency, ability to access or set authentication rules, ability to access or set rules regarding resources, etc. In general, any item which may be restricted or metered is modeled as a resource.
- In one embodiment, resources may have quotas associated with them. A quota is a rule limiting the use or access to a resource. A quota can be placed on a per-project level, a per-role level, a per-user level, or a per-group level. In one embodiment, quotas can be applied to the number of volumes which can be created, the total size of all volumes within a project or group, the number of instances which can be launched, both total and per instance type, the number of processor cores which can be allocated, and publicly accessible IP addresses. Other restrictions are also contemplated as described herein.
- In a preferred embodiment, a project is defined as a flexible association of users, acting in certain roles, which will define and access various resources. A project is typically defined by an administrative user according to varying demands. There may be templates for certain types of projects, but a project is a logical grouping created for administrative purposes and may or may not bear a necessary relation to anything outside the project. In a preferred embodiment, arbitrary roles can be defined relating to one or more particular projects only.
- In a preferred embodiment, a group is defined as a logical association of some other defined entity. There may be groups of users, groups of resources, groups of projects, groups of quotas, or groups which contain multiple different types of defined entities. For example, in one embodiment, a group “development” is defined. The development group may include a group of users with the tag “developers” and a group of virtual machine resources (“developer machines”). These may be connected to a developer-only virtual network (“devnet”). The development group may have a number of ongoing development projects, each with an associated “manager” role. There may be per-user quotas on storage and a group-wide quota on the total monthly bill associated with all development resources.
- The applicable set of rules, roles, and quotas is based upon context. In one embodiment, there are global roles, user-specific roles, project-specific roles, and group-specific roles. In one embodiment, a user's actual permissions in a particular project are the intersection of the global roles, user-specific roles, project-specific roles, and group-specific roles associated with that user, as well as any rules associated with project or group resources possibly affected by the user. Other embodiments may also use other grouping or organizational identifiers, such as domains, realms, locations, and organizations.
- In a preferred embodiment, a credential is an identifying token. This token can be either something that is uniquely known or controlled by the user, such as a password, key, certificate, token, or other piece of information, or it can be something associated with the user's identity, such as voice prints, iris or retinal scan data, fingerprints, etc. In one embodiment, unique machine-identifying data can also be used as a credential, such as IP addresses, MAC addresses, embedded key, computer chip, or serial number information, or computationally secure data processing circuits. In one embodiment, the identity-based credential is unique to the user being identified. In a second embodiment, credentials are treated probabilistically, and a number of credentials can be combined to increase the confidence that the user has been authenticated correctly.
- In a preferred embodiment, authentication is the act of confirming the identity of a user or the truth of a claim. The authn/
authz service 700 confirms that incoming requests are being made by the user who claims to be making the call by validating a set of claims provided by the user. In one embodiment, the claims are initially in the form of a set of credentials (username & password, or login and API key). After initial confirmation, The authn/authz service 700 issues a token that can serve as a credential. - In a preferred embodiment, authorization is the act of confirming the capability of a user to perform some action. The authn/
authz service 700 confirms that the user is authorized to perform the actions corresponding to the user's incoming requests. An individual authorization is called a “permission.” In one embodiment, if a user is not authenticated, a “default” or “anonymous” profile is used as the default identity. A token can contain or refer to a set of authorization permissions. - In a preferred embodiment, a token is a data (such as a string) that corresponds to an identity. Tokens are generally cryptographically generated to prevent forging and are associated with access controls. Each token has a scope and a timeframe that describe the resources that can be accessed using the token.
- In a preferred embodiment, an endpoint is a network-accessible address, usually described by URL (Uniform Resource Locator) or URI (Uniform Resource Identifier), where a service may be accessed.
- These different pieces can be used to collectively authenticate and authorize a user. Although the authn/authz service will be described in terms of specific embodiments below, the logical constructs are more important than the particular implementations, and the representation of any particular part of the system may change from embodiment to embodiment.
- Turning again to
FIG. 7 , the authn/authz service 700 is organized as a group of internal services exposed on one ormany endpoints 702. Many of these services are used in a combined fashion by the frontend APIs (such as theservice endpoints 112 described relative toFIG. 1 ). In one embodiment, the authz/authn service 700 immediately authenticates and authorizes a request arriving to the cloud computing system and creates and return a token credential that can be used internally to authorize and authenticate the services without having to duplicate the original credential checks. This token may be returned to the user or may be a hidden internal implementation detail for operational efficiency within the cloud computing system as a whole. In another embodiment, the authn/authz service 700 has a service-specific endpoint within each subsidiary service of the cloud computing system, such asendpoints - The authn/
authz service 700 has four primary divisions, theauthn provider 710, theauthz provider 720, thestorage provider 730, and thecredential service 740. Theauthn provider 710 performs the calculations or comparisons needed to authenticate a user given the provided credentials. Theauthz provider 720 performs the necessary analysis to determine whether an identified user has the necessary permissions to perform a requested action. Thestorage provider 730 provides access to a durable storage that contains authentication and authorization rules, tokens, credentials, and other necessary data. Thecredential service 740 provides new credentials on demand. - Each of these four divisions presents a unified API for consumers, regardless of the underlying implementation. In one or more embodiments, there are various backend services that actually implement, store, or calculate the data needed to provide the authentication, authorization, user/credential storage and generation services exposed by the
providers rule engine 750, adisk store 760, anLDAP database 770, or an RDBMS (Relational Database Management System) 780. Although the embodiment shown inFIG. 7 shows only one backend provider of each type and an architecture in which all backend providers are commonly accessible, this is for purposes of illustration only and should not be considered a limitation. - For example, a first embodiment of the
rule engine 750 encompasses multiple rule engines working in parallel or asingle rule engine 750. Therule engine 750 may or may not have access to one ormore disks 760,LDAP stores 770, RDBMS's 780. In another embodiment, some of the backend services may be combined. Looking again at therule engine 750, one embodiment uses database logic, triggers, and stored procedures to implement therule engine 750. Another implementation uses an RBAC analysis engine associated with an LDAP datastore like Active Directory or OpenLDAP. A third embodiment of therule engine 750 is implemented as described elsewhere in this document. Other embodiments may use other implementations as known in the art. - In one embodiment, the
disk store 760 is implemented using structured files on a computer-readable medium. For example, one embodiment of adisk store 760 comprises /etc/passwd and /etc/grp files. A second embodiment of adisk store 760 is a structured data store, such as a CSV (Cluster Shared Volumes), XML, SQLite or BerkelyDB-formatted and structured file on a disk. A third embodiment uses an object storage service. Other embodiments may use other implementations as known in the art. - In one embodiment, the
LDAP store 770 is implemented using commercially available tree-structured directory such as Active Directory, OpenLDAP, the Red Hat directory service or Novell eDirectory. In another embodiment, theLDAP store 770 is implemented using a graph database or a key-value database, a document database, or a tree-structured filesystem with disk-based storage and attributes. Other embodiments may use other implementations as known in the art. - In one embodiment, the
RDBMS 780 is implemented using commercially available relational database such as MySQL, PostgreSQL, Oracle database server, Microsoft SQL Server, SQLite, DB2, or other relational databases as known in the art. - Each of the four main divisions of the authn/
authz service 700 can call out to other portions of the service, either using the API bus 704 or the direct access bus 706. The API bus 704 is a communications medium, such as a message service or network connection, that allows one provider to call another provider's API as part of its functionality. For example, in one embodiment thecredential service 740 uses the API bus 704 to retrieve a representation of the permissions associated with a particular user so that the permissions can be encoded into a generated token. - The direct access bus 706 is a communications medium, such as a message service or network connection, which allows one provider to call out to one or more of the backend services to provide part or all of its functionality. For example, in one embodiment the
authz provider 720 uses organizational information stored in theLDAP store 770 to evaluate whether a particular action is authorized. - The final component shown in
FIG. 7 is theservice proxy 790. Theservice proxy 790, either as a singular proxy or viamultiple service proxies 790, transparently connect a remote implementation of one or more of the providers (710, 720, 730, 740) or services (750, 760, 770, 780) for execution in a remote location. In various embodiments the service proxy supports federation, caching, and user-specific business logic in the authn/authz system 700. - In one embodiment, two services capable of providing authn/authz service exist, authn/
authz system 1 and authn/authz system 2. Each system has a set of local backend services providing authn, authz, storage, and credential access via theendpoints 702 for a set of users, where the two sets of users are disjoint. Further, a secure communication channel exists between theservice proxy 790 of each service and the remote service. In one implementation, “fallback” logic is provided for each of theauthn provider 710,authz provider 720, andstorage provider 730. The fallback logic first looks to the local implementation for authentication and authorization of users and permissions. If the lookup fails, either in authentication or in authorization, or if the information necessary for calculation of authentication or authorization is not available to thelocal storage provider 730, then theservice proxy 790 is used to present the same user request to the remote system. Only if both the local and remote calculation of authentication, authorization, or storage fails is a rejection returned. - Those of skill in the art will recognize several implications, extensions or modifications of the embodiment described above. First, it is preferred for the connection from the service proxy to go to an
endpoint 702 or API bus 704 of a remote system so as to reduce the coupling between systems, but it is equally possible to use the service proxy to connect directly to a remote service implementation. Second, the remote service implementation does not need to be controlled by the cloud service provider, and in fact it may be more common to have a secure connection to a customer-internal authentication and authorization service so that sensitive authentication and authorization information can stay inside user-controlled systems and never be transmitted outside of a protected area. Third, the embodiment described above is bidirectional, but it would be equally easy to have a unidirectional federation by only connecting oneservice proxy 790 to the remote system without having the second reciprocal link. Fourth, thecredential service 740 can also be proxied using theservice proxy 790, but it may be more efficient in some circumstances to generate a new temporary credential at the local system so that the cross-service proxied call does not need to be made for subsequent accesses. - In a second embodiment, the
service proxy 790 is used to support caching of remote information. This is related to the federation embodiment described above, except that a copy of the credentials and necessary information needed to identify the correct authentication and authorization responses can be saved in a local data store for use in subsequent connections. A “TTL” (time-to-live) value can be imposed on the cached values so as to force periodic reauthorization, and the connection between theservice proxy 790 and the remote system can be used to revoke particular tokens or credentials for reasons other than the expiration of time. An advantage of particular embodiments is that the response to certain API calls can be cached without revealing the underlying mechanism used to compute the response. - In a third embodiment, the
service proxy 790 is used to implement user-specific business logic via a plugin system. Theservice proxy 790 can be used to replace or proxy any provider or backend service to an arbitrary point, including a local service implementing user-specific logic. Theservice proxy 790 provides a plugin API or RPC API that allows user-defined authentication, authorization, storage, and credential services to be used transparently without any change in theouter API endpoints 702. In one implementation, this is performed similarly to the federation or caching examples above, but the “remote” service is actually provided at a local node running the user-provided plugin. - Other implementations of each of the services are also anticipated. In the same fashion described above, each provider or service is configurable to use a backend so that the authn/
authz service 700 can fit a variety of environments and needs. In one embodiment, the backend for each service is defined in a configuration file and the proper backend is loaded at runtime. - Although graph-based, document-based, RDF-based, and key-value-based databases are not explicitly shown, their use is explicitly contemplated in the context of various embodiments. In one embodiment, a key-value store backend is implemented that can support primary key lookups, the most trivial implementation being an in-memory dictionary lookup. In a second embodiment, a relational database mapper can be used to provide many services on top of a traditional SQL (Sequence Query Language) backend, outside of simple RDBMS service. In third embodiment, PAM (Pluggable Authentication Modules) are used as a backend via a system's PAM service. This provides a one-to-one relationship between Users and Groups with the root User also having the ‘admin’ role.
- In one preferred embodiment, authentication of a user is performed through public/private encryption, with keys used to authenticate particular users, or in some cases, particular resources such as particular machines. A user or machine may have multiple keypairs associated with different roles, projects, groups, or permissions. For example, a different key may be needed for general authentication and for project access. In one such embodiment, a user is identified within the system by the possession and use of one or more cryptographic keys, such as an access and secret key. A user's access key needs to be included in a request, and the request must be signed with the secret key. Upon receipt of API requests, the rules engine verifies the signature and executes commands on behalf of the user.
- Some resources, such as virtual machine images, can be shared by many users. Accordingly, it can be impractical or insecure to include private cryptographic information in association with a shared resource. In one embodiment, the system supports providing public keys to resources dynamically. In one exemplary embodiment, a public key, such as an SSH (Secure Shell) key, is injected into a VM instance before it is booted. This allows a user to login to the instances securely, without sharing private key information and compromising security. Other shared resources that require per-instance authentication are handled similarly.
- In one embodiment, a rule processor is also used to attach and evaluate rule-based restrictions on non-user entities within the system. In this embodiment, a “Cloud Security Group” (or just “security group”) is a named collection of access rules that apply to one or more non-user entities. Typically these will include network access rules, such as firewall policies, applicable to a resource, but the rules may apply to any resource, project, or group. For example, in one embodiment a security group specifies which incoming network traffic should be delivered to all VM instances in the group, all other incoming traffic being discarded. Users with the appropriate permissions (as defined by their roles) can modify rules for a group. New rules are automatically enforced for all running instances and instances launched from then on.
- When launching VM instances, a project or group administrator specifies which security groups it wants the VM to join. If the directive to join the groups has been given by an administrator with sufficient permissions, newly launched VMs will become a member of the specified security groups when they are launched. In one embodiment, an instance is assigned to a “default” group if no groups are specified. In a further embodiment, the default group allows all network traffic from other members of this group and discards traffic from other IP addresses and groups. The rules associated with the default group can be modified by users with roles having the appropriate permissions.
- In some embodiments, a security group is similar to a role for a non-user, extending RBAC to projects, groups, and resources. For example, one rule in a security group can stipulate that servers with the “webapp” role must be able to connect to servers with the “database” role on port 3306. In some embodiments, an instance can be launched with membership of multiple security groups—similar to a server with multiple roles. Security groups are not necessarily limited, and can be equally expressive as any other type of RBAC security. In one preferred embodiment, all rules in security groups are ACCEPT rules, making them easily composible.
- In one embodiment, each rule in a security group must specify the source of packets to be allowed. This can be specified using CIDR (Classless Inter-Domain Routing) notation (such as 10.22.0.0/16, representing a private subnet in the 10.22 IP space, or 0.0.0.0/0 representing the entire Internet) or another security group. The creation of rules with other security groups specified as sources helps deal with the elastic nature of cloud computing; instances are impermanent and IP addresses frequently change. In this embodiment, security groups can be maintained dynamically without having to adjust actual IP addresses.
- In one embodiment, the APIs, RBAC-based authentication system, and various specific roles are used to provide a US eAuthentication-compatible federated authentication system to achieve access controls and limits based on traditional operational roles. In a further embodiment, the implementation of auditing APIs provides the necessary environment to receive a certification under FIPS 199 Moderate classification for a hybrid cloud environment.
- Typical implementations of US eAuth authentication systems are structured as a Federated LDAP user store, back-ending to a SAML Policy Controller. The SAML Policy Controller maps access requests or access paths, such as requests to particular URLs, to a Policy Agent in front of an eAuth-secured application. In a preferred embodiment, the application-specific account information is stored either in extended schema on the LDAP server itself, via the use of a translucent LDAP proxy, or in an independent datastore keyed off of the UID provided via SAML assertion.
- As described above, in one embodiment API calls are secured via access and secret keys, which are used to sign API calls, along with traditional timestamps to prevent replay attacks. The APIs can be logically grouped into sets that align with the following typical roles:
-
- Base User
- System Administrator
- Developer
- Network Administrator
- Project Administrator
- Group Administrator
- Cloud Administrator
- Security
- End-user/Third-party User
- In one currently preferred embodiment, System Administrators and Developers have the same permissions, Project and Group Administrators have the same permissions, and Cloud Administrators and Security have the same permissions. The End-user or Third-party User is optional and external, and may not have access to protected resources, including APIs. Additional granularity of permissions is possible by separating these roles. In various other embodiments, the RBAC security system described above is extended with SAML Token passing. The SAML token is added to the API calls, and the SAML UID is added to the instance metadata, providing end-to-end auditability of ownership and responsibility.
- In an embodiment using the roles above, APIs can be grouped according to role. Any authenticated user may:
-
- Describe Instances
- Describe Images
- Describe Volumes
- Describe Keypairs
- Create Keypair
- Delete Keypair
- Create, Upload, Delete Buckets and Keys
- System Administrators, Developers, Project Administrators, and Group Administrators may:
-
- Create, Attach, Delete Volume (Block Store)
- Launch, Reboot, Terminate Instance
- Register/Unregister Machine Image (project-wide)
- Request or Review Audit Scans
- Project or Group Administrators may:
-
- Add and remove other users
- Set roles
- Manage groups
- Network Administrators may:
-
- Change Machine Image properties (public/private)
- Change Firewall Rules
- Define Cloud Security Groups
- Allocate, Associate, Deassociate Public IP addresses
- In this embodiment, Cloud Administrators and Security personnel would have all permissions. In particular, access to the audit subsystem would be restricted. Audit queries may spawn long-running processes, consuming resources. Further, detailed system information is a system vulnerability, so proper restriction of audit resources and results would be restricted by role.
- In an embodiment as described above, APIs are extended with three additional type declarations, mapping to the “Confidentiality, Integrity, Availability” (“C.I.A.”) classifications of FIPS 199. These additional parameters would also apply to creation of block storage volumes and creation of object storage “buckets.” C.I.A. classifications on a bucket would be inherited by the keys within the bucket. Establishing declarative semantics for individual API calls allows the cloud environment to seamlessly proxy API calls to external, third-party vendors when the requested C.I.A. levels match.
- In one embodiment, a hybrid or multi-vendor cloud uses the networking architecture described relative to
FIG. 4 and the RBAC controls to manage and secure inter-cluster networking. In this way the hybrid cloud environment provides dedicated, potentially co-located physical hardware with a network interconnect to the project or users' cloud virtual network. User-specific security policies are implemented as plugins on the network without necessarily disclosing the underlying implementation or specific business rules implemented by the plugin. - In one embodiment, the interconnect is a bridged VPN connection. In one embodiment, there is a VPN server at each side of the interconnect with a unique shared certificate. A security group is created specifying the access at each end of the bridged connection. In a second embodiment, the interconnect VPN implements audit controls so that the connections between each side of the bridged connection can be queried and controlled. Network discovery protocols (ARP, CDP) can be used to provide information directly, and existing protocols (SNMP location data, DNS LOC records) overloaded to provide audit information.
- Referring now to
FIG. 8 , a diagram showing the logical structure of an object storage service is shown atreference 800. The structure connects a user 802 through aproxy 804, where theproxy 804 provides an API endpoint for interacting with thestorage service 800. Theproxy 804 is connected to one ormore rings 806 such as anobject ring 806 a, acontainer ring 806 b, and anaccount ring 806 c, described in further detail below, that are connected to anobject service 808,container service 810, and anaccount service 812, respectively, described in further detail below. In other embodiments, there are other types of objects managed by rings, such as a structured data ring, a graph storage ring, or another type of ring (not pictured). In such embodiments, each ring would be connected to an appropriate service, such as a structured data service, a graph service, or another service (not pictured). - In one embodiment, these rings and services are implemented as services integrated with or running on top of an
information processing system 210 as described relative toFIG. 2 . These could be running alone, or configured to host one or more virtual machines; the services and rings can be implemented on “bare metal,” or within one or more of the virtual machines (logical containers and operating environments). The services and functionalities described relative to the object storage service may be centralized onto a smaller number of physical devices or may be expanded out to a large number of independent physical machines, including machines in different zones. For ease of explanation, two groupings are shown, wherein astorage management server 816 centralizes theproxy 804 and therings 806, and astorage pool server 818 centralizes theobject service 808, the container service, 810, theaccount service 812, and the storage pools 814. This grouping is for convenience in explanation only, however, and is not intended to be limiting. Further, each of these groupings may be logically present in a particular embodiment but may be implemented using a plurality of physical devices in a configuration that is different from the logical configuration presented. - In one embodiment, each
storage pool 814 is provided by aseparate storage server 818 or includes a virtual server that is included in a portion of one of thestorage servers 818 or across a plurality of thestorage servers 818. For example, thestorage servers 818 may be physically located in one or more data centers, and the resources of thestorage servers 818 may be virtualized according to the requirements of a plurality of users (e.g., the user 802) such that the plurality ofstorage pools 814 are provided to the plurality of users in order to store files and/or data objects. Thus, resources for a particular virtual server or storage pool may span acrossmultiple storage servers 818. - Referring now to
FIG. 9 , a multi-zone object storage service is shown atreference 840. The multi-zoneobject storage service 840 encompasses multiple zones 850 a-850 n. These zones may be included in separate regions, such as theexemplary regions FIG. 8 , possibly including in each object storage service a proxy 854 a-n, one or more rings 856 a-n, object, container, account, or other services 858 a-n, and a storage pool 860 a-n. In one embodiment, the user 802 interacts with each zone independently, addressing any requests directly to the proxies 854 a-n. In a second embodiment of the multi-zoneobject storage service 840, there is an additionalmulti-zone proxy 862 ormulti-zone ring 864. Themulti-zone proxy 862 is used to provide a single entry point to the zones 850 a-n, and themulti-zone ring 864 is used to balance requests across the zones 850 a-n. An embodiment may use either amulti-zone proxy 862, or amulti-zone ring 864, or both, or neither. In an embodiment in which amulti-zone proxy 862 or amulti-zone ring 864 is used, the individual zones 850 a-n can optionally forego the use of the zone-specific proxy 854 a-n or zone-specific ring 856 a-n. - Referring now to
FIG. 10 , a particular user 802 interacts with thestorage service 800 via the API to store and receive data objects. User 802 may create a plurality ofcontainers 1002 in the user account 1000 and store a plurality ofdata objects 1004 in each of thecontainers 1002 for retrieval. In the discussion below, a user account is referred to as an “account,” a container is referred to as a “container,” and a data object is referred to as an “object” for clarity of discussion. One of skill in the art will recognize that the terms “account,” “container” and “object” are generic forms of data naming that are used to direct theobject storage service 800 to a specific data object. When other types of rings and services are used, an appropriate name may be substituted. For clarity, discussion of alternative rings and services will be limited to the “account,” “container” and “object” rings and services. The account as shown inFIG. 10 , as well as other rings and services, are consistent whether or not they are deployed on a logical structure within a single cluster, such as the structure illustrated inFIG. 10 , or are arrayed across a multi-zone system, such as the structure illustrated inFIG. 9 . When reference is made to theproxy 804, therings 806, theservices proxy 804 and therings 806 are contemplated relative to themulti-zone proxy 862 and themulti-cluster ring 864. - The components of the exemplary
object storage service 800 and some of their functions will now be described in detail relative to various embodiments. - As discussed above, the
rings 806 are implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that software is used to implement the rings, it may include software that is stored on a computer-readable medium location in thestorage management server 816 and/or thestorage servers 818. Referring back toFIG. 2 , therings 806 include semiconductor circuits and/or computer-executable instructions that, when executed by a processor, provide subsystems of theobject storage service 800 that provide a mapping between the entities stored in theobject storage service 800 and the locations of those entities in the storage pools 814. In the illustrated embodiment, theobject storage service 800 includes aseparate object ring 806 a,container ring 806 b, andaccount ring 806 c, and when components of theobject storage service 800 need to perform any operation on an object, container, or account, those components interact with theobject ring 806 a,container ring 806 b, andaccount ring 806 c, respectively, to determine the location of that stored entity in the storage pools 814. However, one of skill in the art will recognize that different ring structures may be provided (e.g., a single ring for the objects, containers, and accounts, more than one ring for each of the objects, containers, and account, etc.) without departing from the scope of the present disclosure. Therings 806 maintain the availability and safety of data in theobject storage service 800 through the use of zones, partitions, replicas, and the storage pools 814, as described below. - In one embodiment, availability zones are defined across one or more of the storage pools 814 that are subject to a correlated loss of access or data as a result of a particular event. In addition, zones can be used for identification and management of other aspects of the system, such as the use of specific storage technology. One of skill in the art will recognize that the examples of zones provided above are not limiting, and a variety of zones known in the art will fall into the scope of the present disclosure.
- Logically, a partition is an abstract storage bucket. As discussed in further detail below, the
object storage service 800 maps each partition to a plurality ofstorage pools 814 that are in different zones, and stores data using those partitions. The mapping of a given partition to a plurality ofstorage pools 814 creates a plurality of partition replicas of that partition (e.g., equal to the number ofstorage pools 814 the partition is mapped to.) For example, when a given partition is mapped to 3storage pools 814 that are in different zones, 3 partition replicas of that partition are created. - The
object ring 806 a for the management of objects will be described in detail below. However, one of skill in the art will recognize how the discussion may be applied to thecontainer ring 806 b, theaccount ring 806 c, and/or a ring for any other stored entity, without departing from the scope of the present disclosure. - In various replicated, network-based object storage services, an object from a user is received by a proxy. To determine where the object should be stored, some attribute of the object or the object data itself is hashed. If necessary, some attribute of the object is modified so that three different results are returned from the hashing function. The object is then replicated and stored in the storage pool corresponding to the number returned by the hash function.
- Under typical circumstances, a consistent hashing function is used as the hash function. The use of the consistent hashing function ensures that there will be minimal changes to the assigned storage pools given a change in membership due to adding or removing new storage pools.
- Although the consistent hashing function results in minimal changes to the storage location, sometimes the assignments made by the consistent hashing function or the rearrangements needed due to a change in membership may have undesirable storage characteristics. For example, such methods have been found to result in multiple object replicas for the same object being stored in one or more storage pools that are in the same zone. As discussed above, this is undesirable because then multiple (and possibly all) object replicas for the same object are subject to being lost as a result of a particular event. Alternatively, rebalancing the replicas due to a change in membership has been found to require the movement to two of the replicas 4% of the time, and the movement of all three
replicas 1% of the time. It is desirable to never have to move more than one replica at a time. - In one embodiment, the
object storage service 800 solves the problem of multiple object replicas for the same object being stored in storage pools that are in the same zone through the use of therings 806. Referring now toFIG. 11 , amethod 400 for storing stored entities is illustrated. At block 1102, an object is received by a user. In one embodiment, an object is received from the user 202 by theproxy 804. Themethod 1100 then proceeds to block 1104 where a partition identification is generated. In one embodiment, a consistent hash function is applied to the object received in block 1102, and the hash function returns a partition identification that corresponds to a partition. Themethod 1100 then proceeds to block 1106 where the partition associated with the partition identification is mapped to storage pools that are in different zones. This mapping function is constrained so that the physical location of the storage pools is required to have one or more desirable properties, such as having each partition replica in a separate zone. - There are various embodiments of the constrained mapping function. In one embodiment, the constrained mapping function is the output of a constraint satisfaction solver, in which the desired storage characteristics (such as the requirement that each replica of a partition be in a different availability zone) are inputs to the solving function. The solver then uses one or more search methodologies within the solution space to find a storage layout that maps partitions to storage pools in a desirable manner.
- In a second embodiment, a constrained mapping function is applied to portions of the partition identification (e.g., the portions of the partition identification that the constrained mapping function is applied to) may be bits of the output of the original hashing function is applied to the object. For example, the number of bits to which the constrained mapping function is applied may be known as the partition power, and 2 to the partition power may indicate the partition count. The constrained mapping function is designed to return a storage pool location for each portion of the partition identification to which it is applied, and the storage pool locations returned for a given partition identification will each correspond to
storage pools 814 in different zones. These storage pool locations are then associated with the partition identification. Thus, the partition corresponding to the partition identification is replicated multiple times in the object storage service 800 (i.e., a partition replica is included in each storage pool corresponding to the storage pool locations determined from the constrained mapping function.) Themethod 1100 then proceeds to block 1108 where the object is stored according to the partition. The object received by the user 802 in block 1102 of themethod 1100 may then be stored according to the partition corresponding to the partition identification, which results in multiple object replicas for the object being stored in storage pools that are in different zones in theobject storage service 800. In another embodiment, the constrained mapping function is used to determined storage pool locations that are in different zones for each partition prior to the object being received by the user 802, discussed in further detail below. - The output of the constrained mapping function signifies a particular storage pool where a replica of the partition should be stored. An example of this output is as follows: When an object is received from the user 802 at block 1102 of the
method 1100, and atblock 1104 of themethod 1100, a hash function is applied to the object. In one exemplary embodiment, the user 802 provides data including an account/container/object name to theproxy 804, and a hash function is applied to the account/container/object name as follows: - Hash function (account/container/object name)==123456789
Where 123456789 is the partition identification that is returned by the hash function. Atblock 1106 of themethod 1100, the partition mapping number may then be divided into 3 parts (e.g., the first three digits, the second three digits, and the third three digits of the partition identification,) and the constrained mapping function is applied to each of those parts:
Constrained mapping function (123) storage pool location (zone 1)
Constrained mapping function (456) storage pool location (zone 7)
Constrained mapping function (789) storage pool location (zone 3)
As discussed above, the constrained mapping function is designed to return the storage pool location (zone 1), storage pool location (zone 7), and storage pool location (zone 3) that correspond to storage pools that are in different zones (e.g.,zones 1, 3, and 7). The storage pools locations are then associated with the partition identification:
Partition identification: (storage pool location (zone 1), storage pool location (zone 7), storage pool location (zone 3)
Thus, the partition corresponding to the partition identification is replicated across storage pools that are in different zones (here,zones 1, 3, and 7.) Atblock 1108 of themethod 1100, the object received from the user 802 is then stored, using the partition corresponding to the partition identification, in each of the storage pools corresponding to the storage pool locations returned by the application of the constrained mapping function to portions of the partition identification. Thus, 3 replicas of the object received from the user 802 are stored in theobject storage service 800 in storage pools that are located in different zones (zones 1, 3, and 7.) In one embodiment, each of the storage pool locations are IP addresses, i.e., when each of the storage pools are separate storage servers. In one embodiment, the constrained mapping function is a hash function. However, one of skill in the art will recognize that a variety of functions may be used to ensure that each partition is mapped to storage pools that are in different zones without departing from the scope of the present disclosure. - In another embodiment, the constrained mapping function is applied to the
object storage service 800 before the object is received by the user 802 at block 1102 in order to accomplish the mapping of the partitions to storage pools described above with reference to block 1106 of themethod 1100. For example, the total number of partitions and the total number of storage servers/storage pools in theobject storage service 800 may (and typically will) be known. With that knowledge, the constrained mapping function is used to map each partition in theobject storage service 800 to a plurality of storage pools that are in different zones, and that information is stored in a constrained mapping database. For example, a constrained mapping database may include partitions mapped to storage pools such as: - Partition 1: storage pool location (zone 1), storage pool location (zone 2), storage pool location (zone 3)
Partition 2: storage pool location (zone 4), storage pool location (zone 5), storage pool location (zone 6)
Partition 3: storage pool location (zone 7), storage pool location (zone 8), storage pool location (zone 9) - In one embodiment, the output of the constrained mapping function can be saved for optimized lookup. For example, the saved output may be embodied in a file provided to each of the storage pools 814, or stored in a database that is available for the appropriate systems to query. If the saved output is contained within a file, the storage pools 814 may then periodically check the modification time of this file and reload their in-memory copies of the ring structure as needed.
- Thus, when an object is received from a user 802 at block 1102, the hash function is applied to that object to get the partition identification (e.g.,
partition block 1104, and then atblock 1106, the partition identification may then be used with the constrained mapping database to determine the corresponding partition and its associated storage pool locations. This embodiment allows the processing necessary to map partitions to multiple storage pools in different zones to be conducted before objects are received from users so that such processing does not have to be conducted each time an object is received from a user. - For example, referring now to
FIG. 12 , a method for building aring 806 is illustrated. Atblock 1202, an ideal number of partitions for each storage pool in the object storage service is determined. In one embodiment, the number of partitions that should ideally be assigned to eachstorage pool 814 is calculated based the weight (e.g., storage capacity) of eachstorage pool 814. For example, if the partition power is 20, thering 806 will have 1,048,576 (220) partitions. If there are 1,000storage pools 814 of equal weight, eachstorage pool 814 will ideally be assigned 1,048.576 partitions. This may be referred to as an ideal partition count, and in the example, eachstorage pool 814 starts off empty with a 1,048.576 ideal partition count. Themethod 1200 then proceeds to block 1204 where the storage pools are placed in a sorting order. In one embodiment, the storage pools 814 are placed in a sorting order based on their ideal partition count (e.g., highest to lowest), and this sorting order is maintained throughout themethod 1200 as partitions are assigned storage pools 814. Themethod 1200 then proceeds to block 1206 where partitions are assigned to storage pools based on their sorting order but with a zone restriction. In one embodiment, the partitions are assigned to thestorage pool 814 with the highest ideal partition count, but subject to the restriction that thestorage pool 814 to which a partition is being assigned is not in the same zone as anyother storage pool 814 that includes a partition replica for that same partition. Themethod 1200 then proceeds to block 1208 where the sorting order of the storage pools is adjusted. In one embodiment, once a partition is assigned to astorage pool 814, thatstorage pool 814 will have its ideal partition count decremented and thus thatstorage pool 814 is moved to a lower position in the sorting order, and themethod 1200 then returns to block 1206 to continue to assign partitions to storage pools based on their sorting order but with the zone restriction. In such a manner, each partition is assignedmultiple storage pools 814 in different zones, and thus objects received from users may have multiple object replicas stored in storage pools in different zones simply by associating those objects with the partitions. - As mentioned above, another problem relates to the rebalancing of object replicas stored in the object storage service due to changing membership (i.e., adding or subtracting storage servers or storage pools from the object storage service.) Such methods have been found to require the moving of multiple object replicas of the same object in response to a membership change, which is undesirable.
- In one embodiment, the mapping of partitions to multiple storage pools in different zones in the
object storage service 800 described above solves these problems. The use of the constrained mapping function to ensure that each partition is mapped to storage pools in different zones ensures that object replicas for the same object are never located instorage pools 814 that are in the same zone (i.e., because any given object received from a user is stored in a partition that is replicated in storage pools that are in different zones.) For example, with eachstorage server 816 defined as a separate zone, the addition or subtraction of a givenstorage server 816 from theobject storage service 800 thus can only effect one partition replica, and hence one object replica of a given object (i.e., because only one of the partition replica will ever be located on a storage server that is defined as a separate zone.) In similar fashion, the rebalancing associated with changing the zone membership can be accomplished without affecting more than one replica because each zone is guaranteed to only contain one replica of a given partition. - Periodically, partitions may need to be reassigned to
different storage pools 814, and the reassignment of partitions will result in the building of a new ring from an old ring. Such an event may occur due to the removal and/or addition of astorage pool 814 from the object storage service 800 (e.g., a membership change.) Referring now toFIG. 13 , amethod 1300 for reassigning partitions in response to the removal of a storage pool is illustrated. Themethod 1300 begins atblock 1302 where the ideal number of partitions for each storage pool is recalculated. In one embodiment, the ideal partition count for the storage pools 814 remaining in the object storage service 800 (subsequent to the removal of a storage pool) is recalculated. Themethod 1300 then proceeds to block 1304 where the storage pools are placed in a sorting order as described above with reference to block 504 of themethod 500. The method then proceeds to block 1306 where partitions to be reassigned are grouped. In one embodiment, a partition list for the partitions to be reassigned is created. For example, anystorage pools 814 that have been removed from thefilesystem 800 may have all their assigned partitions unassigned and added to the partition list, and anystorage pools 814 that have more partitions than their ideal partition count may have random partitions unassigned from them and added to the partition list (i.e., such that those storage pools have a number of partitions that is within a predetermined amount of their ideal partition count.) The partitions on the partition list may then be reassigned to thestorage pool 814 inblocks 1308 and 610 of themethod 1300 substantially as discussed above with reference toblocks 506 and 508 of themethod 500. In one embodiment, atblock 1308 of themethod 1300, whenever a partition is reassigned to astorage pool 814, the time of the reassignment is recorded. Reassignment times may be used when gathering partitions to reassign tostorage pools 814, such that no partition replica for a given partition is moved twice in a predetermined amount of time. However, such reassignment restrictions based on reassignment times may be ignored for partition replicas onstorage pools 814 that have been removed from theobject storage service 800, as removing astorage pool 814 only happens uponstorage pool 814/storage server 816 failure and thus requires the reassignment of the partitions. - In one embodiment, the
method 1300 is conducted periodically to help balance the amount of data stored bystorage pools 814 in theobject storage service 800. For example, thepartition reassignment method 1300 discussed above may repeated until eachstorage pool 814 is within a predetermined threshold of a predetermined storage capacity (e.g., within 1% of 130% storage capacity for that storage pool) or when it is determined that partition reassignment will not improve the balance of data stored by theobject storage service 800 by more than a predetermined amount. For example, if afirst storage server 816 includes 2TB of storage, asecond storage server 816 includes 4TB of storage, and athird storage server 816 includes 6TB of storage, data balancing may be conducted to ensure that each of thestorage servers 818 holds the same percentage of its storage capacity (i.e., thefirst storage server 816 holds 0.66TB of data, thesecond storage server 816 holds 1.33TB of data, and thethird storage server 816 holds 2TB of data such that each of thestorage servers 818 is at 33% of its storage capacity.) Weights may be applied tostorage servers 818 to balance the distribution of data on thestorage servers 818 in theobject storage service 800 to account for different storage capacities. In some embodiments, step 1310 adjusts the sorting order for efficiency, load sharing, or other reasons. - Those of skill in the art will recognize that the constrained mapping within the
rings 806 is modifiable to deal with more than issues of availability. Even though the embodiment above has been described in terms of availability zones, any type of zone can be used for the mapping from partition to storage device. - In some implementations the constrained mapping is extensible via a plugin mechanism. The constrained mapping function is a rule-based hash, and so it can be extended similarly to other rule-based plugin mechanisms described herein. In one embodiment, the
ring 806 has an API by which mapping rules can be set by user 802. In a second embodiment, a series of rules can be provided and loaded by thering 806. This loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module. When set of partitions is provided to the plugin, the constrained mapping function logically associates it with a particular resource and assigns the virtual partition to the applicable physical resource. When a request arrives addressed to that resource, the tenant-defined rules and filters are applied to the request and the appropriate object is uploaded, downloaded, edited, or redirected accordingly. In a further embodiment, the API provided to the user 802 is sufficient to for the user to direct the creation of the ring structure via interaction with a remote, user-controlled entity on the network. - There are two levels at which the user can direct the allocation of underlying resources via the plugin mechanism. At a first, high level, the user can present weights, correlations, or affinities between different groups of resources so that an underlying cloud-provider-specific ring generation function can take those into account. In a second embodiment, the user can have direct control over the allocation of storage resources and those can be mapped onto available physical resources provided by the cloud provider. Plugging into the constraint mapping portion of the ring generation function is sufficient for both levels of control, but it is anticipated that different cloud providers will favor or provide one level of control over another.
- As discussed above, the
object service 808 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium located in thestorage management server 816 and/or thestorage servers 818. Theobject service 808 may include instructions that, when executed by a processor, provide object storage and objection manipulation functionality such that theobject service 808 is operable to, for example, store, retrieve and delete stored objects in the storage pools 814. In one embodiment, anobject service 808 is provided for each storage pool that holds object data. For example, anobject service 808 may be included on a server that further includes one or more storage drives that provide a storage pool for objects. In one embodiment, the objects are stored as binary files with metadata stored as extended attributes of the file in the filesystem used by the object storage service. In such an embodiment, theobject service 808 will uses the extended attributes of the filesystem to manage the metadata. In a second embodiment, the metadata is stored in a machine-readable format next to the data itself. For example, the metadata for a file is stored in a text file or single file database. - In one embodiment, subsequent processing based on file metadata is also pluggable by the end user. For example, one embodiment stores different sets of files and associates them with the same URL. An ordinary operation to read or modify the object could also have a “processor” function associated with it that allows for user-directed modification or logging of the request.
- In one embodiment, objects are stored by the
object service 808 using a path derived by applying a hash function to the name of the object along with a timestamp. For example, an incoming object for a user account to be written to a container will have a hash applied to its account/container/object name and the path generated for the object is: - /objects/<partition>/<storage pool location>/objectname_hash.15673.data
where “objects” indicate that the object data is stored in anobject storage pool 814, <partition>is the partition identification that maps the object to a partition, <storage pool location>is the storage pool location that maps the partition replica to astorage pool 814 in a different zone than its related partition replicas, objectname_hash is the hash of the account/container/object name, and 15672 is the timestamp. - When there is a request for an object, the
object storage service 800 will find all the object replicas in theobject storage service 800 that include the objectname_hash and return the object data that has the most recent timestamp value. Special care is needed to record updates that should be persisted as the new canonical value. For example, when an object replica is deleted, a modification sentinel (e.g., a 0 byte “tombstone” file or “.ts” file) is written to thestorage pool 814 where the deleted object replica was located and that includes the same objectname_hash as the deleted object replica (i.e., /objectname_hash.15784.ts,) and that tombstone file stays in theobject storage service 800 for a predetermined amount of time (e.g., 7 days.) During object replication, discussed in further detail below, when theobject storage service 800 encounters a tombstone file, theobject storage service 800 checks whether the tombstone file has been in the system for 7 days. If not, theobject storage service 800 searches for and deletes any object replicas that it finds related to that tombstone file (e.g., replicas that same objectname_hash as the tombstone file) to ensure that objects that were meant to be deleted from theobject storage service 800 are removed and older versions of object replicas of a given object do not appear in theobject storage service 800 due to, for example, the temporary failure of astorage server 816 orstorage pool 814 that might have prevented the deletion of that object replica previously. If theobject storage service 800 determines that a tombstone file has been in theobject storage service 800 for longer than the predetermined time, that tombstone file is deleted. - The mechanism used for recording file deletion is also used to record other types of updates. For example, a “purge” marker indicates that the system should overwrite all copies of the object and set the space to free; a “version” marker indicates that the system should create a copy and mark the copy with a version number; and a “ttl” (time-to-live) marker indicates that the system should check an authoritative source for updates after the expiry of a set time period. Other types of out-of-band changes to the file are also contemplated.
- As discussed above, the
container service 810 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium located in thestorage management server 816 and/or thestorage servers 818. Thecontainer service 810 may include instructions that, when executed by a processor, provide container storage and container manipulation functionality such that thecontainer service 810 is operable to store, retrieve and delete stored containers in the storage pools 814. In one embodiment, acontainer service 810 is provided for each storage pool that holds container data. For example, acontainer service 810 may be included on a server that further includes one or more storage drives that provide a storage pool for containers, and thecontainer service 810 may include the names of containers and objects in those containers. Thus, in one embodiment, thecontainer service 810 handles the listing of containers, and does not hold the location where the objects are stored (e.g., the storage pool where a given object replica resides), but rather the locations of containers that hold the objects. The listings for the container locations may be stored as database files, and those listings may be replicated across the storage pools 814 in a manner that is similar to the replication of objects (e.g., through their association with partitions) as discussed above. Container storage statistics for the container service(s) 810 may be tracked by theobject storage service 800 and may include total number of objects stored by one or more containers, the total storage provided by any given container, and/or a variety of other statistics known in the art. - As discussed above, the
account service 812 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium located in thestorage management server 816 and/or thestorage servers 818. Theaccount service 812 may include instructions that, when executed by a processor, provide account storage and account manipulation functionality such that theaccount service 812 is operable to store, retrieve and delete stored accounts in the storage pools 814. In one embodiment, anaccount service 812 is provided for each storage pool that holds account data. For example, anaccount service 812 may be implemented by a server that includes storage drives that provide a storage pool for accounts, and theaccount service 812 may include the names of accounts and containers in those accounts. Thus, theaccount service 812 is very similar to thecontainer service 810, discussed above, with the exception that accountstorage 812 handles the listings of accounts. - As discussed above, other types of services may be implemented in similar fashion to the object, container, and account services described above. For example, one implementation includes an authorization service. The authorization service may include instructions that, when executed by a processor, handle the storage and manipulation of authorization metadata so that the authorization service is operable to store, retrieve, delete, and query stored credentials from in the storage pools 814. In one embodiment, an authorization service provides an ACL-based authorization. In a second embodiment, the authorization service provides posix-compatible authorization. In a third embodiment, the authorization service provides tree or graph-based authorization, such as would be provided with an LDAP-based authorization service.
- A second implementation includes a structured data service. The structured data service may include instructions that, when executed by a processor, provide the storage and manipulation of structured data such that the structured data service is operable to store, retrieve, delete, and query tabular, graph, or tree-based data from in the storage pools 814. In one embodiment, a structured data service provides a JSON (JavaScript Object Notation)-based output. In a second embodiment, the structured data service provides XML-based output. In a third embodiment, the structured data service provides HTML output.
- The
proxy 804 is implemented in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. Theproxy 804 is responsible for tying together theobject storage service 800. For each request received from a user, theproxy 804 determines the location of the account, container, or object in the appropriate ring 806 (e.g., theobject ring 806 a, thecontainer ring 806 b, or theaccount ring 806 c,) and routes the request accordingly. A public Application Programming Interface (API) may be exposed to users through theproxy 804. A large number of failures may be handled by theproxy 804. For example, if astorage server 816 and/orstorage pool 814 is unavailable for an object PUT, theproxy 804 may use therings 806 to determine anappropriate storage server 816 and/orstorage pool 814 for that object and route the object there instead. - In another embodiment, there are multiple proxies associated with an object storage service. The existence of multiple proxies may be ascertainable from outside the object storage service, or it may be transparent. Different proxies may be used for different purposes. For example, in one embodiment different proxies are used for different types of files. In another embodiment, different proxies are used for different types of requests. In a third embodiment, an appropriate proxy is chosen to minimize latency, geographic, or network distance between the proxy and the system making the request.
-
FIG. 14 is a functional block diagram of a virtual machine (VM)image service 1400 according to various aspects of the current disclosure. Generally, theVM image service 1400 is an IaaS-style cloud computing system for registering, storing, and retrieving virtual machine images and associated metadata. In a preferred embodiment, theVM image service 1400 is deployed as a service resource 130 in the cloud computing system 110 (FIG. 1 ). Theservice 1400 presents to clients of thecloud computing system 110 an endpoint from which the clients may store, lookup, and retrieve virtual machine images on demand. - As shown in the illustrated embodiment of
FIG. 14 , theVM image service 1400 comprises a component-based architecture that includes anAPI server 1402, animage store 1404, and aregistry server 1406. TheAPI server 1402 is a communication hub that routes VM image requests and data betweenclients 1408 a-n and theimage store 1404 andregistry server 1406. Theimage store 1404 is an autonomous and extensible storage resource that stores VM images managed by theservice 1400, and theregistry server 1406 is a componentized service that stores metadata associated with the managed VM images. In the illustrated embodiment, theAPI server 1402 includes an “external”API endpoint 1410 through which theclients 1408 a-n may programmatically access VM images managed by theservice 1400. In that regard, theAPI endpoint 1410 exposes both metadata about managed VM images and the image data itself to requesting clients. In one embodiment, theAPI endpoint 1410 is implemented with an RPC-style system, such as CORBA, DCE/COM, SOAP, or XML-RPC, and adheres to the calling structure and conventions defined by these respective standards. In another embodiment, theexternal API endpoint 1410 is a basic HTTP web service adhering to a representational state transfer (REST) style and may be identifiable via a URL. Specific functionality of theAPI endpoint 1410 will be described in greater detail below. - The
API server 1402 may be implemented in software or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that software is used to implement theAPI server 1402, it may include software that is stored on a non-transitory computer-readable medium in an information processing system, such as theinformation processing system 210 ofFIG. 2 . - As mentioned above, the
VM image service 1400 stores VM images in theimage store 1404. In the illustrated embodiment, theimage store 1404 is any local or remote storage resource that is programmatically accessible by an “internal” API endpoint within theAPI server 1402. In one embodiment, theimage store 1404 may simply be afile system storage 1412 a that is physically associated with theAPI server 1402. In such an embodiment, theAPI server 1402 includes a filesystem API endpoint 1414 a that communicates natively with thefile system storage 1412 a. The filesystem API endpoint 1414 a conforms to a standardized storage API for reading, writing, and deleting VM image data. Thus, when aclient 1408 requests a VM image that is stored in thefile system storage 1412 a, theAPI server 1402 makes an internal API call to the filesystem API endpoint 1414 a, which, in turn, sends a read command to the file system storage. In other embodiments, theimage store 1404 may be implemented with anoutside cloud storage 1412 b, objectstorage service storage 1412 c, and/orHTTP storage 1412 n that are respectively associated with an outsidecloud service endpoint 1414 b, objectstorage service endpoint 1414 c, andHTTP endpoint 1414 n on theAPI server 1402. In one embodiment, theHTTP storage 1412 n may comprise a URL that points to a virtual machine image hosted somewhere on the Internet, and thus may be read-only. It is understood that any number of additional storage resources, such as Sheepdog, a RADOS block device (RBD), a storage area network (SAN), and any other programmatically accessible storage solutions, may be provisioned as theimage store 1404. Further, in some embodiments, multiple storage resources may be simultaneously available as image stores withinservice 1400 such that theAPI server 1402 may select a specific storage option based on the size, availability requirements, etc of a VM image. Accordingly, theimage store 1404 provides theimage service 1400 with redundant, scalable, and/or distributed storage for VM images. - Additionally, in some embodiments, the
API server 1402 may include a local image cache that temporarily stores VM image data. In such a scenario, if a client requests a VM image that is held in the image cache, the API server can distribute the VM image to the client without having to retrieve the image from theimage store 1404. Locally caching VM images on the API server not only decreases response time but it also enhances the scalability of theVM image service 1400. For example, in one embodiment, theimage service 1400 may include a plurality of API servers, where each may cache the same VM image and simultaneously distribute portions of the image to a client. - The
registry server 1406 stores and publishes VM image metadata corresponding to VM images stored by thesystem 1400 in theimage store 1404. In one embodiment, each VM image managed by theservice 1400 includes at least the following metadata properties stored in the registry server 1406: UUID, name, status of the image, disk format, container format, size, public availability, and user-defined properties. Additional and/or different metadata may be associated with VM images in alternative embodiments. Theregistry server 1406 includes aregistry database 1416 in which the metadata is stored. In one embodiment, theregistry database 1416 is a relational database such as MySQL, but, in other embodiments, it may be a non-relational structured data storage system like MongoDB, Apache Cassandra, or Redis. For standardized communication with theAPI server 1402, theregistry server 1406 includes aregistry API endpoint 1418. Theregistry API endpoint 1418 is a RESTful API that programmatically exposes the database functions to theAPI server 1402 so that the API server may query, insert, and delete VM image metadata upon receiving requests from clients. In one embodiment, theregistry server 1406 may be any public or private web service that exposes the RESTful API to theAPI server 1402. In alternative embodiments, theregistry server 1402 may be implemented on a dedicated information processing system of may be a software component stored on a non-transitory computer-readable medium in the same information processing system as theAPI server 1402. - In operation,
clients 1408 a-n exploit theexternal API endpoint 1410 exposed by theAPI server 1402 to lookup, store, and retrieve VM images managed by theVM image service 1400. In the example embodiment described below, clients may issue HTTP GETs, PUTs, POSTs, and HEADs to communicate with theAPI server 1402. For example, a client may issue a GET request to <API_server_URL>/images/ to retrieve the list of available public images managed by theimage service 1400. Upon receiving the GET request from the client, the API server sends a corresponding HTTP GET request to theregistry server 1406. In response, theregistry server 1406 queries theregistry database 1416 for all images with metadata indicating that they are public. Theregistry server 1406 returns the image list to theAPI server 1402 which forwards it on to the client. For each image in the returned list, the client may receive a JSON-encoded mapping containing the following information: URI, name, disk_format, container format, and size. As another example, a client may retrieve a virtual machine image from theservice 1400 by sending a GET request to <API_server_URL>/imageskimage_URI>. Upon receipt of the GET request, theAPI server 1404 retrieves the VM image data from theimage store 1404 by making an internal API call to one of the storage API endpoints 1414 a-n and also requests the metadata associated with the image from theregistry server 1406. TheAPI server 1402 returns the metadata to the client as a set of HTTP headers and the VM image as data encoded into the response body. Further, to store a VM image and metadata in theservice 1400, a client may issue a POST request to <API_server_URL>/images/ with the metadata in the HTTP header and the VM image data in the body of the request. Upon receiving the POST request, theAPI server 1402 issues a corresponding POST request to theregistry API endpoint 1418 to store the metadata in theregistry database 1416 and makes an internal API call to one of the storage API endpoints 1414 a-n to store the VM image in theimage store 1404. It should be understood that the above is an example embodiment and communication via the API endpoints in theVM image service 1400 may be implemented in various other manners, such as through non-RESTful HTTP interactions, RPC-style communications, internal function calls, shared memory communication, or other communication mechanisms. - In one embodiment, the
API server 1402 includes a plugin mechanism that allows for a user-configurable response to image requests. For example, in one embodiment sensitive security information is held by a user outside the scope of the cloud computing system. When a request is made for a generic VM image, the plugin creates a secure connection to a user-controlled endpoint and receives the necessary information from the user's system. The plugin then injects the custom authentication information into the image prior to the image being instantiated. Other embodiments can modify the image or the image request in arbitrary user-defined ways. - Further, in some embodiments, the
VM image service 1400 may include security features such as an authentication manager to authenticate and manage user, account, role, project, group, quota, and security group information associated with the managed VM images. For example, an authentication manager may filter every request received by theAPI server 1402 to determine if the requesting client has permission to access specific VM images. In some embodiments, Role-Based Access Control (RBAC) may be implemented in the context of theVM image service 1400, whereby a user's roles defines the API commands that user may invoke. For example, certain API calls to theAPI server 1402, such as POST requests, may be only associated with a specific subset of roles. - To the extent that some components described relative to the
VM image service 1400 are similar to components of the largercloud computing system 110, those components may be shared between the cloud computing system and the VM image service, or they may be completely separate. Further, to the extend that “controllers,” “nodes,” “servers,” “managers,” “VMs,” or similar terms are described relative to theVM image service 1400, those can be understood to comprise any of a singleinformation processing device 210 as described relative toFIG. 2 , multipleinformation processing devices 210, a single VM as described relative toFIG. 2 , a group or cluster of VMs or information processing devices as described relative toFIG. 3 . These may run on a single machine or a group of machines, but logically work together to provide the described function within the system. - Turning now to
FIG. 15 , an IaaS-style computational cloud service (a “compute” service) is shown at 1500 according to one embodiment. This is one embodiment of a cloud controller 120 with associated cloud service 130 as described relative toFIG. 1 . Except as described relative to specific embodiments, the existence of a compute service does not require or prohibit the existence of other portions of thecloud computing system 110 nor does it require or prohibit the existence of other cloud controllers 120 with other respective services 130. - To the extent that some components described relative to the
compute service 1500 are similar to components of the largercloud computing system 110, those components may be shared between thecloud computing system 110 and thecompute service 1500, or they may be completely separate. Further, to the extend that “controllers,” “nodes,” “servers,” “managers,” “VMs,” or similar terms are described relative to thecompute service 1500, those can be understood to comprise any of a singleinformation processing device 210 as described relative toFIG. 2 , multipleinformation processing devices 210, a single VM as described relative toFIG. 2 , a group or cluster of VMs or information processing devices as described relative toFIG. 3 . These may run on a single machine or a group of machines, but logically work together to provide the described function within the system. - In one embodiment,
compute service 1500 includes anAPI Server 1510, aCompute Controller 1520, anAuth Manager 1530, anObject Store 1540, aVolume Controller 1550, aNetwork Controller 1560, and aCompute Manager 1570. These components are coupled by a communications network of the type previously described. In one embodiment, communications between various components are message-oriented, using HTTP or a messaging protocol such as AMQP, ZeroMQ, or STOMP. - Although various components are described as “calling” each other or “sending” data or messages, one embodiment makes the communications or calls between components asynchronous with callbacks that get triggered when responses are received. This allows the system to be architected in a “shared-nothing” fashion. To achieve the shared-nothing property with multiple copies of the same component,
compute service 1500 further includes distributeddata store 1590. Global state forcompute service 1500 is written into this store using atomic transactions when required. Requests for system state are read out of this store. In some embodiments, results are cached within controllers for short periods of time to improve performance. In various embodiments, the distributeddata store 1590 can be the same as, or share the same implementation asObject Store 800 discussed relative toFIG. 8 . - In one embodiment, the
API server 1510 includesexternal API endpoints 1512. In one embodiment, theexternal API endpoints 1512 are provided over an RPC-style system, such as CORBA, DCE/COM, SOAP, or XML-RPC. These follow the calling structure and conventions defined in their respective standards. In another embodiment, theexternal API endpoints 1512 are basic HTTP web services following a REST pattern and identifiable via URL. Requests to read a value from a resource are mapped to HTTP GETs, requests to create resources are mapped to HTTP PUTs, requests to update values associated with a resource are mapped to HTTP POSTs, and requests to delete resources are mapped to HTTP DELETEs. In some embodiments, other REST-style verbs are also available, such as the ones associated with WebDay. In a third embodiment, theAPI endpoints 1512 are provided via internal function calls, IPC (Inter-process Communication), or a shared memory mechanism. Regardless of how the API is presented, theexternal API endpoints 1512 are used to handle authentication, authorization, and basic command and control functions using various API interfaces. In one embodiment, the same functionality is available via multiple APIs, including APIs associated with other cloud computing systems. This enables API compatibility with multiple existing tool sets created for interaction with offerings from other vendors. - The
Compute Controller 1520 coordinates the interaction of the various parts of thecompute service 1500. In one embodiment, the various internal services that work together to provide thecompute service 1500, are internally decoupled by adopting a service-oriented architecture (SOA). TheCompute Controller 1520 serves as an internal API server, allowing the various internal controllers, managers, and other components to request and consume services from the other components. In one embodiment, all messages pass through theCompute Controller 1520. In a second embodiment, theCompute Controller 1520 brings up services and advertises service availability, but requests and responses go directly between the components making and serving the request. In a third embodiment, there is a hybrid model in which some services are requested through theCompute Controller 1520, but the responses are provided directly from one component to another. - In one embodiment, communication to and from the
Compute Controller 1520 is mediated via one or moreinternal API endpoints 1522, provided in a similar fashion to those discussed above. Theinternal API endpoints 1522 differ from theexternal API endpoints 1512 in that theinternal API endpoints 1522 advertise services only available within theoverall compute service 1500, whereas theexternal API endpoints 1512 advertise services available outside thecompute service 1500. There may be one or moreinternal APIs 1522 that correspond toexternal APIs 1512, but it is expected that there will be a greater number and variety of internal API calls available from theCompute Controller 1520. - In one embodiment, the
Compute Controller 1520 includes aninstruction processor 1524 for receiving and processing instructions associated with directing thecompute service 1500. For example, in one embodiment, responding to an API call involves making a series of coordinated internal API calls to the various services available within thecompute service 1500, and conditioning later API calls on the outcome or results of earlier API calls. Theinstruction processor 1524 is the component within theCompute Controller 1520 responsible for marshalling arguments, calling services, and making conditional decisions to respond appropriately to API calls. - In one embodiment, the
instruction processor 1524 is implemented as described above relative toFIG. 3 , specifically as a tailored electrical circuit or as software instructions to be used in conjunction with a hardware processor to create a hardware-software combination that implements the specific functionality described herein. To the extent that one embodiment includes computer-executable instructions, those instructions may include software that is stored on a computer-readable medium. Further, one or more embodiments have associated with them a buffer. The buffer can take the form of data structures, a memory, a computer-readable medium, or an off-script-processor facility. For example, one embodiment uses a language runtime as aninstruction processor 1524, running as a discrete operating environment, as a process in an active operating environment, or can be run from a low-power embedded processor. In a second embodiment, theinstruction processor 1524 takes the form of a series of interoperating but discrete components, some or all of which may be implemented as software programs. In another embodiment, theinstruction processor 1524 is a discrete component, using a small amount of flash and a low power processor, such as a low-power ARM processor. In a further embodiment, the instruction processor includes a rule engine as a submodule as described herein. - In one embodiment, the
Compute Controller 1520 includes a message queue as provided bymessage service 1526. In accordance with the service-oriented architecture described above, the various functions within thecompute service 1500 are isolated into discrete internal services that communicate with each other by passing data in a well-defined, shared format, or by coordinating an activity between two or more services. In one embodiment, this is done using a message queue as provided bymessage service 1526. Themessage service 1526 brokers the interactions between the various services inside and outside theCompute Service 1500. - In one embodiment, the
message service 1526 is implemented similarly to the message service described relative toFIGS. 6a -6 c. Themessage service 1526 may use themessage service 140 directly, with a set of unique exchanges, or may use a similarly configured but separate service. - The
Auth Manager 1530 provides services for authenticating and managing user, account, role, project, group, quota, and security group information for thecompute service 1500. In a first embodiment, every call is necessarily associated with an authenticated and authorized entity within the system, and so is or can be checked before any action is taken. In another embodiment, internal messages are assumed to be authorized, but all messages originating from outside the service are suspect. In this embodiment, the Auth Manager checks the keys provided associated with each call received overexternal API endpoints 1512 and terminates and/or logs any call that appears to come from an unauthenticated or unauthorized source. In a third embodiment, theAuth Manager 1530 is also used for providing resource-specific information such as security groups, but the internal API calls for that information are assumed to be authorized. External calls are still checked for proper authentication and authorization. Other schemes for authentication and authorization can be implemented by flagging certain API calls as needing verification by theAuth Manager 1530, and others as needing no verification. In one embodiment, theAuth Manager 1530 is implemented similarly to or as a proxy to authn/authz service 700 as described relative toFIG. 7 . - In one embodiment, external communication to and from the
Auth Manager 1530 is mediated via one or more authentication andauthorization API endpoints 1532, provided in a similar fashion to those discussed above. The authentication andauthorization API endpoints 1532 differ from theexternal API endpoints 1512 in that the authentication andauthorization API endpoints 1532 are only used for managing users, resources, projects, groups, and rules associated with those entities, such as security groups, RBAC roles, etc. In another embodiment, the authentication andauthorization API endpoints 1532 are provided as a subset ofexternal API endpoints 1512. - In one embodiment, the
Auth Manager 1530 includes arules processor 1534 for processing the rules associated with the different portions of thecompute service 1500. In one embodiment, this is implemented in a similar fashion to theinstruction processor 1524 described above. - The
Object Store 1540 provides redundant, scalable object storage capacity for arbitrary data used by other portions of thecompute service 1500. At its simplest, theObject Store 1540 can be implemented one or more block devices exported over the network. In a second embodiment, theObject Store 1540 is implemented as a structured and possibly distributed data organization system. Examples include relational database systems—both standalone and clustered—as well as non-relational structured data storage systems like MongoDB, Apache Cassandra, or Redis. In a third embodiment, theObject Store 1540 is implemented as a redundant, eventually consistent, fully distributed data storage service, such as theobject storage service 800 described relative toFIG. 8 . - In one embodiment, external communication to and from the
Object Store 1540 is mediated via one or more objectstorage API endpoints 1542, provided in a similar fashion to those discussed above. In one embodiment, the objectstorage API endpoints 1542 are internal APIs only. In a second embodiment, theObject Store 1540 is provided by a separate cloud service 130, so the “internal” API used forcompute service 1500 is the same as the external API provided by the object storage service itself. - In one embodiment, the
Object Store 1540 includes anImage Service 1544. TheImage Service 1544 is a lookup and retrieval system for virtual machine images. In one embodiment, various virtual machine images can be associated with a unique project, group, user, or name and stored in theObject Store 1540 under an appropriate key. In this fashion multiple different virtual machine image files can be provided and programmatically loaded by thecompute service 1500. In one embodiment, theimage service 1544 is implemented asimage service 1400, as described relative toFIG. 14 . - The
Volume Controller 1550 coordinates the provision of block devices for use and attachment to virtual machines. In one embodiment, theVolume Controller 1550 includesVolume Workers 1552. TheVolume Workers 1552 are implemented as unique virtual machines, processes, or threads of control that interact with one or morebackend volume providers 1554 to create, update, delete, manage, and attach one ormore volumes 1556 to a requesting VM. - In a first embodiment, the
Volume Controller 1550 is implemented using a SAN that provides a sharable, network-exported block device that is available to one or more VMs, using a network block protocol such as iSCSI. In this embodiment, theVolume Workers 1552 interact with the SAN to manage and iSCSI (Internet Small Computer System Interface) storage to manage LVM-based (Logical Volume Manager-based) instance volumes, stored on one or more smart disks or independent processing devices that act asvolume providers 1554 using their embeddedstorage 1556. In a second embodiment,disk volumes 1556 are stored in theObject Store 1540 as image files under appropriate keys. TheVolume Controller 1550 interacts with theObject Store 1540 to retrieve adisk volume 1556 and place it within an appropriate logical container on the same information processing system 240 that contains the requesting VM. An instruction processing module acting in concert with the instruction processor and hypervisor on the information processing system 240 acts as thevolume provider 1554, managing, mounting, and unmounting thevolume 1556 on the requesting VM. In a further embodiment, thesame volume 1556 may be mounted on two or more VMs, and a block-level replication facility may be used to synchronize changes that occur in multiple places. In a third embodiment, theVolume Controller 1550 acts as a block-device proxy for theObject Store 1540, and directly exports a view of one or more portions of theObject Store 1540 as a volume. In this embodiment, the volumes are simply views onto portions of theObject Store 1540, and theVolume Workers 1554 are part of the internal implementation of theObject Store 1540. - In one embodiment, the
Network Controller 1560 manages the networking resources for VM hosts managed by thecompute manager 1570. Messages received byNetwork Controller 1560 are interpreted and acted upon to create, update, and manage network resources for compute nodes within the compute service, such as allocating fixed IP addresses, configuring VLANs for projects or groups, or configuring networks for compute nodes. - In one embodiment, the
Network Controller 1560 is implemented similarly to thenetwork service 400 described relative toFIG. 4 . Thenetwork controller 1560 may use a shared cloud controller directly, with a set of unique addresses, identifiers, and routing rules, or may use a similarly configured but separate service. In another embodiment - In one embodiment, the
Compute Manager 1570 manages computing instances for use by API users using thecompute service 1500. In one embodiment, theCompute Manager 1570 is coupled to a plurality of resource pools 1572, each of which includes one ormore compute nodes 1574. Eachcompute node 1574 is a virtual machine management system as described relative toFIG. 3 and includes acompute worker 1576, a module working in conjunction with the hypervisor and instruction processor to create, administer, and destroy multiple user- or system-defined logical containers and operating environments—VMs—according to requests received through the API. In various embodiments, the compute nodes may be organized into clusters, such as clusters 1572 a and 1572 b. In one embodiment, each resource pool 1572 is physically located in one or more data centers in one or more different locations. In another embodiment, resource pools have different physical or software resources, such as different available hardware, higher-throughput network connections, or lower latency to a particular location. - In one embodiment, the
Compute Manager 1570 allocates VM images toparticular compute nodes 1574 via aScheduler 1578. TheScheduler 1578 is a matching service; requests for the creation of new VM instances come in and the mostapplicable Compute nodes 1574 are selected from the pool of potential candidates. In one embodiment, theScheduler 1578 selects acompute node 1574 using a random algorithm. Because the node is chosen randomly, the load on any particular node tends to be non-coupled and the load across all resource pools tends to stay relatively even. - In a second embodiment, a
smart scheduler 1578 is used. A smart scheduler analyzes the capabilities associated with a particular resource pool 1572 and its component services to make informed decisions on where a new instance should be created. When making this decision it consults not only all the Compute nodes across the resource pools 1572 until the ideal host is found. - In a third embodiment, a distributed
scheduler 1578 is used. A distributed scheduler is designed to coordinate the creation of instances acrossmultiple compute services 1500. Not only does the distributedscheduler 1578 analyze the capabilities associated with the resource pools 1572 available to thecurrent compute service 1500, it also recursively consults the schedulers of any linked compute services until the ideal host is found. - In one embodiment, either the smart scheduler or the distributed scheduler is implemented using
rules engine 1579 and a series of associated rules regarding costs and weights associated with desired compute node characteristics. When deciding where to place an Instance, rulesengine 1579 compares a Weighted Cost for each node. In one embodiment, the Weighting is just the sum of the total Costs. In a second embodiment, a Weighting is calculated using an exponential or polynomial algorithm. In the simplest embodiment, costs are nothing more than integers along a fixed scale, although costs can also be represented by floating point numbers, vectors, or matrices. Costs are computed by looking at the various Capabilities of the available node relative to the specifications of the Instance being requested. The costs are calculated so that a “good” match has lower cost than a “bad” match, where the relative goodness of a match is determined by how closely the available resources match the requested specifications. - In one embodiment, specifications can be hierarchical, and can include both hard and soft constraints. A hard constraint is a constraint that cannot be violated and have an acceptable response. This can be implemented by having hard constraints be modeled as infinite-cost requirements. A soft constraint is a constraint that is preferable, but not required. Different soft constraints can have different weights, so that fulfilling one soft constraint may be more cost-effective than another. Further, constraints can take on a range of values, where a good match can be found where the available resource is close, but not identical, to the requested specification. Constraints may also be conditional, such that constraint A is a hard constraint or high-cost constraint if Constraint B is also fulfilled, but can be low-cost if Constraint C is fulfilled.
- As implemented in one embodiment, the constraints are implemented as a series of rules with associated cost functions. These rules can be abstract, such as preferring nodes that don't already have an existing instance from the same project or group. Other constraints (hard or soft), may include: a node with available GPU hardware; a node with an available network connection over 100 Mbps; a node that can run Windows instances; a node in a particular geographic location, etc.
- When evaluating the cost to place a VM instance on a particular node, the constraints are computed to select the group of possible nodes, and then a weight is computed for each available node and for each requested instance. This allows large requests to have dynamic weighting; if 1000 instances are requested, the consumed resources on each node are “virtually” depleted so the Cost can change accordingly.
- In a further embodiment, the scheduler can be directly accessed or modified using an API and/or a plugin interface. In one embodiment, a plugin is implemented as a series of loadable and executable rules that can be provided to the
compute service 1500 and run by the scheduler as part of its weighting or allocation process. In one embodiment, this loadable ruleset can be implemented using a dynamically loadable or runnable program that has a series of expected inputs and provides its information on a defined series of outputs. In one embodiment these are “stdin” and “stdout.” In a second embodiment, these are provided as function calls within a program module. - Turning now to
FIG. 16 , a diagram showing one embodiment of the process of instantiating and launching a VM instance is shown as diagram 1600. Although the implementation of the image instantiating and launching process will be shown in a manner consistent with the embodiment of thecompute service 1500 as shown relative toFIG. 15 , the process is not limited to the specific functions or elements shown inFIG. 15 . For clarity of explanation, internal details not relevant to diagram 1600 have been removed from the diagram relative toFIG. 15 . Further, while some requests and responses are shown in terms of direct component-to-component messages, in at least one embodiment the messages are sent via a message service, such asmessage service 1526 as described relative toFIG. 15 . - At
time 1602, theAPI Server 1510 receives a request to create and run an instance with the appropriate arguments. In one embodiment, this is done by using a command-line tool that issues arguments to theAPI server 1510. In a second embodiment, this is done by sending a message to theAPI Server 1510. In one embodiment, the API to create and run the instance includes arguments specifying a resource type, a resource image, and control arguments. A further embodiment includes requester information and is signed and/or encrypted for security and privacy. Attime 1604,API server 1510 accepts the message, examines it for API compliance, and relays a message to ComputeController 1520, including the information needed to service the request. In an embodiment in which user information accompanies the request, either explicitly or implicitly via a signing and/or encrypting key or certificate, theCompute Controller 1520 sends a message toAuth Manager 1530 to authenticate and authorize the request attime 1606 andAuth Manager 1530 sends back a response toCompute Controller 1520 indicating whether the request is allowable attime 1608. If the request is allowable, a message is sent to theCompute Manager 1570 to instantiate the requested resource attime 1610. At time 1612, the Compute Manager selects aCompute Worker 1576 and sends a message to the selected Worker to instantiate the requested resource. Attime 1614, Compute Worker identifies and interacts withNetwork Controller 1560 to get a proper VLAN and IP address as described in steps 451-457 relative toFIG. 4 . Attime 1616, the selectedWorker 1576 interacts with theObject Store 1540 and/or theImage Service 1544 to locate and retrieve an image corresponding to the requested resource. If requested via the API, or used in an embodiment in which configuration information is included on a mountable volume, the selected Worker interacts with theVolume Controller 1550 attime 1618 to locate and retrieve a volume for the to-be-instantiated resource. Attime 1620, the selectedWorker 1576 uses the available virtualization infrastructure as described relative toFIG. 2 to instantiate the resource, mount any volumes, and perform appropriate configuration. Attime 1622, selectedWorker 1576 interacts withNetwork Controller 1560 to configure routing as described relative to step 460 as discussed relative toFIG. 4 . Attime 1624, a message is sent back to theCompute Controller 1520 via theCompute Manager 1570 indicating success and providing necessary operational details relating to the new resource. Attime 1626, a message is sent back to theAPI Server 1510 with the results of the operation as a whole. Attime 1699, the API-specified response to the original command is provided from theAPI Server 1510 back to the originally requesting entity. If at any time a requested operation cannot be performed, then an error is returned to the API Server at time 1690 and the API-specified response to the original command is provided from the API server attime 1692. For example, an error can be returned if a request is not allowable attime 1608, if a VLAN cannot be created or an IP allocated attime 1614, if an image cannot be found or transferred attime 1616, etc. - The foregoing text describes the workings of a cloud infrastructure that utilizes a physical resource pool to run virtualized resources for use within the cloud computing system. The present specification discloses a multi-level cloud computing system wherein a first cloud infrastructure includes a first set of virtualized resources run by an underlying physical resource pool, and a second cloud infrastructure utilizes the first set of virtualized resources to run a second set of virtualized resources. Such a multi-level cloud computing system will now be described in more detail.
-
FIG. 17 is a diagram showing a multi-levelcloud computing system 1700. According to certain illustrative embodiments, the multi-levelcloud computing system 1700 includes aphysical resource pool 1702, afirst cloud infrastructure 1706, a first set ofvirtualized resources 1712, asecond cloud infrastructure 1716, and a second set ofvirtualized resources 1722. - The
physical resource pool 1702 includes a number ofinformation processing devices FIG. 2 . Thus, the information processing devices 1704 are configured to host one or more virtual machines. In one embodiment, the information processing devices 1704 comprise a number of computing systems within a data center. The number of information processing devices 1704 within thephysical resource pool 1702 may vary depending on the expected purposes of the multi-levelcloud computing system 1700. For various purposes, including energy efficiency and security, the information processing devices 1704 not currently being utilized are either shut down or in a sleep/standby mode. The information processing systems 1704 that are not shut down or on standby are used to provide physical resources to thefirst cloud infrastructure 1706. - The
first cloud infrastructure 1706 is a cloud infrastructure similar to that illustrated inFIG. 1 . Thus, thefirst cloud infrastructure 1706 includes at least one cloud controller, which will be referred to for illustrative purposes as thefirst cloud controller 1708. In addition, thefirst cloud infrastructure 1706 includes a number ofcloud services - The
first cloud infrastructure 1706 provides a first set ofvirtualized resources 1712 with access to the underlyingphysical resource pool 1702. For illustrative purposes, resources within the first set ofvirtualized resources 1712 will be referred to as primaryvirtualized resources 1714. Avirtualized resource 1714 may include a virtual information processing device, a logical container, an operating environment. A virtualized resource may also include network services and storage services. A virtual resource may also include an emulator for any type of physical computing device. In one embodiment, a virtualized resource is provisioned using a process similar or identical to the process discussed relative toFIG. 16 . - As is apparent to those of skill in the art, the particular parameters used to launch a virtual resource can vary based upon need. For example, an underlying information processing device with 16 GB of RAM and 16 processor cores can be divided into sixteen, eight, four, two, or one single virtualized computing device, each with a proportionate share of the underlying resources. Similarly, a virtualized computing device can be instantiated that has a higher or lower number of VNICs, access (shared or exclusive) to specialized processing hardware, access to high-performance storage, etc. In this way a virtualized resource can be customized to meet various usage scenarios with better or more predictable performance.
- The first set of
virtualized resources 1712 is used to support asecond cloud infrastructure 1716. Like thefirst cloud infrastructure 1706 that uses thephysical resource pool 1702 as support, thesecond cloud infrastructure 1716 utilizes the primaryvirtualized resources 1714 of thefirst cloud infrastructure 1706 as support. Thus, thesecond cloud infrastructure 1716 has access to thephysical resource pool 1702 through thefirst cloud infrastructure 1706. - In one embodiment, the
first cloud infrastructure 1706 is used to bootstrap thesecond cloud infrastructure 1716. Typically, the bootstrap process for a cloud computing system can be complicated as various physical machines need to be started up in particular sequences. This process may typically require manual coordination of the initial cloud infrastructure. But, by having thefirst cloud infrastructure 1706 bootstrap thesecond cloud infrastructure 1716, the bootstrap process is simpler and does not require manual startup or provisioning of additional physical machines. Virtual resources can be efficiently started up as necessary in order to provide the appropriate support for thesecond cloud infrastructure 1716. - The
second cloud infrastructure 1716 is also similar to the cloud infrastructure illustrated inFIG. 1 . Thus, thesecond cloud infrastructure 1716 includes at least one cloud controller, which will be referred to for illustrative purposes as thesecond cloud controller 1718. In addition, thesecond cloud infrastructure 1716 includes a number ofcloud services - The
second cloud infrastructure 1716 provides a second set ofvirtualized resources 1722 with access to thefirst cloud infrastructure 1706, which in turn provides access to the underlyingphysical resource pool 1702. For illustrative purposes, resources within the second set ofvirtualized resources 1722 will be referred to as secondaryvirtualized resources 1724. As mentioned above, avirtualized resource 1722 may include a virtual information processing device, a logical container, an operating environment. A virtualized resource may also include network services and storage services. - In one embodiment, the
first cloud infrastructure 1706 may include a cluster as described above in the text accompanyingFIG. 3 . Thefirst cloud infrastructure 1706 may be referred to as a parent cluster while thesecond cloud infrastructure 1716 may be referred to as a child cluster. In some embodiments, the secondaryvirtualized resources 1724 may support further cloud infrastructures. - Both the
first cloud infrastructure 1706 and thesecond cloud infrastructure 1716 may exhibit characteristics described and explained above and illustrated inFIGS. 1-16 . Thesecond cloud structure 1716, however, utilizes virtual resources while thefirst cloud infrastructure 1706 utilizes the physical resources of thephysical resource pool 1702. The first cloud infrastructure may also utilize various types of virtual resources. Thesecond cloud infrastructure 1716 ultimately accesses the underlying physical resources through thefirst cloud infrastructure 1706. - This two-level infrastructure is able to virtualize itself and boot a second cloud service (or set of cloud services) on top of the underlying first cloud infrastructure. An exemplary process for performing this function will be described relative to an IaaS-style compute service as described above, but any of the services described herein may be bootstrapped in a similar fashion. In one embodiment, this is performed by creating a first set of virtual machine images that include the software needed to run a compute controller (like compute controller 1520), an auth manager (like auth manager 1530), a network controller (like network controller 1580), a volume controller (like volume controller 1550), a compute manager (like compute manager 1570), and an API server (like API server 1510). These images have configuration information that in most cases only refers to the
second cloud infrastructure 1716; the configuration information is independent of thefirst cloud infrastructure 1706. One notable exception is that the compute manager configuration would include information about the same underlyingphysical resource pool 1702. In some embodiments, a portion of thephysical resource pool 1702 is reserved for the exclusive use of thefirst cloud infrastructure 1706. - Using the process described relative to
FIG. 16 , a network controllervirtualized resource 1714 is instantiated using the network controller image from the image service (or object service) associated with thefirst cloud infrastructure 1706 within the first set ofvirtualized resources 1712. The network controller creates a virtual network for the provision of the new services and sets itself up to provide configuration information as a boot server, DHCP server, DNS server, etc. After the network controller for the second cloud infrastructure is instantiated and the network configuration services are up and running, the remaining essential services (queues, controllers, servers) 1718 are instantiated asvirtualized resources 1714 from the first set ofvirtualized resources 1712 using the images described above. Each reaches out over the network to the configuration server, which provides configuration information corresponding to thesecond cloud infrastructure 1716, particularly the addresses and information necessary for each individual service to see the other independent services that make up thefull cloud service 1720 a. Finally, the API server starts accepting requests from outside to create user-directedvirtualized resources 1724 within the second set ofvirtualized resources 1722. - In another embodiment, the services of either the
first cloud infrastructure 1706 or thesecond cloud infrastructure 1716 include an API server, a compute controller, a compute manager, a network controller, and a volume controller, and an image or object store may be shared between the first andsecond cloud infrastructures second cloud infrastructure 1716 to be completely separate from thefirst cloud infrastructure 1706. For example, a volume controller (one type of cloud controller 1708) could provide volume services to both thefirst cloud infrastructure 1706, thesecond cloud infrastructure 1716, and to user-directedresources 1724. In one embodiment, a second-post-bootstrap cloud service 1720 b is created in thesecond cloud infrastructure 1716 to serve user requests. For example,cloud service 1720 b could be an image or object service provided only for user requests, separate from the image or object service used to store the virtual machine images used to instantiate thesecond cloud infrastructure 1716. - In one embodiment, the
first cloud infrastructure 1706 is designed to support multiple secondary cloud infrastructures. Particularly, a number of secondary cloud infrastructures similar to thesecond cloud infrastructure 1716 may utilize the primaryvirtualized resources 1714. In one example, each different secondary cloud infrastructure supported by the primaryvirtualized resources 1714 may have a different operator. Particularly, one entity may control thefirst cloud infrastructure 1706 and the primaryvirtualized resources 1714 associated therewith. Additionally, different entities may control the various secondary cloud infrastructures that access thephysical resource pool 1702 through thefirst cloud infrastructure 1706. - A multi-level cloud computing system allows for more efficient scaling. Based on the demand for resources made by a user, a cloud computing system provisions a number of resources. In response to an increased demand made by the user, the cloud computing system has to provision additional resources. Typically, in a standard single level cloud, the cloud infrastructure has to spin up more virtualized resources which often requires provisioning more physical information processing devices. As physical information processing devices are relatively slow to start up, the process of scaling up or down can be complicated and time-consuming.
- Through use of a multi-level
cloud computing system 1700 as illustrated inFIG. 17 , an increased demand for resources made by a user on thesecond cloud infrastructure 1716 can be handled by having thefirst cloud infrastructure 1706 start up additional primaryvirtualized resources 1714 for use by thesecond cloud infrastructure 1724. The virtualized resources can be provisioned or torn down much faster than a physical resource can be booted up or shut down. Thus, the multi-level cloud system provides more efficient response to changes in demand. -
FIG. 18 is a flow chart showing aprocess 1800 performed by amonitoring service 1802. Themonitoring service 1802 may be implemented in software, as a subsidiary information processing system, or in a tailored electrical circuit or as software instructions to be used in conjunction with a processor to create a hardware-software combination that implements the specific functionality described herein. - The
monitoring service 1802 monitors 1804 the demand for services made by a user on thesecond cloud infrastructure 1716. Themonitoring service 1802 also monitors 1804 the capacity of the primaryvirtualized resources 1714 that support thatsecond cloud infrastructure 1716. This capacity may be, for example, a computational capacity, a storage capacity, or a network capacity. - The monitoring service then determines 1806 whether there is a discrepancy between the demand for services and capacity for services. If there is no discrepancy (1806, NO), then no action is taken. If, however, there is a discrepancy (1806, YES), then the
monitoring service 1802 then determines 1808 whether the demand is sufficiently greater than or sufficiently less than the capacity. If the demand for services is sufficiently greater than the capacity (1808, YES), then the monitoring system reports this to thefirst cloud controller 1708. Thefirst cloud controller 1708 of thefirst cloud infrastructure 1706 can then create 1810 an additional primaryvirtualized resource 1714. If, however, the demand for services is not sufficiently greater than the capacity (1808, NO), then the discrepancy is that the demand is sufficiently less than the capacity. This will be reported to thefirst cloud controller 1708. Thefirst cloud controller 1708 can then tear down 1812 any primaryvirtual resources 1714 not being used. The demand is sufficiently greater or less than the capacity if a change in the number or virtual resources is required under a particular policy. - In one example, the
monitoring service 1802 may determine that the demand for computational capacity is far less than the computational capacity currently being provided to thesecond cloud infrastructure 1716 by the primaryvirtual resources 1714. In such a case, the resources used to support that virtual computational capacity may be better utilized elsewhere. Thus, upon being notified by themonitoring service 1802 of such a situation, thefirst cloud controller 1708 can tear down a number of virtual computational services not currently being used. - In a further example, the
monitoring service 1802 may determine that the demand for storage capacity on thesecond cloud infrastructure 1716 is greater than the storage capacity currently being provided by the underlying primaryvirtual resources 1714 of thefirst cloud infrastructure 1706. Themonitoring service 1802 then reports this to thefirst cloud controller 1708. Thefirst cloud controller 1708 can then cause one or more additional virtual storage services to be started up. As these additional resources are virtual, they can be started up relatively quickly. - The
monitoring service 1802 may also be used to detect any virtual resources that are no longer being provided. For example, if thesecond cloud infrastructure 1716 is utilizing a particular primaryvirtual resource 1714, and for whatever reason that primaryvirtual resource 1714 gets shut down, themonitoring service 1802 can report that to thefirst cloud controller 1708. Thefirst cloud controller 1708 can then quickly start up a replacement primaryvirtual resource 1714 relatively quickly. -
FIG. 19 is a flow chart showing anillustrative method 1900 for operating a multi-level cloud computing system. According to certain illustrative examples, the method includes, with a first cloud controller for a first cloud infrastructure, providing 1902 a number of primary virtualized resources access to a physical resource pool, the physical resource pool comprising a number of information processing devices, each information processing device comprising a processor, a computer-readable medium, and a network interface. The method further includes, with a second cloud controller for a second cloud infrastructure comprising the primary virtualized resources, providing 1904 a number of secondary virtualized resources access to the physical resource pool through the primary virtualized resources. - Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.
Claims (1)
1. A cloud computing system comprising:
a physical resource pool comprising a number of information processing devices, each information processing device comprising a processor, a computer-readable medium, and a network interface;
a first cloud controller to manage a first cloud infrastructure, the first cloud infrastructure operating a first set of virtualized resources, the first set of virtualized resources having access to the physical resource pool through the first cloud controller; and
a second cloud controller to manage a second cloud infrastructure, the second cloud infrastructure utilizing the first set of virtual resources to operate a second set of virtual resources, the second set of virtual resources being provided access to the physical resource pool through the second cloud controller and the first cloud controller.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/425,484 US20170223117A1 (en) | 2012-08-21 | 2017-02-06 | Multi-level cloud computing system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/590,944 US9563480B2 (en) | 2012-08-21 | 2012-08-21 | Multi-level cloud computing system |
US15/425,484 US20170223117A1 (en) | 2012-08-21 | 2017-02-06 | Multi-level cloud computing system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/590,944 Continuation US9563480B2 (en) | 2012-08-21 | 2012-08-21 | Multi-level cloud computing system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170223117A1 true US20170223117A1 (en) | 2017-08-03 |
Family
ID=49111545
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/590,944 Active 2032-10-15 US9563480B2 (en) | 2012-08-21 | 2012-08-21 | Multi-level cloud computing system |
US15/425,484 Abandoned US20170223117A1 (en) | 2012-08-21 | 2017-02-06 | Multi-level cloud computing system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/590,944 Active 2032-10-15 US9563480B2 (en) | 2012-08-21 | 2012-08-21 | Multi-level cloud computing system |
Country Status (2)
Country | Link |
---|---|
US (2) | US9563480B2 (en) |
WO (1) | WO2014031473A2 (en) |
Cited By (73)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150281111A1 (en) * | 2014-03-28 | 2015-10-01 | Amazon Technologies, Inc. | Implementation of a service that coordinates the placement and execution of containers |
US20150288651A1 (en) * | 2012-11-26 | 2015-10-08 | Huawei Technologies Co., Ltd. | Ip packet processing method and apparatus, and network system |
US20170220250A1 (en) * | 2016-01-30 | 2017-08-03 | HGST Netherlands B.V. | Synchronization method and apparatus for an interconnection network using parallel-headerless tdma routing |
US20180027049A1 (en) * | 2016-07-20 | 2018-01-25 | Adbrain Ltd | Computing system and method of operating the computer system |
US20180276090A1 (en) * | 2016-04-29 | 2018-09-27 | Netapp Inc. | Cross-platform replication |
US20190004844A1 (en) * | 2017-06-26 | 2019-01-03 | Wangsu Science & Technology Co., Ltd. | Cloud platform construction method and cloud platform |
FR3075422A1 (en) * | 2017-12-19 | 2019-06-21 | Atos Information Technology GmbH | PUBLIC CLOUD PLATFORM |
WO2019213427A1 (en) * | 2018-05-04 | 2019-11-07 | Laibson Benjamin William | Emulation of cloud computing service regions |
US10498652B2 (en) | 2015-04-13 | 2019-12-03 | Nicira, Inc. | Method and system of application-aware routing with crowdsourcing |
US10523539B2 (en) | 2017-06-22 | 2019-12-31 | Nicira, Inc. | Method and system of resiliency in cloud-delivered SD-WAN |
US10574528B2 (en) | 2017-02-11 | 2020-02-25 | Nicira, Inc. | Network multi-source inbound quality of service methods and systems |
US10594516B2 (en) | 2017-10-02 | 2020-03-17 | Vmware, Inc. | Virtual network provider |
US10659544B2 (en) * | 2017-11-14 | 2020-05-19 | Google Llc | Opt-out compliance |
US10749711B2 (en) | 2013-07-10 | 2020-08-18 | Nicira, Inc. | Network-link method useful for a last-mile connectivity in an edge-gateway multipath system |
US10778528B2 (en) | 2017-02-11 | 2020-09-15 | Nicira, Inc. | Method and system of connecting to a multipath hub in a cluster |
US10805406B2 (en) * | 2018-06-21 | 2020-10-13 | Microsoft Technology Licensing, Llc | Zone redundant computing services using multiple local services in distributed computing systems |
US10805272B2 (en) | 2015-04-13 | 2020-10-13 | Nicira, Inc. | Method and system of establishing a virtual private network in a cloud service for branch networking |
US10824489B2 (en) * | 2017-09-30 | 2020-11-03 | Oracle International Corporation | Dynamic node rebalancing between container platforms |
US10959098B2 (en) | 2017-10-02 | 2021-03-23 | Vmware, Inc. | Dynamically specifying multiple public cloud edge nodes to connect to an external multi-computer node |
US10992558B1 (en) | 2017-11-06 | 2021-04-27 | Vmware, Inc. | Method and apparatus for distributed data network traffic optimization |
US10992568B2 (en) | 2017-01-31 | 2021-04-27 | Vmware, Inc. | High performance software-defined core network |
US10999100B2 (en) | 2017-10-02 | 2021-05-04 | Vmware, Inc. | Identifying multiple nodes in a virtual network defined over a set of public clouds to connect to an external SAAS provider |
US10999137B2 (en) | 2019-08-27 | 2021-05-04 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US10999165B2 (en) | 2017-10-02 | 2021-05-04 | Vmware, Inc. | Three tiers of SaaS providers for deploying compute and network infrastructure in the public cloud |
US11016793B2 (en) * | 2018-11-26 | 2021-05-25 | Red Hat, Inc. | Filtering based containerized virtual machine networking |
US11032381B2 (en) * | 2019-06-19 | 2021-06-08 | Servicenow, Inc. | Discovery and storage of resource tags |
US11044190B2 (en) | 2019-10-28 | 2021-06-22 | Vmware, Inc. | Managing forwarding elements at edge nodes connected to a virtual network |
US11050588B2 (en) | 2013-07-10 | 2021-06-29 | Nicira, Inc. | Method and system of overlay flow control |
US11089111B2 (en) | 2017-10-02 | 2021-08-10 | Vmware, Inc. | Layer four optimization for a virtual network defined over public cloud |
US11115480B2 (en) | 2017-10-02 | 2021-09-07 | Vmware, Inc. | Layer four optimization for a virtual network defined over public cloud |
US11121962B2 (en) | 2017-01-31 | 2021-09-14 | Vmware, Inc. | High performance software-defined core network |
US20210314419A1 (en) * | 2020-04-01 | 2021-10-07 | Kang Wing Leung | Direct network connections using cloud instance for internet application services |
US20210367837A1 (en) * | 2016-05-24 | 2021-11-25 | Apstra, Inc. | Configuring system resources for different reference architectures |
US20210409409A1 (en) * | 2020-06-29 | 2021-12-30 | Illumina, Inc. | Temporary cloud provider credentials via secure discovery framework |
US11223514B2 (en) | 2017-11-09 | 2022-01-11 | Nicira, Inc. | Method and system of a dynamic high-availability mode based on current wide area network connectivity |
US11228573B1 (en) * | 2018-03-09 | 2022-01-18 | Equinix, Inc. | Application programming interface exchange |
US11245641B2 (en) | 2020-07-02 | 2022-02-08 | Vmware, Inc. | Methods and apparatus for application aware hub clustering techniques for a hyper scale SD-WAN |
US11252079B2 (en) | 2017-01-31 | 2022-02-15 | Vmware, Inc. | High performance software-defined core network |
US20220086156A1 (en) * | 2020-09-16 | 2022-03-17 | Jpmorgan Chase Bank, N.A. | Method and system for providing platform as a service with security access controls |
US11363124B2 (en) | 2020-07-30 | 2022-06-14 | Vmware, Inc. | Zero copy socket splicing |
US11375005B1 (en) | 2021-07-24 | 2022-06-28 | Vmware, Inc. | High availability solutions for a secure access service edge application |
US11374904B2 (en) | 2015-04-13 | 2022-06-28 | Nicira, Inc. | Method and system of a cloud-based multipath routing protocol |
US20220206832A1 (en) * | 2020-12-31 | 2022-06-30 | Nutanix, Inc. | Configuring virtualization system images for a computing cluster |
US11381499B1 (en) | 2021-05-03 | 2022-07-05 | Vmware, Inc. | Routing meshes for facilitating routing through an SD-WAN |
US11388272B2 (en) * | 2018-03-30 | 2022-07-12 | Intel Corporation | Technologies for network packet processing between cloud and telecommunications networks |
US11394640B2 (en) | 2019-12-12 | 2022-07-19 | Vmware, Inc. | Collecting and analyzing data regarding flows associated with DPI parameters |
US11418997B2 (en) | 2020-01-24 | 2022-08-16 | Vmware, Inc. | Using heart beats to monitor operational state of service classes of a QoS aware network link |
US11444865B2 (en) | 2020-11-17 | 2022-09-13 | Vmware, Inc. | Autonomous distributed forwarding plane traceability based anomaly detection in application traffic for hyper-scale SD-WAN |
US20220345521A1 (en) * | 2019-09-19 | 2022-10-27 | Guizhou Baishancloud Technology Co., Ltd. | Network edge computing method, apparatus, device and medium |
US11489720B1 (en) | 2021-06-18 | 2022-11-01 | Vmware, Inc. | Method and apparatus to evaluate resource elements and public clouds for deploying tenant deployable elements based on harvested performance metrics |
US11489783B2 (en) | 2019-12-12 | 2022-11-01 | Vmware, Inc. | Performing deep packet inspection in a software defined wide area network |
US11558258B1 (en) * | 2022-01-31 | 2023-01-17 | Walmart Apollo, Llc | Methods and apparatuses for identifying characteristics of users of electronic resources |
US20230036454A1 (en) * | 2021-07-30 | 2023-02-02 | Nutanix, Inc. | Computing cluster bring-up on public cloud infrastructure using expressed intents |
US11575600B2 (en) | 2020-11-24 | 2023-02-07 | Vmware, Inc. | Tunnel-less SD-WAN |
US11582298B2 (en) * | 2017-05-04 | 2023-02-14 | Amazon Technologies, Inc. | Coordinating inter-region operations in provider network environments |
US11601356B2 (en) | 2020-12-29 | 2023-03-07 | Vmware, Inc. | Emulating packet flows to assess network links for SD-WAN |
US11606286B2 (en) | 2017-01-31 | 2023-03-14 | Vmware, Inc. | High performance software-defined core network |
WO2023076109A1 (en) * | 2021-10-25 | 2023-05-04 | Tencent America LLC | Method and apparatus for using local area network as service for edge cloud |
US11706127B2 (en) | 2017-01-31 | 2023-07-18 | Vmware, Inc. | High performance software-defined core network |
US11706126B2 (en) | 2017-01-31 | 2023-07-18 | Vmware, Inc. | Method and apparatus for distributed data network traffic optimization |
US11729065B2 (en) | 2021-05-06 | 2023-08-15 | Vmware, Inc. | Methods for application defined virtual network service among multiple transport in SD-WAN |
US11770364B2 (en) | 2013-12-17 | 2023-09-26 | Amazon Technologies, Inc. | Private network peering in virtual network environments |
US11777944B2 (en) | 2021-06-29 | 2023-10-03 | International Business Machines Corporation | Scalable authentication management |
US11792127B2 (en) | 2021-01-18 | 2023-10-17 | Vmware, Inc. | Network-aware load balancing |
US11861342B2 (en) | 2022-01-28 | 2024-01-02 | Microstrategy Incorporated | Enhanced cloud-computing environment deployment |
US11909815B2 (en) | 2022-06-06 | 2024-02-20 | VMware LLC | Routing based on geolocation costs |
US11934886B2 (en) | 2020-10-31 | 2024-03-19 | Nutanix, Inc. | Intra-footprint computing cluster bring-up |
US11943146B2 (en) | 2021-10-01 | 2024-03-26 | VMware LLC | Traffic prioritization in SD-WAN |
US11954473B2 (en) | 2021-09-20 | 2024-04-09 | Microstrategy Incorporated | Deployment architecture for multi-tenant cloud computing systems |
US11979325B2 (en) | 2021-01-28 | 2024-05-07 | VMware LLC | Dynamic SD-WAN hub cluster scaling with machine learning |
US12009987B2 (en) | 2021-05-03 | 2024-06-11 | VMware LLC | Methods to support dynamic transit paths through hub clustering across branches in SD-WAN |
US12015536B2 (en) | 2021-06-18 | 2024-06-18 | VMware LLC | Method and apparatus for deploying tenant deployable elements across public clouds based on harvested performance metrics of types of resource elements in the public clouds |
US12034587B1 (en) | 2023-03-27 | 2024-07-09 | VMware LLC | Identifying and remediating anomalies in a self-healing network |
Families Citing this family (317)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8675847B2 (en) | 2007-01-03 | 2014-03-18 | Cisco Technology, Inc. | Scalable conference bridge |
US8756329B2 (en) | 2010-09-15 | 2014-06-17 | Oracle International Corporation | System and method for parallel multiplexing between servers in a cluster |
US9086909B2 (en) * | 2011-05-17 | 2015-07-21 | Oracle International Corporation | System and method for supporting work sharing muxing in a cluster |
US9135269B2 (en) * | 2011-12-07 | 2015-09-15 | Egnyte, Inc. | System and method of implementing an object storage infrastructure for cloud-based services |
TWI461929B (en) * | 2011-12-09 | 2014-11-21 | Promise Tecnnology Inc | Cloud data storage system |
EP2850584A4 (en) | 2012-05-14 | 2016-04-06 | Sgrouples Inc | Social platform with enhanced privacy and integrated customization features |
US9881017B2 (en) | 2012-08-03 | 2018-01-30 | Egnyte, Inc. | System and method for event-based synchronization of remote and local file systems |
US10127081B2 (en) * | 2012-08-30 | 2018-11-13 | International Business Machines Corporation | Efficient resource management in a virtualized computing environment |
US20140068042A1 (en) * | 2012-09-06 | 2014-03-06 | International Business Machines Corporation | Method and Apparatus for Accelerated Virtual Image Provisioning in Distributed Cloud Environments |
US9467355B2 (en) | 2012-09-07 | 2016-10-11 | Oracle International Corporation | Service association model |
US9621435B2 (en) | 2012-09-07 | 2017-04-11 | Oracle International Corporation | Declarative and extensible model for provisioning of cloud based services |
US10148530B2 (en) | 2012-09-07 | 2018-12-04 | Oracle International Corporation | Rule based subscription cloning |
US9276942B2 (en) | 2012-09-07 | 2016-03-01 | Oracle International Corporation | Multi-tenancy identity management system |
US9542400B2 (en) | 2012-09-07 | 2017-01-10 | Oracle International Corporation | Service archive support |
US9667470B2 (en) | 2012-09-07 | 2017-05-30 | Oracle International Corporation | Failure handling in the execution flow of provisioning operations in a cloud environment |
US9792338B2 (en) | 2012-09-07 | 2017-10-17 | Oracle International Corporation | Role assignments in a cloud infrastructure |
US9274843B2 (en) * | 2012-09-14 | 2016-03-01 | Ca, Inc. | Multi-redundant switchable process pooling for cloud it services delivery |
US9311161B2 (en) | 2012-09-14 | 2016-04-12 | Ca, Inc. | Automatically configured management service payloads for cloud IT services delivery |
CN103685399B (en) * | 2012-09-17 | 2018-03-23 | 腾讯科技(深圳)有限公司 | A kind of methods, devices and systems for logging in class Unix virtual containers |
US9450784B2 (en) * | 2012-09-27 | 2016-09-20 | Blackberry Limited | Communicating data among personal clouds |
US9251114B1 (en) * | 2012-10-12 | 2016-02-02 | Egnyte, Inc. | Systems and methods for facilitating access to private files using a cloud storage system |
US9122841B2 (en) * | 2012-10-16 | 2015-09-01 | Sap Se | Providing remote application logs for cloud applications |
US9485234B1 (en) | 2012-11-14 | 2016-11-01 | Amazon Technologies, Inc. | Virtualized endpoints in a multi-tenant environment |
US9148414B1 (en) * | 2012-11-14 | 2015-09-29 | Amazon Technologies, Inc. | Credential management in a multi-tenant environment |
US9569328B2 (en) * | 2012-11-29 | 2017-02-14 | Sap Se | Managing application log levels in cloud environment |
US9231819B2 (en) * | 2012-11-30 | 2016-01-05 | Red Hat Israel, Ltd. | Abstracting restful web service HTTP between a client and a server |
US9275408B1 (en) * | 2013-01-25 | 2016-03-01 | Amazon Technologies, Inc. | Transferring ownership of computing resources |
US9912521B2 (en) * | 2013-03-13 | 2018-03-06 | Dell Products L.P. | Systems and methods for managing connections in an orchestrated network |
US9590872B1 (en) | 2013-03-14 | 2017-03-07 | Ca, Inc. | Automated cloud IT services delivery solution model |
US10454999B2 (en) * | 2013-03-14 | 2019-10-22 | Red Hat, Inc. | Coordination of inter-operable infrastructure as a service (IAAS) and platform as a service (PAAS) |
CN108156180B (en) * | 2013-06-08 | 2021-04-09 | 科大国盾量子技术股份有限公司 | Communication key distribution method based on Android intelligent mobile terminal |
US20150006730A1 (en) * | 2013-06-27 | 2015-01-01 | Sap Ag | Enabling multi-tenant virtual servers in a cloud system |
US9218193B2 (en) * | 2013-07-12 | 2015-12-22 | International Business Machines Corporation | Distributed virtual machine image management for cloud computing |
US9710292B2 (en) * | 2013-08-02 | 2017-07-18 | International Business Machines Corporation | Allowing management of a virtual machine by multiple cloud providers |
US9348634B2 (en) | 2013-08-12 | 2016-05-24 | Amazon Technologies, Inc. | Fast-booting application image using variation points in application source code |
US10346148B2 (en) | 2013-08-12 | 2019-07-09 | Amazon Technologies, Inc. | Per request computer system instances |
US9280372B2 (en) * | 2013-08-12 | 2016-03-08 | Amazon Technologies, Inc. | Request processing techniques |
WO2015021629A1 (en) * | 2013-08-15 | 2015-02-19 | 华为技术有限公司 | Resource allocation method |
US9612815B1 (en) | 2013-08-22 | 2017-04-04 | Ca, Inc. | Method and tool for automating deployment of reference implementation architectures for pre-integrated multi-product solutions |
US20150081400A1 (en) * | 2013-09-19 | 2015-03-19 | Infosys Limited | Watching ARM |
US9754303B1 (en) | 2013-10-03 | 2017-09-05 | Ca, Inc. | Service offering templates for user interface customization in CITS delivery containers |
US9684685B2 (en) * | 2013-10-24 | 2017-06-20 | Sap Se | Using message-passing with procedural code in a database kernel |
US9600551B2 (en) | 2013-10-24 | 2017-03-21 | Sap Se | Coexistence of message-passing-like algorithms and procedural coding |
US9411612B2 (en) * | 2013-11-12 | 2016-08-09 | Software Ag | Techniques for creating and/or maintaining scalable heterogeneous read-only federations of registries |
US9229795B2 (en) * | 2013-12-09 | 2016-01-05 | Hewlett Packard Enterprise Development Lp | Execution of end-to-end processes across applications |
US10084839B2 (en) | 2013-12-27 | 2018-09-25 | Sap Se | Upload operation using multiple connections |
US10742520B2 (en) * | 2013-12-31 | 2020-08-11 | Citrix Systems, Inc. | Providing mobile device management functionalities |
US9813516B2 (en) | 2014-02-18 | 2017-11-07 | Salesforce.Com, Inc. | Transparent sharding of traffic across messaging brokers |
US9842152B2 (en) | 2014-02-19 | 2017-12-12 | Snowflake Computing, Inc. | Transparent discovery of semi-structured data schema |
US10545917B2 (en) | 2014-02-19 | 2020-01-28 | Snowflake Inc. | Multi-range and runtime pruning |
US9158909B2 (en) | 2014-03-04 | 2015-10-13 | Amazon Technologies, Inc. | Authentication of virtual machine images using digital certificates |
US9916188B2 (en) | 2014-03-14 | 2018-03-13 | Cask Data, Inc. | Provisioner for cluster management system |
US9419855B2 (en) | 2014-03-14 | 2016-08-16 | Nicira, Inc. | Static routes for logical routers |
US9647883B2 (en) | 2014-03-21 | 2017-05-09 | Nicria, Inc. | Multiple levels of logical routers |
US9825854B2 (en) * | 2014-03-27 | 2017-11-21 | Nicira, Inc. | Host architecture for efficient cloud service access |
US9794186B2 (en) | 2014-03-27 | 2017-10-17 | Nicira, Inc. | Distributed network address translation for efficient cloud service access |
US9519510B2 (en) * | 2014-03-31 | 2016-12-13 | Amazon Technologies, Inc. | Atomic writes for multiple-extent operations |
US10372685B2 (en) | 2014-03-31 | 2019-08-06 | Amazon Technologies, Inc. | Scalable file storage service |
US10264071B2 (en) | 2014-03-31 | 2019-04-16 | Amazon Technologies, Inc. | Session management in distributed storage systems |
US10361924B2 (en) | 2014-04-04 | 2019-07-23 | International Business Machines Corporation | Forecasting computer resources demand |
US10043194B2 (en) | 2014-04-04 | 2018-08-07 | International Business Machines Corporation | Network demand forecasting |
US9385934B2 (en) | 2014-04-08 | 2016-07-05 | International Business Machines Corporation | Dynamic network monitoring |
US10439891B2 (en) | 2014-04-08 | 2019-10-08 | International Business Machines Corporation | Hyperparameter and network topology selection in network demand forecasting |
US10084669B2 (en) | 2014-04-09 | 2018-09-25 | Centurylink Intellectual Property Llc | System and method for cloud computing adaptive cloud services |
US9838253B2 (en) * | 2014-04-10 | 2017-12-05 | Fujitsu Limited | Object-oriented network virtualization |
US10713574B2 (en) | 2014-04-10 | 2020-07-14 | International Business Machines Corporation | Cognitive distributed network |
US9430268B2 (en) | 2014-05-02 | 2016-08-30 | Cavium, Inc. | Systems and methods for supporting migration of virtual machines accessing remote storage devices over network via NVMe controllers |
US9294567B2 (en) * | 2014-05-02 | 2016-03-22 | Cavium, Inc. | Systems and methods for enabling access to extensible storage devices over a network as local storage via NVME controller |
US9529773B2 (en) * | 2014-05-02 | 2016-12-27 | Cavium, Inc. | Systems and methods for enabling access to extensible remote storage over a network as local storage via a logical storage controller |
US10193769B2 (en) | 2014-05-16 | 2019-01-29 | Centurylink Intellectual Property Llc | Network services API |
US10129344B2 (en) | 2014-06-19 | 2018-11-13 | Microsoft Technology Licensing, Llc | Integrated user interface for consuming services across different distributed networks |
US9560037B2 (en) * | 2014-06-19 | 2017-01-31 | Microsoft Technology Licensing, Llc | Integrated APIs and UIs for consuming services across different distributed networks |
US9262200B2 (en) | 2014-06-25 | 2016-02-16 | Independenceit, Inc. | Methods and systems for provisioning a virtual resource in a mixed-use server |
US9712542B1 (en) * | 2014-06-27 | 2017-07-18 | Amazon Technologies, Inc. | Permissions decisions in a service provider environment |
WO2016014592A1 (en) | 2014-07-21 | 2016-01-28 | Egnyte, Inc. | System and method for policy based synchronization of remote and local file systems |
US10547562B2 (en) | 2014-07-31 | 2020-01-28 | Hewlett Packard Enterprise Development Lp | Cloud resource pool |
US9900391B2 (en) * | 2014-08-05 | 2018-02-20 | Microsoft Technology Licensing, Llc | Automated orchestration of infrastructure service blocks in hosted services |
US10291597B2 (en) | 2014-08-14 | 2019-05-14 | Cisco Technology, Inc. | Sharing resources across multiple devices in online meetings |
US9800673B2 (en) | 2014-08-20 | 2017-10-24 | At&T Intellectual Property I, L.P. | Service compiler component and service controller for open systems interconnection layer 4 through layer 7 services in a cloud computing system |
US9473567B2 (en) | 2014-08-20 | 2016-10-18 | At&T Intellectual Property I, L.P. | Virtual zones for open systems interconnection layer 4 through layer 7 services in a cloud computing system |
US9749242B2 (en) | 2014-08-20 | 2017-08-29 | At&T Intellectual Property I, L.P. | Network platform as a service layer for open systems interconnection communication model layer 4 through layer 7 services |
US10291689B2 (en) | 2014-08-20 | 2019-05-14 | At&T Intellectual Property I, L.P. | Service centric virtual network function architecture for development and deployment of open systems interconnection communication model layer 4 through layer 7 services in a cloud computing system |
US9742690B2 (en) | 2014-08-20 | 2017-08-22 | At&T Intellectual Property I, L.P. | Load adaptation architecture framework for orchestrating and managing services in a cloud computing system |
WO2016036293A1 (en) | 2014-09-04 | 2016-03-10 | Telefonaktiebolaget L M Ericsson (Publ) | Method and apparatuses for enabling routing of data packets between a wireless device and a service provider based in the local service cloud |
US20160072704A1 (en) * | 2014-09-09 | 2016-03-10 | Microsoft Corporation | Resource control for virtual datacenters |
US11455181B1 (en) * | 2014-09-19 | 2022-09-27 | Amazon Technologies, Inc. | Cross-network connector appliances |
US9146764B1 (en) | 2014-09-30 | 2015-09-29 | Amazon Technologies, Inc. | Processing event messages for user requests to execute program code |
US10048974B1 (en) | 2014-09-30 | 2018-08-14 | Amazon Technologies, Inc. | Message-based computation request scheduling |
US9678773B1 (en) | 2014-09-30 | 2017-06-13 | Amazon Technologies, Inc. | Low latency computational capacity provisioning |
US9323556B2 (en) | 2014-09-30 | 2016-04-26 | Amazon Technologies, Inc. | Programmatic event detection and message generation for requests to execute program code |
US9715402B2 (en) | 2014-09-30 | 2017-07-25 | Amazon Technologies, Inc. | Dynamic code deployment and versioning |
US9830193B1 (en) | 2014-09-30 | 2017-11-28 | Amazon Technologies, Inc. | Automatic management of low latency computational capacity |
US9600312B2 (en) | 2014-09-30 | 2017-03-21 | Amazon Technologies, Inc. | Threading as a service |
US9537788B2 (en) | 2014-12-05 | 2017-01-03 | Amazon Technologies, Inc. | Automatic determination of resource sizing |
US20160182667A1 (en) * | 2014-12-19 | 2016-06-23 | Tobias Gregor Oberstein | Management Application for an Application Router |
US10542126B2 (en) | 2014-12-22 | 2020-01-21 | Cisco Technology, Inc. | Offline virtual participation in an online conference meeting |
JP6979264B2 (en) * | 2014-12-30 | 2021-12-08 | エヌエイチエヌ コーポレーション | Cloud service provision method and system |
US9519553B2 (en) * | 2014-12-31 | 2016-12-13 | Servicenow, Inc. | Failure resistant distributed computing system |
US10079779B2 (en) | 2015-01-30 | 2018-09-18 | Nicira, Inc. | Implementing logical router uplinks |
US9900313B2 (en) * | 2015-02-04 | 2018-02-20 | Red Hat, Inc. | Secure shell (SSH) proxy for a platform-as-a-service system |
US9588790B1 (en) | 2015-02-04 | 2017-03-07 | Amazon Technologies, Inc. | Stateful virtual compute system |
US9727725B2 (en) | 2015-02-04 | 2017-08-08 | Amazon Technologies, Inc. | Security protocols for low latency execution of program code |
US9733967B2 (en) | 2015-02-04 | 2017-08-15 | Amazon Technologies, Inc. | Security protocols for low latency execution of program code |
US9742726B2 (en) * | 2015-02-26 | 2017-08-22 | Red Hat Israel, Ltd. | Distributed dynamic host configuration protocol |
EP3561672B1 (en) * | 2015-04-07 | 2022-06-01 | Huawei Technologies Co., Ltd. | Method and apparatus for a mobile device based cluster computing infrastructure |
US9930103B2 (en) | 2015-04-08 | 2018-03-27 | Amazon Technologies, Inc. | Endpoint management system providing an application programming interface proxy service |
US9785476B2 (en) | 2015-04-08 | 2017-10-10 | Amazon Technologies, Inc. | Endpoint management system and virtual compute system |
US10437789B2 (en) | 2015-04-10 | 2019-10-08 | Egnyte, Inc. | System and method for delete fencing during synchronization of remote and local file systems |
US9948786B2 (en) | 2015-04-17 | 2018-04-17 | Cisco Technology, Inc. | Handling conferences using highly-distributed agents |
US9948552B2 (en) * | 2015-04-17 | 2018-04-17 | Equinix, Inc. | Cloud-based services exchange |
US10778597B1 (en) * | 2015-05-21 | 2020-09-15 | EMC IP Holding Company LLC | Orchestration management system and method for managing a resource pool across multiple computing clouds |
US10701037B2 (en) | 2015-05-27 | 2020-06-30 | Ping Identity Corporation | Scalable proxy clusters |
US9826030B1 (en) * | 2015-06-04 | 2017-11-21 | Amazon Technologies, Inc. | Placement of volume partition replica pairs |
US9826041B1 (en) * | 2015-06-04 | 2017-11-21 | Amazon Technologies, Inc. | Relative placement of volume partitions |
US10148493B1 (en) * | 2015-06-08 | 2018-12-04 | Infoblox Inc. | API gateway for network policy and configuration management with public cloud |
US11144510B2 (en) | 2015-06-11 | 2021-10-12 | Egnyte, Inc. | System and method for synchronizing file systems with large namespaces |
US9935896B2 (en) * | 2015-06-18 | 2018-04-03 | Cisco Technology, Inc. | System and method for scaling multiclouds in a hybrid cloud architecture |
US10965609B2 (en) * | 2015-07-06 | 2021-03-30 | Telefonaktiebolaget Lm Ericsson (Publ) | Mapping of service requirements for a virtualized network on a packet flow |
US10129142B2 (en) | 2015-08-11 | 2018-11-13 | Nicira, Inc. | Route configuration for logical router |
US10142174B2 (en) | 2015-08-25 | 2018-11-27 | Oracle International Corporation | Service deployment infrastructure request provisioning |
US10057157B2 (en) | 2015-08-31 | 2018-08-21 | Nicira, Inc. | Automatically advertising NAT routes between logical routers |
US10326710B1 (en) | 2015-09-02 | 2019-06-18 | Amazon Technologies, Inc. | Propagating access rules on virtual networks in provider network environments |
US9928108B1 (en) | 2015-09-29 | 2018-03-27 | Amazon Technologies, Inc. | Metaevent handling for on-demand code execution environments |
US10042660B2 (en) | 2015-09-30 | 2018-08-07 | Amazon Technologies, Inc. | Management of periodic requests for compute capacity |
US10664590B2 (en) * | 2015-10-01 | 2020-05-26 | Twistlock, Ltd. | Filesystem action profiling of containers and security enforcement |
US10706145B2 (en) | 2015-10-01 | 2020-07-07 | Twistlock, Ltd. | Runtime detection of vulnerabilities in software containers |
US10599833B2 (en) | 2015-10-01 | 2020-03-24 | Twistlock, Ltd. | Networking-based profiling of containers and security enforcement |
US10223534B2 (en) | 2015-10-15 | 2019-03-05 | Twistlock, Ltd. | Static detection of vulnerabilities in base images of software containers |
US10943014B2 (en) | 2015-10-01 | 2021-03-09 | Twistlock, Ltd | Profiling of spawned processes in container images and enforcing security policies respective thereof |
US10567411B2 (en) | 2015-10-01 | 2020-02-18 | Twistlock, Ltd. | Dynamically adapted traffic inspection and filtering in containerized environments |
US10586042B2 (en) | 2015-10-01 | 2020-03-10 | Twistlock, Ltd. | Profiling of container images and enforcing security policies respective thereof |
US10922418B2 (en) | 2015-10-01 | 2021-02-16 | Twistlock, Ltd. | Runtime detection and mitigation of vulnerabilities in application software containers |
US10693899B2 (en) * | 2015-10-01 | 2020-06-23 | Twistlock, Ltd. | Traffic enforcement in containerized environments |
US10778446B2 (en) | 2015-10-15 | 2020-09-15 | Twistlock, Ltd. | Detection of vulnerable root certificates in software containers |
US9794292B2 (en) * | 2015-10-26 | 2017-10-17 | Amazon Technologies, Inc. | Providing fine-grained access remote command execution for virtual machine instances in a distributed computing environment |
US10095535B2 (en) | 2015-10-31 | 2018-10-09 | Nicira, Inc. | Static route types for logical routers |
CN106682004A (en) * | 2015-11-06 | 2017-05-17 | 网宿科技股份有限公司 | Redis Key management method and system |
US10291762B2 (en) | 2015-12-04 | 2019-05-14 | Cisco Technology, Inc. | Docking station for mobile computing devices |
US10298720B1 (en) * | 2015-12-07 | 2019-05-21 | Amazon Technologies, Inc. | Client-defined rules in provider network environments |
US10275467B2 (en) | 2015-12-15 | 2019-04-30 | Microsoft Technology Licensing, Llc | Multi-level high availability model for an object storage service |
US10013267B1 (en) | 2015-12-16 | 2018-07-03 | Amazon Technologies, Inc. | Pre-triggers for code execution environments |
US9811363B1 (en) | 2015-12-16 | 2017-11-07 | Amazon Technologies, Inc. | Predictive management of on-demand code execution |
US9830449B1 (en) | 2015-12-16 | 2017-11-28 | Amazon Technologies, Inc. | Execution locations for request-driven code |
US9811434B1 (en) | 2015-12-16 | 2017-11-07 | Amazon Technologies, Inc. | Predictive management of on-demand code execution |
US10754701B1 (en) | 2015-12-16 | 2020-08-25 | Amazon Technologies, Inc. | Executing user-defined code in response to determining that resources expected to be utilized comply with resource restrictions |
US9830175B1 (en) | 2015-12-16 | 2017-11-28 | Amazon Technologies, Inc. | Predictive management of on-demand code execution |
US9910713B2 (en) | 2015-12-21 | 2018-03-06 | Amazon Technologies, Inc. | Code execution request routing |
US10002026B1 (en) | 2015-12-21 | 2018-06-19 | Amazon Technologies, Inc. | Acquisition and maintenance of dedicated, reserved, and variable compute capacity |
US10067801B1 (en) | 2015-12-21 | 2018-09-04 | Amazon Technologies, Inc. | Acquisition and maintenance of compute capacity |
US20170177613A1 (en) | 2015-12-22 | 2017-06-22 | Egnyte, Inc. | Event-Based User State Synchronization in a Cloud Storage System |
US10079693B2 (en) * | 2015-12-28 | 2018-09-18 | Netapp, Inc. | Storage cluster management proxy |
US10171457B2 (en) * | 2015-12-29 | 2019-01-01 | International Business Machines Corporation | Service provider initiated additional authentication in a federated system |
US10104170B2 (en) * | 2016-01-05 | 2018-10-16 | Oracle International Corporation | System and method of assigning resource consumers to resources using constraint programming |
US10074066B2 (en) | 2016-01-16 | 2018-09-11 | International Business Machines Corporation | Two phase predictive approach for supply network optimization |
US10891145B2 (en) | 2016-03-30 | 2021-01-12 | Amazon Technologies, Inc. | Processing pre-existing data sets at an on demand code execution environment |
US10038694B1 (en) * | 2016-03-30 | 2018-07-31 | EMC IP Holding Company LLC | System and method for security mode-based authorization for data management operations in a multi-tenant protection storage system |
US10162672B2 (en) | 2016-03-30 | 2018-12-25 | Amazon Technologies, Inc. | Generating data streams from pre-existing data sets |
US10212169B2 (en) | 2016-03-30 | 2019-02-19 | Oracle International Corporation | Enforcing data security in a cleanroom data processing environment |
US11132213B1 (en) | 2016-03-30 | 2021-09-28 | Amazon Technologies, Inc. | Dependency-based process of pre-existing data sets at an on demand code execution environment |
US10200461B2 (en) * | 2016-04-07 | 2019-02-05 | Virtustream Ip Holding Company Llc | Virtualized capacity management |
US10498765B2 (en) * | 2016-06-01 | 2019-12-03 | At&T Intellectual Property I, L.P. | Virtual infrastructure perimeter regulator |
US10282229B2 (en) | 2016-06-28 | 2019-05-07 | Amazon Technologies, Inc. | Asynchronous task management in an on-demand network code execution environment |
US9952896B2 (en) | 2016-06-28 | 2018-04-24 | Amazon Technologies, Inc. | Asynchronous task management in an on-demand network code execution environment |
US10574609B2 (en) | 2016-06-29 | 2020-02-25 | Cisco Technology, Inc. | Chat room access control |
US10102040B2 (en) | 2016-06-29 | 2018-10-16 | Amazon Technologies, Inc | Adjusting variable limit on concurrent code executions |
US10153973B2 (en) * | 2016-06-29 | 2018-12-11 | Nicira, Inc. | Installation of routing tables for logical router in route server mode |
US10203990B2 (en) | 2016-06-30 | 2019-02-12 | Amazon Technologies, Inc. | On-demand network code execution with cross-account aliases |
US10277708B2 (en) | 2016-06-30 | 2019-04-30 | Amazon Technologies, Inc. | On-demand network code execution with cross-account aliases |
US10437780B2 (en) | 2016-07-14 | 2019-10-08 | Snowflake Inc. | Data pruning based on metadata |
CN107645532B (en) * | 2016-07-22 | 2020-07-24 | 腾讯科技(深圳)有限公司 | User management method and device of hybrid cloud |
US10425465B1 (en) * | 2016-07-29 | 2019-09-24 | Google Llc | Hybrid cloud API management |
US10409622B2 (en) | 2016-08-10 | 2019-09-10 | Bank Of America Corporation | Orchestration pipeline for providing and operating segmented computing resources |
US9977670B2 (en) | 2016-08-10 | 2018-05-22 | Bank Of America Corporation | Application programming interface for providing access to computing platform definitions |
US10469315B2 (en) | 2016-08-10 | 2019-11-05 | Bank Of America Corporation | Using computing platform definitions to provide segmented computing platforms in a computing system |
US10454758B2 (en) | 2016-08-31 | 2019-10-22 | Nicira, Inc. | Edge node cluster network redundancy and fast convergence using an underlay anycast VTEP IP |
US10884787B1 (en) | 2016-09-23 | 2021-01-05 | Amazon Technologies, Inc. | Execution guarantees in an on-demand network code execution system |
US10061613B1 (en) | 2016-09-23 | 2018-08-28 | Amazon Technologies, Inc. | Idempotent task execution in on-demand network code execution systems |
US20180091449A1 (en) * | 2016-09-26 | 2018-03-29 | Telefonaktiebolaget Lm Ericsson (Publ) | Event-driven policy-based distributed container management system |
US11119813B1 (en) | 2016-09-30 | 2021-09-14 | Amazon Technologies, Inc. | Mapreduce implementation using an on-demand network code execution system |
US10341236B2 (en) | 2016-09-30 | 2019-07-02 | Nicira, Inc. | Anycast edge service gateways |
US10616347B1 (en) * | 2016-10-20 | 2020-04-07 | R&D Industries, Inc. | Devices, systems and methods for internet and failover connectivity and monitoring |
US10681012B2 (en) | 2016-10-26 | 2020-06-09 | Ping Identity Corporation | Methods and systems for deep learning based API traffic security |
US10592867B2 (en) | 2016-11-11 | 2020-03-17 | Cisco Technology, Inc. | In-meeting graphical user interface display using calendar information and system |
US10630682B1 (en) | 2016-11-23 | 2020-04-21 | Amazon Technologies, Inc. | Lightweight authentication protocol using device tokens |
US10129223B1 (en) * | 2016-11-23 | 2018-11-13 | Amazon Technologies, Inc. | Lightweight encrypted communication protocol |
US10776385B2 (en) | 2016-12-02 | 2020-09-15 | Vmware, Inc. | Methods and apparatus for transparent database switching using master-replica high availability setup in relational databases |
US10362110B1 (en) * | 2016-12-08 | 2019-07-23 | Amazon Technologies, Inc. | Deployment of client data compute kernels in cloud |
US10873501B2 (en) * | 2016-12-09 | 2020-12-22 | Vmware, Inc. | Methods, systems and apparatus to propagate node configuration changes to services in a distributed environment |
US10516707B2 (en) | 2016-12-15 | 2019-12-24 | Cisco Technology, Inc. | Initiating a conferencing meeting using a conference room device |
US11271904B2 (en) * | 2016-12-22 | 2022-03-08 | Nicira, Inc. | Identification and adjustment of ineffective firewall rules |
US20180191718A1 (en) * | 2016-12-29 | 2018-07-05 | Ingram Micro Inc. | Technologies for securely extending cloud service apis in a cloud service marketplace |
US10552191B2 (en) * | 2017-01-26 | 2020-02-04 | Cisco Technology, Inc. | Distributed hybrid cloud orchestration model |
US10311248B1 (en) * | 2017-01-27 | 2019-06-04 | Intuit Inc. | Managing delegated access permissions |
US10454977B2 (en) | 2017-02-14 | 2019-10-22 | At&T Intellectual Property I, L.P. | Systems and methods for allocating and managing resources in an internet of things environment using location based focus of attention |
US10515117B2 (en) | 2017-02-14 | 2019-12-24 | Cisco Technology, Inc. | Generating and reviewing motion metadata |
US9942519B1 (en) | 2017-02-21 | 2018-04-10 | Cisco Technology, Inc. | Technologies for following participants in a video conference |
US11089105B1 (en) * | 2017-12-14 | 2021-08-10 | Pure Storage, Inc. | Synchronously replicating datasets in cloud-based storage systems |
US10440073B2 (en) | 2017-04-11 | 2019-10-08 | Cisco Technology, Inc. | User interface for proximity based teleconference transfer |
US10375125B2 (en) | 2017-04-27 | 2019-08-06 | Cisco Technology, Inc. | Automatically joining devices to a video conference |
US10630399B2 (en) * | 2017-05-05 | 2020-04-21 | Dell Products, L.P. | Testing distributed applications that have an established exchange in an advanced message queuing protocol (AMQP) message broker |
US11204816B2 (en) | 2017-05-09 | 2021-12-21 | Microsoft Technology Licensing, Llc | Deployment of modular applications from the cloud to local devices |
US11429410B2 (en) * | 2017-05-09 | 2022-08-30 | Vmware, Inc. | Tag based firewall implementation in software defined networks |
US11108788B1 (en) * | 2017-05-15 | 2021-08-31 | RiskIQ, Inc. | Techniques for managing projects and monitoring network-based assets |
US10467046B2 (en) | 2017-05-30 | 2019-11-05 | Red Hat, Inc. | Fast and greedy scheduling machine based on a distance matrix |
US10404481B2 (en) | 2017-06-06 | 2019-09-03 | Cisco Technology, Inc. | Unauthorized participant detection in multiparty conferencing by comparing a reference hash value received from a key management server with a generated roster hash value |
US10375474B2 (en) | 2017-06-12 | 2019-08-06 | Cisco Technology, Inc. | Hybrid horn microphone |
US10477148B2 (en) | 2017-06-23 | 2019-11-12 | Cisco Technology, Inc. | Speaker anticipation |
US10516709B2 (en) | 2017-06-29 | 2019-12-24 | Cisco Technology, Inc. | Files automatically shared at conference initiation |
US10706391B2 (en) | 2017-07-13 | 2020-07-07 | Cisco Technology, Inc. | Protecting scheduled meeting in physical room |
US10091348B1 (en) | 2017-07-25 | 2018-10-02 | Cisco Technology, Inc. | Predictive model for voice/video over IP calls |
US11106682B2 (en) | 2017-10-12 | 2021-08-31 | Target Brands, Inc. | Decision platform for directed information delivery |
EP3471007B1 (en) | 2017-10-13 | 2022-02-23 | Ping Identity Corporation | Methods and apparatus for analyzing sequences of application programming interface traffic to identify potential malicious actions |
US10771621B2 (en) | 2017-10-31 | 2020-09-08 | Cisco Technology, Inc. | Acoustic echo cancellation based sub band domain active speaker detection for audio and video conferencing applications |
US10587507B2 (en) | 2017-11-09 | 2020-03-10 | International Business Machines Corporation | Routing between software defined networks and physical networks |
KR101858715B1 (en) * | 2017-11-10 | 2018-05-16 | 베스핀글로벌 주식회사 | Management System for Service Resource and Method thereof |
US10467211B2 (en) | 2017-11-15 | 2019-11-05 | International Business Machines Corporation | Representing and analyzing cloud computing data as pseudo systems |
JP2019096244A (en) * | 2017-11-28 | 2019-06-20 | 京セラドキュメントソリューションズ株式会社 | Information processing system and information processing method |
US10564946B1 (en) | 2017-12-13 | 2020-02-18 | Amazon Technologies, Inc. | Dependency handling in an on-demand network code execution system |
US10303492B1 (en) | 2017-12-13 | 2019-05-28 | Amazon Technologies, Inc. | Managing custom runtimes in an on-demand code execution system |
US10904068B2 (en) * | 2018-01-12 | 2021-01-26 | Datera, Inc. | System and method to provide seamless data placement, data movement, and data management into cloud |
US10572375B1 (en) | 2018-02-05 | 2020-02-25 | Amazon Technologies, Inc. | Detecting parameter validity in code including cross-service calls |
US10353678B1 (en) | 2018-02-05 | 2019-07-16 | Amazon Technologies, Inc. | Detecting code characteristic alterations due to cross-service calls |
US10733085B1 (en) | 2018-02-05 | 2020-08-04 | Amazon Technologies, Inc. | Detecting impedance mismatches due to cross-service calls |
US10831898B1 (en) | 2018-02-05 | 2020-11-10 | Amazon Technologies, Inc. | Detecting privilege escalations in code including cross-service calls |
US10725752B1 (en) | 2018-02-13 | 2020-07-28 | Amazon Technologies, Inc. | Dependency handling in an on-demand network code execution system |
US10776091B1 (en) | 2018-02-26 | 2020-09-15 | Amazon Technologies, Inc. | Logging endpoint in an on-demand code execution system |
US10848552B2 (en) * | 2018-03-29 | 2020-11-24 | Hewlett Packard Enterprise Development Lp | Determining whether to perform address translation to forward a service request or deny a service request based on blocked service attributes in an IP table in a container-based computing cluster management system |
US11128530B2 (en) | 2018-03-29 | 2021-09-21 | Hewlett Packard Enterprise Development Lp | Container cluster management |
US10762202B2 (en) * | 2018-04-11 | 2020-09-01 | Crowdstrike, Inc. | Securely and efficiently providing user notifications about security actions |
US20190317866A1 (en) | 2018-04-13 | 2019-10-17 | Rubrik, Inc. | Database restoration across cloud environments |
US10382260B1 (en) | 2018-06-07 | 2019-08-13 | Capital One Services, Llc | Utilizing maching learning to reduce cloud instances in a cloud computing environment |
US10853115B2 (en) | 2018-06-25 | 2020-12-01 | Amazon Technologies, Inc. | Execution of auxiliary functions in an on-demand network code execution system |
US10649749B1 (en) | 2018-06-26 | 2020-05-12 | Amazon Technologies, Inc. | Cross-environment application of tracing information for improved code execution |
US11146569B1 (en) | 2018-06-28 | 2021-10-12 | Amazon Technologies, Inc. | Escalation-resistant secure network services using request-scoped authentication information |
US10949237B2 (en) | 2018-06-29 | 2021-03-16 | Amazon Technologies, Inc. | Operating system customization in an on-demand network code execution system |
US11099870B1 (en) | 2018-07-25 | 2021-08-24 | Amazon Technologies, Inc. | Reducing execution times in an on-demand network code execution system using saved machine states |
US10809987B2 (en) | 2018-08-14 | 2020-10-20 | Hyperblox Inc. | Software acceleration platform for supporting decomposed, on-demand network services |
US11099917B2 (en) | 2018-09-27 | 2021-08-24 | Amazon Technologies, Inc. | Efficient state maintenance for execution environments in an on-demand code execution system |
US11243953B2 (en) | 2018-09-27 | 2022-02-08 | Amazon Technologies, Inc. | Mapreduce implementation in an on-demand network code execution system and stream data processing system |
US10778709B2 (en) | 2018-10-31 | 2020-09-15 | International Business Machines Corporation | Cloud-native extensibility provided to security analytics |
US11165697B2 (en) * | 2018-11-16 | 2021-11-02 | Juniper Networks, Inc. | Network controller subclusters for distributed compute deployments |
US11943093B1 (en) | 2018-11-20 | 2024-03-26 | Amazon Technologies, Inc. | Network connection recovery after virtual machine transition in an on-demand network code execution system |
US10931560B2 (en) | 2018-11-23 | 2021-02-23 | Vmware, Inc. | Using route type to determine routing protocol behavior |
US11588712B2 (en) * | 2018-11-30 | 2023-02-21 | Nutanix, Inc. | Systems including interfaces for communication of run-time configuration information |
US10797998B2 (en) | 2018-12-05 | 2020-10-06 | Vmware, Inc. | Route server for distributed routers using hierarchical routing protocol |
US11765174B2 (en) * | 2018-12-07 | 2023-09-19 | Vmware, Inc. | Identity-based access control for cloud applications |
US10938788B2 (en) | 2018-12-12 | 2021-03-02 | Vmware, Inc. | Static routes for policy-based VPN |
US10884812B2 (en) | 2018-12-13 | 2021-01-05 | Amazon Technologies, Inc. | Performance-based hardware emulation in an on-demand network code execution system |
US11496475B2 (en) | 2019-01-04 | 2022-11-08 | Ping Identity Corporation | Methods and systems for data traffic based adaptive security |
US11178208B2 (en) | 2019-01-24 | 2021-11-16 | KLDiscovery Ontrack, LLC | Automatic initialization process for standalone e-discovery machine |
US11010188B1 (en) | 2019-02-05 | 2021-05-18 | Amazon Technologies, Inc. | Simulated data object storage using on-demand computation of data objects |
US11206243B2 (en) * | 2019-03-04 | 2021-12-21 | Cyxtera Cybersecurity, Inc. | Multiple gateway controllers to establish network access |
US11861386B1 (en) | 2019-03-22 | 2024-01-02 | Amazon Technologies, Inc. | Application gateways in an on-demand network code execution system |
US11520750B2 (en) * | 2019-03-28 | 2022-12-06 | Ctera Networks, Ltd. | Global file system for data-intensive applications |
US11388218B2 (en) * | 2019-03-28 | 2022-07-12 | Ctera Networks, Ltd. | Cloud file transfers using cloud file descriptors |
US11119809B1 (en) | 2019-06-20 | 2021-09-14 | Amazon Technologies, Inc. | Virtualization-based transaction handling in an on-demand network code execution system |
US11115404B2 (en) | 2019-06-28 | 2021-09-07 | Amazon Technologies, Inc. | Facilitating service connections in serverless code executions |
US11159528B2 (en) | 2019-06-28 | 2021-10-26 | Amazon Technologies, Inc. | Authentication to network-services using hosted authentication information |
US11190609B2 (en) | 2019-06-28 | 2021-11-30 | Amazon Technologies, Inc. | Connection pooling for scalable network services |
US10996961B2 (en) | 2019-09-27 | 2021-05-04 | Amazon Technologies, Inc. | On-demand indexing of data in input path of object storage service |
US11394761B1 (en) | 2019-09-27 | 2022-07-19 | Amazon Technologies, Inc. | Execution of user-submitted code on a stream of data |
US11023416B2 (en) | 2019-09-27 | 2021-06-01 | Amazon Technologies, Inc. | Data access control system for object storage service based on owner-defined code |
US11386230B2 (en) | 2019-09-27 | 2022-07-12 | Amazon Technologies, Inc. | On-demand code obfuscation of data in input path of object storage service |
US11656892B1 (en) | 2019-09-27 | 2023-05-23 | Amazon Technologies, Inc. | Sequential execution of user-submitted code and native functions |
US11263220B2 (en) | 2019-09-27 | 2022-03-01 | Amazon Technologies, Inc. | On-demand execution of object transformation code in output path of object storage service |
US11023311B2 (en) | 2019-09-27 | 2021-06-01 | Amazon Technologies, Inc. | On-demand code execution in input path of data uploaded to storage service in multiple data portions |
US10908927B1 (en) | 2019-09-27 | 2021-02-02 | Amazon Technologies, Inc. | On-demand execution of object filter code in output path of object storage service |
US11055112B2 (en) | 2019-09-27 | 2021-07-06 | Amazon Technologies, Inc. | Inserting executions of owner-specified code into input/output path of object storage service |
US11250007B1 (en) | 2019-09-27 | 2022-02-15 | Amazon Technologies, Inc. | On-demand execution of object combination code in output path of object storage service |
US11106477B2 (en) | 2019-09-27 | 2021-08-31 | Amazon Technologies, Inc. | Execution of owner-specified code during input/output path to object storage service |
US11550944B2 (en) | 2019-09-27 | 2023-01-10 | Amazon Technologies, Inc. | Code execution environment customization system for object storage service |
US11416628B2 (en) | 2019-09-27 | 2022-08-16 | Amazon Technologies, Inc. | User-specific data manipulation system for object storage service based on user-submitted code |
US11360948B2 (en) | 2019-09-27 | 2022-06-14 | Amazon Technologies, Inc. | Inserting owner-specified data processing pipelines into input/output path of object storage service |
US11706162B2 (en) * | 2019-10-21 | 2023-07-18 | Sap Se | Dynamic, distributed, and scalable single endpoint solution for a service in cloud platform |
US11640315B2 (en) * | 2019-11-04 | 2023-05-02 | Vmware, Inc. | Multi-site virtual infrastructure orchestration of network service in hybrid cloud environments |
US11709698B2 (en) | 2019-11-04 | 2023-07-25 | Vmware, Inc. | Multi-site virtual infrastructure orchestration of network service in hybrid cloud environments |
US11537809B2 (en) | 2019-11-21 | 2022-12-27 | Kyndryl, Inc. | Dynamic container grouping |
US11119826B2 (en) | 2019-11-27 | 2021-09-14 | Amazon Technologies, Inc. | Serverless call distribution to implement spillover while avoiding cold starts |
US10942795B1 (en) | 2019-11-27 | 2021-03-09 | Amazon Technologies, Inc. | Serverless call distribution to utilize reserved capacity without inhibiting scaling |
US11743325B1 (en) | 2019-11-29 | 2023-08-29 | Amazon Technologies, Inc. | Centralized load balancing of resources in cloud edge locations embedded in telecommunications networks |
US11095534B1 (en) * | 2019-11-29 | 2021-08-17 | Amazon Technologies, Inc. | API-based endpoint discovery of resources in cloud edge locations embedded in telecommunications networks |
US11075905B2 (en) * | 2019-12-09 | 2021-07-27 | Google Llc | Requesting and transmitting data for related accounts |
US11425224B2 (en) * | 2020-02-25 | 2022-08-23 | Level 3 Communications, Llc | Disaggregated and distributed composable infrastructure |
US11714682B1 (en) | 2020-03-03 | 2023-08-01 | Amazon Technologies, Inc. | Reclaiming computing resources in an on-demand code execution system |
US11188391B1 (en) | 2020-03-11 | 2021-11-30 | Amazon Technologies, Inc. | Allocating resources to on-demand code executions under scarcity conditions |
CN111447146B (en) * | 2020-03-20 | 2022-04-29 | 上海中通吉网络技术有限公司 | Method, device, equipment and storage medium for dynamically updating physical routing information |
US11775640B1 (en) | 2020-03-30 | 2023-10-03 | Amazon Technologies, Inc. | Resource utilization-based malicious task detection in an on-demand code execution system |
US11231967B2 (en) * | 2020-04-20 | 2022-01-25 | Stratum, Llc | Dynamically allocating and managing cloud workers |
US11348167B2 (en) | 2020-05-28 | 2022-05-31 | EMC IP Holding Company LLC | Method and storage medium for private edge-station auction house |
US11323509B2 (en) * | 2020-05-28 | 2022-05-03 | EMC IP Holding Company LLC | Union formation of edge cloud-native clusters |
US11327673B1 (en) | 2020-10-23 | 2022-05-10 | Oracle International Corporation | Techniques for persisting data across instances of a cloud shell |
US11368306B2 (en) | 2020-08-14 | 2022-06-21 | Oracle International Corporation | Techniques for using signed nonces to secure cloud shells |
US11374792B2 (en) * | 2020-08-14 | 2022-06-28 | Oracle International Corporation | Techniques for utilizing multiple network interfaces for a cloud shell |
US11604705B2 (en) | 2020-08-14 | 2023-03-14 | Nutanix, Inc. | System and method for cloning as SQL server AG databases in a hyperconverged system |
US11153412B1 (en) * | 2020-08-26 | 2021-10-19 | Software Ag | Systems and/or methods for non-intrusive injection of context for service mesh applications |
US11907167B2 (en) | 2020-08-28 | 2024-02-20 | Nutanix, Inc. | Multi-cluster database management services |
US11550713B1 (en) | 2020-11-25 | 2023-01-10 | Amazon Technologies, Inc. | Garbage collection in distributed systems using life cycled storage roots |
US11593270B1 (en) | 2020-11-25 | 2023-02-28 | Amazon Technologies, Inc. | Fast distributed caching using erasure coded object parts |
US11604806B2 (en) | 2020-12-28 | 2023-03-14 | Nutanix, Inc. | System and method for highly available database service |
CN114765580B (en) * | 2020-12-30 | 2023-11-03 | 腾讯科技(深圳)有限公司 | Network acceleration method, device, equipment and storage medium for off-domain network resources |
US11520715B2 (en) | 2021-01-15 | 2022-12-06 | Western Digital Technologies, Inc. | Dynamic allocation of storage resources based on connection type |
US11677726B2 (en) | 2021-02-12 | 2023-06-13 | Raytheon Company | Scalable brokerless messaging strategy with sidecar security container stack |
CN113010897B (en) * | 2021-03-19 | 2023-06-13 | 中国联合网络通信集团有限公司 | Cloud computing security management method and system |
US11892918B2 (en) | 2021-03-22 | 2024-02-06 | Nutanix, Inc. | System and method for availability group database patching |
US11567883B2 (en) | 2021-06-04 | 2023-01-31 | Western Digital Technologies, Inc. | Connection virtualization for data storage device arrays |
US11507321B1 (en) | 2021-06-04 | 2022-11-22 | Western Digital Technologies, Inc. | Managing queue limit overflow for data storage device arrays |
US11388210B1 (en) | 2021-06-30 | 2022-07-12 | Amazon Technologies, Inc. | Streaming analytics using a serverless compute system |
US11888869B2 (en) | 2021-09-02 | 2024-01-30 | Saudi Arabian Oil Company | System and method for securing network users in an enterprise network through cybersecurity controls |
US11546358B1 (en) * | 2021-10-01 | 2023-01-03 | Netskope, Inc. | Authorization token confidence system |
LU500755B1 (en) * | 2021-10-19 | 2023-04-20 | Microsoft Technology Licensing Llc | Confining lateral traversal within a computer network |
US11968280B1 (en) | 2021-11-24 | 2024-04-23 | Amazon Technologies, Inc. | Controlling ingestion of streaming data to serverless function executions |
US12015603B2 (en) | 2021-12-10 | 2024-06-18 | Amazon Technologies, Inc. | Multi-tenant mode for serverless code execution |
US11979277B2 (en) * | 2022-02-02 | 2024-05-07 | Oracle International Corporation | Enhanced network-link architecture for improved end-to-end latency in communication between different cloud environments |
WO2023250023A1 (en) * | 2022-06-22 | 2023-12-28 | Nutanix, Inc. | Database as a service on cloud |
CN117834726A (en) * | 2022-09-28 | 2024-04-05 | 戴尔产品有限公司 | Event subscription to serviceability via bi-directional proxy |
CN115550333B (en) * | 2022-12-02 | 2023-03-31 | 中国电子科技集团公司第十五研究所 | Web-based system and method for accessing application in multi-level multi-domain environment |
CN116545781B (en) * | 2023-07-06 | 2023-11-24 | 广东维信智联科技有限公司 | Cloud access data security management system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070028238A1 (en) * | 2005-07-27 | 2007-02-01 | Bennett Steven M | Virtualization event processing in a layered virtualization architecture |
US20120260247A1 (en) * | 2011-04-05 | 2012-10-11 | International Business Machines Corporation | Fine-Grained Cloud Management Control Using Nested Virtualization |
US20130041987A1 (en) * | 2011-08-11 | 2013-02-14 | Computer Associates Think, Inc. | System and Method for Deploying a Dynamic Virtual Network Address Translation Appliance |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8327353B2 (en) * | 2005-08-30 | 2012-12-04 | Microsoft Corporation | Hierarchical virtualization with a multi-level virtualization mechanism |
US8332688B1 (en) * | 2009-07-21 | 2012-12-11 | Adobe Systems Incorporated | Failover and recovery of a computing application hosted by a virtual instance of a machine |
US8566822B2 (en) * | 2009-07-22 | 2013-10-22 | Broadcom Corporation | Method and system for distributing hypervisor functionality over multiple physical devices in a network and configuring sub-hypervisor to control the virtual machines |
US20110078303A1 (en) | 2009-09-30 | 2011-03-31 | Alcatel-Lucent Usa Inc. | Dynamic load balancing and scaling of allocated cloud resources in an enterprise network |
US8301746B2 (en) | 2010-01-26 | 2012-10-30 | International Business Machines Corporation | Method and system for abstracting non-functional requirements based deployment of virtual machines |
US9201677B2 (en) * | 2011-05-23 | 2015-12-01 | Intelligent Intellectual Property Holdings 2 Llc | Managing data input/output operations |
US8978030B2 (en) * | 2011-04-07 | 2015-03-10 | Infosys Limited | Elastic provisioning of resources via distributed virtualization |
US9465632B2 (en) * | 2012-02-04 | 2016-10-11 | Global Supercomputing Corporation | Parallel hardware hypervisor for virtualizing application-specific supercomputers |
-
2012
- 2012-08-21 US US13/590,944 patent/US9563480B2/en active Active
-
2013
- 2013-08-16 WO PCT/US2013/055264 patent/WO2014031473A2/en active Application Filing
-
2017
- 2017-02-06 US US15/425,484 patent/US20170223117A1/en not_active Abandoned
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070028238A1 (en) * | 2005-07-27 | 2007-02-01 | Bennett Steven M | Virtualization event processing in a layered virtualization architecture |
US20120260247A1 (en) * | 2011-04-05 | 2012-10-11 | International Business Machines Corporation | Fine-Grained Cloud Management Control Using Nested Virtualization |
US20130041987A1 (en) * | 2011-08-11 | 2013-02-14 | Computer Associates Think, Inc. | System and Method for Deploying a Dynamic Virtual Network Address Translation Appliance |
Cited By (147)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150288651A1 (en) * | 2012-11-26 | 2015-10-08 | Huawei Technologies Co., Ltd. | Ip packet processing method and apparatus, and network system |
US10454880B2 (en) * | 2012-11-26 | 2019-10-22 | Huawei Technologies Co., Ltd. | IP packet processing method and apparatus, and network system |
US10749711B2 (en) | 2013-07-10 | 2020-08-18 | Nicira, Inc. | Network-link method useful for a last-mile connectivity in an edge-gateway multipath system |
US11050588B2 (en) | 2013-07-10 | 2021-06-29 | Nicira, Inc. | Method and system of overlay flow control |
US11212140B2 (en) | 2013-07-10 | 2021-12-28 | Nicira, Inc. | Network-link method useful for a last-mile connectivity in an edge-gateway multipath system |
US11804988B2 (en) | 2013-07-10 | 2023-10-31 | Nicira, Inc. | Method and system of overlay flow control |
US11770364B2 (en) | 2013-12-17 | 2023-09-26 | Amazon Technologies, Inc. | Private network peering in virtual network environments |
US10218633B2 (en) * | 2014-03-28 | 2019-02-26 | Amazon Technologies, Inc. | Implementation of a service that coordinates the placement and execution of containers |
US20150281111A1 (en) * | 2014-03-28 | 2015-10-01 | Amazon Technologies, Inc. | Implementation of a service that coordinates the placement and execution of containers |
US11677720B2 (en) | 2015-04-13 | 2023-06-13 | Nicira, Inc. | Method and system of establishing a virtual private network in a cloud service for branch networking |
US10805272B2 (en) | 2015-04-13 | 2020-10-13 | Nicira, Inc. | Method and system of establishing a virtual private network in a cloud service for branch networking |
US11374904B2 (en) | 2015-04-13 | 2022-06-28 | Nicira, Inc. | Method and system of a cloud-based multipath routing protocol |
US10498652B2 (en) | 2015-04-13 | 2019-12-03 | Nicira, Inc. | Method and system of application-aware routing with crowdsourcing |
US11444872B2 (en) | 2015-04-13 | 2022-09-13 | Nicira, Inc. | Method and system of application-aware routing with crowdsourcing |
US10222992B2 (en) * | 2016-01-30 | 2019-03-05 | Western Digital Technologies, Inc. | Synchronization method and apparatus for an interconnection network using parallel-headerless TDMA routing |
US20170220250A1 (en) * | 2016-01-30 | 2017-08-03 | HGST Netherlands B.V. | Synchronization method and apparatus for an interconnection network using parallel-headerless tdma routing |
US11921597B2 (en) | 2016-04-29 | 2024-03-05 | Netapp, Inc. | Cross-platform replication |
US20180276090A1 (en) * | 2016-04-29 | 2018-09-27 | Netapp Inc. | Cross-platform replication |
US10585768B2 (en) * | 2016-04-29 | 2020-03-10 | Netapp Inc. | Cross-platform replication |
US11347605B2 (en) | 2016-04-29 | 2022-05-31 | Netapp Inc. | Cross-platform replication |
US11689413B2 (en) * | 2016-05-24 | 2023-06-27 | Apstra, Inc. | Configuring system resources for different reference architectures |
US20210367837A1 (en) * | 2016-05-24 | 2021-11-25 | Apstra, Inc. | Configuring system resources for different reference architectures |
US20180027049A1 (en) * | 2016-07-20 | 2018-01-25 | Adbrain Ltd | Computing system and method of operating the computer system |
US10992568B2 (en) | 2017-01-31 | 2021-04-27 | Vmware, Inc. | High performance software-defined core network |
US11252079B2 (en) | 2017-01-31 | 2022-02-15 | Vmware, Inc. | High performance software-defined core network |
US11706126B2 (en) | 2017-01-31 | 2023-07-18 | Vmware, Inc. | Method and apparatus for distributed data network traffic optimization |
US11706127B2 (en) | 2017-01-31 | 2023-07-18 | Vmware, Inc. | High performance software-defined core network |
US11606286B2 (en) | 2017-01-31 | 2023-03-14 | Vmware, Inc. | High performance software-defined core network |
US11121962B2 (en) | 2017-01-31 | 2021-09-14 | Vmware, Inc. | High performance software-defined core network |
US11700196B2 (en) | 2017-01-31 | 2023-07-11 | Vmware, Inc. | High performance software-defined core network |
US11349722B2 (en) | 2017-02-11 | 2022-05-31 | Nicira, Inc. | Method and system of connecting to a multipath hub in a cluster |
US10574528B2 (en) | 2017-02-11 | 2020-02-25 | Nicira, Inc. | Network multi-source inbound quality of service methods and systems |
US10778528B2 (en) | 2017-02-11 | 2020-09-15 | Nicira, Inc. | Method and system of connecting to a multipath hub in a cluster |
US11582298B2 (en) * | 2017-05-04 | 2023-02-14 | Amazon Technologies, Inc. | Coordinating inter-region operations in provider network environments |
US11902367B2 (en) * | 2017-05-04 | 2024-02-13 | Amazon Technologies, Inc. | Coordinating inter-region operations in provider network environments |
US20230283661A1 (en) * | 2017-05-04 | 2023-09-07 | Amazon Technologies, Inc. | Coordinating inter-region operations in provider network environments |
US11533248B2 (en) | 2017-06-22 | 2022-12-20 | Nicira, Inc. | Method and system of resiliency in cloud-delivered SD-WAN |
US10523539B2 (en) | 2017-06-22 | 2019-12-31 | Nicira, Inc. | Method and system of resiliency in cloud-delivered SD-WAN |
US10938693B2 (en) | 2017-06-22 | 2021-03-02 | Nicira, Inc. | Method and system of resiliency in cloud-delivered SD-WAN |
US10761869B2 (en) * | 2017-06-26 | 2020-09-01 | Wangsu Science & Technology Co., Ltd. | Cloud platform construction method and cloud platform storing image files in storage backend cluster according to image file type |
US20190004844A1 (en) * | 2017-06-26 | 2019-01-03 | Wangsu Science & Technology Co., Ltd. | Cloud platform construction method and cloud platform |
US10838788B2 (en) | 2017-09-30 | 2020-11-17 | Oracle International Corporation | Real-time debugging instances in a deployed container platform |
US11755393B2 (en) | 2017-09-30 | 2023-09-12 | Oracle International Corporation | API registry in a container platform for automatically generating client code libraries |
US10824489B2 (en) * | 2017-09-30 | 2020-11-03 | Oracle International Corporation | Dynamic node rebalancing between container platforms |
US11681573B2 (en) | 2017-09-30 | 2023-06-20 | Oracle International Corporation | API registry in a container platform providing property-based API functionality |
US11132241B2 (en) | 2017-09-30 | 2021-09-28 | Oracle International Corporation | API registry in a container platform for automatically generating client code libraries |
US11115480B2 (en) | 2017-10-02 | 2021-09-07 | Vmware, Inc. | Layer four optimization for a virtual network defined over public cloud |
US10999100B2 (en) | 2017-10-02 | 2021-05-04 | Vmware, Inc. | Identifying multiple nodes in a virtual network defined over a set of public clouds to connect to an external SAAS provider |
US11606225B2 (en) | 2017-10-02 | 2023-03-14 | Vmware, Inc. | Identifying multiple nodes in a virtual network defined over a set of public clouds to connect to an external SAAS provider |
US10686625B2 (en) | 2017-10-02 | 2020-06-16 | Vmware, Inc. | Defining and distributing routes for a virtual network |
US11089111B2 (en) | 2017-10-02 | 2021-08-10 | Vmware, Inc. | Layer four optimization for a virtual network defined over public cloud |
US10841131B2 (en) | 2017-10-02 | 2020-11-17 | Vmware, Inc. | Distributed WAN security gateway |
US10958479B2 (en) | 2017-10-02 | 2021-03-23 | Vmware, Inc. | Selecting one node from several candidate nodes in several public clouds to establish a virtual network that spans the public clouds |
US10666460B2 (en) | 2017-10-02 | 2020-05-26 | Vmware, Inc. | Measurement based routing through multiple public clouds |
US10608844B2 (en) | 2017-10-02 | 2020-03-31 | Vmware, Inc. | Graph based routing through multiple public clouds |
US11894949B2 (en) | 2017-10-02 | 2024-02-06 | VMware LLC | Identifying multiple nodes in a virtual network defined over a set of public clouds to connect to an external SaaS provider |
US10805114B2 (en) | 2017-10-02 | 2020-10-13 | Vmware, Inc. | Processing data messages of a virtual network that are sent to and received from external service machines |
US10959098B2 (en) | 2017-10-02 | 2021-03-23 | Vmware, Inc. | Dynamically specifying multiple public cloud edge nodes to connect to an external multi-computer node |
US11102032B2 (en) | 2017-10-02 | 2021-08-24 | Vmware, Inc. | Routing data message flow through multiple public clouds |
US11005684B2 (en) * | 2017-10-02 | 2021-05-11 | Vmware, Inc. | Creating virtual networks spanning multiple public clouds |
US11855805B2 (en) | 2017-10-02 | 2023-12-26 | Vmware, Inc. | Deploying firewall for virtual network defined over public cloud infrastructure |
US11895194B2 (en) | 2017-10-02 | 2024-02-06 | VMware LLC | Layer four optimization for a virtual network defined over public cloud |
US10999165B2 (en) | 2017-10-02 | 2021-05-04 | Vmware, Inc. | Three tiers of SaaS providers for deploying compute and network infrastructure in the public cloud |
US11516049B2 (en) | 2017-10-02 | 2022-11-29 | Vmware, Inc. | Overlay network encapsulation to forward data message flows through multiple public cloud datacenters |
US10594516B2 (en) | 2017-10-02 | 2020-03-17 | Vmware, Inc. | Virtual network provider |
US10778466B2 (en) | 2017-10-02 | 2020-09-15 | Vmware, Inc. | Processing data messages of a virtual network that are sent to and received from external service machines |
US10992558B1 (en) | 2017-11-06 | 2021-04-27 | Vmware, Inc. | Method and apparatus for distributed data network traffic optimization |
US11223514B2 (en) | 2017-11-09 | 2022-01-11 | Nicira, Inc. | Method and system of a dynamic high-availability mode based on current wide area network connectivity |
US11323307B2 (en) | 2017-11-09 | 2022-05-03 | Nicira, Inc. | Method and system of a dynamic high-availability mode based on current wide area network connectivity |
US11902086B2 (en) | 2017-11-09 | 2024-02-13 | Nicira, Inc. | Method and system of a dynamic high-availability mode based on current wide area network connectivity |
US11336737B2 (en) | 2017-11-14 | 2022-05-17 | Google Llc | Opt-out compliance |
US10659544B2 (en) * | 2017-11-14 | 2020-05-19 | Google Llc | Opt-out compliance |
FR3075422A1 (en) * | 2017-12-19 | 2019-06-21 | Atos Information Technology GmbH | PUBLIC CLOUD PLATFORM |
WO2019121677A1 (en) * | 2017-12-19 | 2019-06-27 | Atos Information Technology GmbH | Managed public cloud |
US11228573B1 (en) * | 2018-03-09 | 2022-01-18 | Equinix, Inc. | Application programming interface exchange |
US11388272B2 (en) * | 2018-03-30 | 2022-07-12 | Intel Corporation | Technologies for network packet processing between cloud and telecommunications networks |
WO2019213427A1 (en) * | 2018-05-04 | 2019-11-07 | Laibson Benjamin William | Emulation of cloud computing service regions |
US10958734B2 (en) | 2018-05-04 | 2021-03-23 | Digital Age Experts, Llc | Emulation of cloud computing service regions |
US11223685B2 (en) * | 2018-05-04 | 2022-01-11 | Digital Age Experts, Llc | Emulation of cloud computing service regions |
US11647079B2 (en) | 2018-05-04 | 2023-05-09 | Digital Age Experts, Llc | Emulation of cloud computing service regions |
US10805406B2 (en) * | 2018-06-21 | 2020-10-13 | Microsoft Technology Licensing, Llc | Zone redundant computing services using multiple local services in distributed computing systems |
US11016793B2 (en) * | 2018-11-26 | 2021-05-25 | Red Hat, Inc. | Filtering based containerized virtual machine networking |
US11032381B2 (en) * | 2019-06-19 | 2021-06-08 | Servicenow, Inc. | Discovery and storage of resource tags |
US11153230B2 (en) | 2019-08-27 | 2021-10-19 | Vmware, Inc. | Having a remote device use a shared virtual network to access a dedicated virtual network defined over public clouds |
US11171885B2 (en) | 2019-08-27 | 2021-11-09 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US10999137B2 (en) | 2019-08-27 | 2021-05-04 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US11121985B2 (en) | 2019-08-27 | 2021-09-14 | Vmware, Inc. | Defining different public cloud virtual networks for different entities based on different sets of measurements |
US11606314B2 (en) | 2019-08-27 | 2023-03-14 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US11258728B2 (en) | 2019-08-27 | 2022-02-22 | Vmware, Inc. | Providing measurements of public cloud connections |
US11018995B2 (en) | 2019-08-27 | 2021-05-25 | Vmware, Inc. | Alleviating congestion in a virtual network deployed over public clouds for an entity |
US11252105B2 (en) | 2019-08-27 | 2022-02-15 | Vmware, Inc. | Identifying different SaaS optimal egress nodes for virtual networks of different entities |
US11310170B2 (en) | 2019-08-27 | 2022-04-19 | Vmware, Inc. | Configuring edge nodes outside of public clouds to use routes defined through the public clouds |
US11212238B2 (en) | 2019-08-27 | 2021-12-28 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US11252106B2 (en) | 2019-08-27 | 2022-02-15 | Vmware, Inc. | Alleviating congestion in a virtual network deployed over public clouds for an entity |
US11831414B2 (en) | 2019-08-27 | 2023-11-28 | Vmware, Inc. | Providing recommendations for implementing virtual networks |
US11863612B2 (en) * | 2019-09-19 | 2024-01-02 | Guizhou Baishancloud Technology Co., Ltd. | Network edge computing and network edge computation scheduling method, device and medium |
US20220345521A1 (en) * | 2019-09-19 | 2022-10-27 | Guizhou Baishancloud Technology Co., Ltd. | Network edge computing method, apparatus, device and medium |
US11611507B2 (en) | 2019-10-28 | 2023-03-21 | Vmware, Inc. | Managing forwarding elements at edge nodes connected to a virtual network |
US11044190B2 (en) | 2019-10-28 | 2021-06-22 | Vmware, Inc. | Managing forwarding elements at edge nodes connected to a virtual network |
US11489783B2 (en) | 2019-12-12 | 2022-11-01 | Vmware, Inc. | Performing deep packet inspection in a software defined wide area network |
US11394640B2 (en) | 2019-12-12 | 2022-07-19 | Vmware, Inc. | Collecting and analyzing data regarding flows associated with DPI parameters |
US11716286B2 (en) | 2019-12-12 | 2023-08-01 | Vmware, Inc. | Collecting and analyzing data regarding flows associated with DPI parameters |
US11438789B2 (en) | 2020-01-24 | 2022-09-06 | Vmware, Inc. | Computing and using different path quality metrics for different service classes |
US11418997B2 (en) | 2020-01-24 | 2022-08-16 | Vmware, Inc. | Using heart beats to monitor operational state of service classes of a QoS aware network link |
US11606712B2 (en) | 2020-01-24 | 2023-03-14 | Vmware, Inc. | Dynamically assigning service classes for a QOS aware network link |
US11689959B2 (en) | 2020-01-24 | 2023-06-27 | Vmware, Inc. | Generating path usability state for different sub-paths offered by a network link |
US11722925B2 (en) | 2020-01-24 | 2023-08-08 | Vmware, Inc. | Performing service class aware load balancing to distribute packets of a flow among multiple network links |
US20210314419A1 (en) * | 2020-04-01 | 2021-10-07 | Kang Wing Leung | Direct network connections using cloud instance for internet application services |
US20210409409A1 (en) * | 2020-06-29 | 2021-12-30 | Illumina, Inc. | Temporary cloud provider credentials via secure discovery framework |
US11245641B2 (en) | 2020-07-02 | 2022-02-08 | Vmware, Inc. | Methods and apparatus for application aware hub clustering techniques for a hyper scale SD-WAN |
US11477127B2 (en) | 2020-07-02 | 2022-10-18 | Vmware, Inc. | Methods and apparatus for application aware hub clustering techniques for a hyper scale SD-WAN |
US11363124B2 (en) | 2020-07-30 | 2022-06-14 | Vmware, Inc. | Zero copy socket splicing |
US11709710B2 (en) | 2020-07-30 | 2023-07-25 | Vmware, Inc. | Memory allocator for I/O operations |
US20220086156A1 (en) * | 2020-09-16 | 2022-03-17 | Jpmorgan Chase Bank, N.A. | Method and system for providing platform as a service with security access controls |
US11736485B2 (en) * | 2020-09-16 | 2023-08-22 | Jpmorgan Chase Bank, N.A. | Method and system for providing platform as a service with security access controls |
US11934886B2 (en) | 2020-10-31 | 2024-03-19 | Nutanix, Inc. | Intra-footprint computing cluster bring-up |
US11444865B2 (en) | 2020-11-17 | 2022-09-13 | Vmware, Inc. | Autonomous distributed forwarding plane traceability based anomaly detection in application traffic for hyper-scale SD-WAN |
US11575591B2 (en) | 2020-11-17 | 2023-02-07 | Vmware, Inc. | Autonomous distributed forwarding plane traceability based anomaly detection in application traffic for hyper-scale SD-WAN |
US11575600B2 (en) | 2020-11-24 | 2023-02-07 | Vmware, Inc. | Tunnel-less SD-WAN |
US11601356B2 (en) | 2020-12-29 | 2023-03-07 | Vmware, Inc. | Emulating packet flows to assess network links for SD-WAN |
US11929903B2 (en) | 2020-12-29 | 2024-03-12 | VMware LLC | Emulating packet flows to assess network links for SD-WAN |
US20220206832A1 (en) * | 2020-12-31 | 2022-06-30 | Nutanix, Inc. | Configuring virtualization system images for a computing cluster |
US11734044B2 (en) * | 2020-12-31 | 2023-08-22 | Nutanix, Inc. | Configuring virtualization system images for a computing cluster |
US20240069952A1 (en) * | 2020-12-31 | 2024-02-29 | Nutanix, Inc. | Configuring virtualization system images for a computing cluster |
US11792127B2 (en) | 2021-01-18 | 2023-10-17 | Vmware, Inc. | Network-aware load balancing |
US11979325B2 (en) | 2021-01-28 | 2024-05-07 | VMware LLC | Dynamic SD-WAN hub cluster scaling with machine learning |
US11509571B1 (en) | 2021-05-03 | 2022-11-22 | Vmware, Inc. | Cost-based routing mesh for facilitating routing through an SD-WAN |
US11381499B1 (en) | 2021-05-03 | 2022-07-05 | Vmware, Inc. | Routing meshes for facilitating routing through an SD-WAN |
US12009987B2 (en) | 2021-05-03 | 2024-06-11 | VMware LLC | Methods to support dynamic transit paths through hub clustering across branches in SD-WAN |
US11388086B1 (en) | 2021-05-03 | 2022-07-12 | Vmware, Inc. | On demand routing mesh for dynamically adjusting SD-WAN edge forwarding node roles to facilitate routing through an SD-WAN |
US11637768B2 (en) | 2021-05-03 | 2023-04-25 | Vmware, Inc. | On demand routing mesh for routing packets through SD-WAN edge forwarding nodes in an SD-WAN |
US11582144B2 (en) | 2021-05-03 | 2023-02-14 | Vmware, Inc. | Routing mesh to provide alternate routes through SD-WAN edge forwarding nodes based on degraded operational states of SD-WAN hubs |
US11729065B2 (en) | 2021-05-06 | 2023-08-15 | Vmware, Inc. | Methods for application defined virtual network service among multiple transport in SD-WAN |
US11489720B1 (en) | 2021-06-18 | 2022-11-01 | Vmware, Inc. | Method and apparatus to evaluate resource elements and public clouds for deploying tenant deployable elements based on harvested performance metrics |
US12015536B2 (en) | 2021-06-18 | 2024-06-18 | VMware LLC | Method and apparatus for deploying tenant deployable elements across public clouds based on harvested performance metrics of types of resource elements in the public clouds |
US11777944B2 (en) | 2021-06-29 | 2023-10-03 | International Business Machines Corporation | Scalable authentication management |
US11375005B1 (en) | 2021-07-24 | 2022-06-28 | Vmware, Inc. | High availability solutions for a secure access service edge application |
US11900172B2 (en) * | 2021-07-30 | 2024-02-13 | Nutanix, Inc. | Computing cluster bring-up on public cloud infrastructure using expressed intents |
US20230036454A1 (en) * | 2021-07-30 | 2023-02-02 | Nutanix, Inc. | Computing cluster bring-up on public cloud infrastructure using expressed intents |
US11954473B2 (en) | 2021-09-20 | 2024-04-09 | Microstrategy Incorporated | Deployment architecture for multi-tenant cloud computing systems |
US11943146B2 (en) | 2021-10-01 | 2024-03-26 | VMware LLC | Traffic prioritization in SD-WAN |
WO2023076109A1 (en) * | 2021-10-25 | 2023-05-04 | Tencent America LLC | Method and apparatus for using local area network as service for edge cloud |
US11861342B2 (en) | 2022-01-28 | 2024-01-02 | Microstrategy Incorporated | Enhanced cloud-computing environment deployment |
US11558258B1 (en) * | 2022-01-31 | 2023-01-17 | Walmart Apollo, Llc | Methods and apparatuses for identifying characteristics of users of electronic resources |
US11909815B2 (en) | 2022-06-06 | 2024-02-20 | VMware LLC | Routing based on geolocation costs |
US12034587B1 (en) | 2023-03-27 | 2024-07-09 | VMware LLC | Identifying and remediating anomalies in a self-healing network |
US12034630B2 (en) | 2023-07-17 | 2024-07-09 | VMware LLC | Method and apparatus for distributed data network traffic optimization |
Also Published As
Publication number | Publication date |
---|---|
WO2014031473A2 (en) | 2014-02-27 |
US20140059226A1 (en) | 2014-02-27 |
US9563480B2 (en) | 2017-02-07 |
WO2014031473A3 (en) | 2015-02-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10516623B2 (en) | Pluggable allocation in a cloud computing system | |
US9563480B2 (en) | Multi-level cloud computing system | |
US9544289B2 (en) | Method and system for identity-based authentication of virtual machines | |
US9471384B2 (en) | Method and system for utilizing spare cloud resources | |
US10680946B2 (en) | Adding multi-tenant awareness to a network packet processing device on a software defined network (SDN) | |
US9489395B2 (en) | System and method for exposing cloud stored data to a content delivery network | |
US10069690B2 (en) | Methods and systems of tracking and verifying records of system change events in a distributed network system | |
US20130205028A1 (en) | Elastic, Massively Parallel Processing Data Warehouse | |
US20130238785A1 (en) | System and Method for Metadata Discovery and Metadata-Aware Scheduling | |
CN111164571B (en) | Control plane function virtualization based on security processing in cloud system | |
US20170264622A1 (en) | Providing a virtual security appliance architecture to a virtual cloud infrastructure | |
AU2013266420B2 (en) | Pluggable allocation in a cloud computing system | |
US11451643B2 (en) | Managed traffic processing for applications with multiple constituent services | |
McDonald | High-Performance Service-Oriented Computing |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RACKSPACE US, INC., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MESSERLI, ANTONY JOEL;VOCCIO, PAUL;HINCHER, JOHN CARLISLE;REEL/FRAME:041185/0342 Effective date: 20120802 |
|
AS | Assignment |
Owner name: CITIBANK, N.A., NEW YORK Free format text: SECURITY INTEREST;ASSIGNOR:RACKSPACE US, INC.;REEL/FRAME:045562/0371 Effective date: 20180416 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |