JP2022538826A - Virtualization block storage server in cloud provider board expansion - Google Patents

Abstract

A first block storage server virtual machine that hosts a first volume using one or more storage devices of the computer system is executed by the computer system. A second virtual machine that has access to the virtual block storage device is executed by the computer system. A block storage client is executed by a computer system. A first block storage operation is received by the block storage client from the second virtual machine, and the first block storage operation is performed on the virtual block storage device. The message is sent by the block storage client to the first block storage server virtual machine to cause the first block storage server virtual machine to perform block storage operations on the first volume. [Selection drawing] Fig. 4

Description

Many businesses and other organizations operate computer networks interconnecting many computing systems to support their operations, e.g. ) are co-located, or alternatively located in multiple different geographic locations (eg, connected via one or more intermediate private or public networks). For example, data centers housing a significant number of interconnected computing systems have become commonplace, including private data centers operated by and on behalf of a single organization. , and public data centers operated by entities as a business to provide computing resources to customers. Some public data center operators provide network access, power, and secure installation facilities for hardware owned by various customers, while other public data center operators make available for use by their customers. Offer a "full service" facility that also includes hardware resources enabled. However, as the size and scope of typical data centers increase, the task of provisioning, overseeing, and managing physical computing resources becomes increasingly complex.

The advent of general-purpose hardware virtualization technology has brought benefits in terms of managing large-scale computing resources for many customers with diverse needs, allowing various computing resources to be efficiently and effectively shared by multiple customers. Allows for secure sharing. For example, virtualization technology encourages sharing a single physical computing machine among multiple users by providing each user with one or more virtual machines hosted by the single physical computing machine. can make it possible. Each such virtual machine is a software simulation that functions as a unique logical computing system that provides the user with the illusion of being the sole operator and administrator of a given hardware computing resource, while also providing various virtual machines. It also provides application isolation and security between machines. Additionally, some virtualization technologies are capable of providing virtual resources that span two or more physical resources, such as a single virtual machine with multiple virtual processors that span multiple distinct physical computing systems. be. As another example, virtualization technology may allow data storage hardware to be shared among multiple users by providing each user with a virtualized data store that may be distributed across multiple data storage devices; Each such virtualized data store functions as a unique logical data store that gives the user the illusion that he or she is the sole operator and manager of the data storage resources.

Different virtual machine types, optimized for different types of applications such as compute-intensive applications, memory-intensive applications, etc., are set up in the data centers of several cloud computing provider networks according to client requirements. can be In addition, higher-level services that rely on such Provider Network's Virtual Computing Services, such as some database services whose database instances are instantiated using Virtual Computing Service virtual machines, are also available to Provider Network Clients. can be possible. However, for some types of applications, such as applications that process fairly large amounts of data that need to be stored on the customer's premises outside the provider network, virtualization using hardware located in the provider network's data center A service that is limited to providing commoditized resources may, for example, be suboptimal for latency-related and/or other reasons.

Various embodiments according to the present disclosure are described with reference to the following drawings.

FIG. 4 is a block diagram illustrating an exemplary provider network extended by provider board extensions located within the network external to the provider network, in accordance with at least some embodiments; FIG. 4 is a block diagram illustrating an example provider board extension, according to at least some embodiments; FIG. 4 is a block diagram illustrating exemplary connections between a provider network and provider board extensions, according to at least some embodiments; 1 is a block diagram illustrating an example virtualized block storage system, according to at least some embodiments; FIG. 1 is a block diagram illustrating an example virtualized block storage system, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using a first technique, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using a second technology, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for booting additional compute instances of provider board extensions from a block storage server, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for managing virtualized block storage servers, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for providing volume mappings to block storage clients, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary system for tracking volume mappings, according to at least some embodiments; FIG. FIG. 11 is a flow diagram illustrating operations of a method for starting a virtualized block storage server, according to at least some embodiments; FIG. 4 is a flow diagram illustrating operations of a method for using a virtualized block storage server, according to at least some embodiments; FIG. 10 is a flow diagram illustrating operations of a method for managing virtualized block storage servers in a provider board extension, according to at least some embodiments; 1 illustrates an exemplary provider network environment, according to at least some embodiments; 1 is a block diagram of an exemplary provider network for providing storage services and hardware virtualization services to customers, according to at least some embodiments; FIG. 1 is a block diagram illustrating an exemplary computing device that may be used with at least some embodiments; FIG.

The present disclosure provides methods for configuring provider board extensions to communicate with networks external to the provider network and for providing virtualized resources on board extensions that are the same or similar to resources available in the provider network. , apparatus, systems, and non-transitory computer-readable storage media. Provider A network operator (or provider) offers its users (or customers) the ability to utilize one or more of various types of computing-related resources, such as computing resources (e.g., running virtual machines (VMs) and/or containers, running batch jobs, executing code without provisioning servers)), data/storage resources (e.g. object storage, block level storage, data archive storage, databases and databases) tables, etc.), network-related resources (e.g., configuring virtual networks containing groups of compute resources, content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build/deployment services), Access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other computing resources may be provided as services.

Provider network operators often offer these and other computing resources as a service that relies on virtualization technology. For example, using virtualization technology, compute instances (e.g., VMs using guest operating systems (OS) running using a hypervisor that may or may not run additionally on the underlying host OS; containers that may or may not run in VMs, instances that can run on “bare metal” hardware without an underlying hypervisor), and use a single electronic device. can implement one or more compute instances. Thus, users can directly utilize compute instances provided by instance management services (sometimes referred to as hardware virtualization services) hosted by provider networks to perform various computing tasks. Additionally or alternatively, a user may indirectly utilize Compute Instances by submitting code to be executed by a provider network (e.g., utilized by an on-demand code execution service), which in turn utilizes Compute Instances. to execute code (generally without any control or knowledge of the underlying compute instance(s) associated with the user).

Resources that support both services that provide computing-related resources to users and those computing-related resources provisioned to users may generally be referred to as provider network substrates. Such resources typically include hardware and software in the form of many networked computer systems. Provider network substrate traffic and operations can be broadly subdivided into two categories, control plane traffic carried on the logical control plane and data plane operations carried on the logical data plane, in various embodiments. The data plane represents the movement of user data through the distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. A control plane generally includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic is typically used to establish isolated virtual networks for various customers, monitor resource usage and health, identify specific hosts or servers on which requested compute instances are launched, and optionally Includes management operations such as provisioning additional hardware. The data plane includes customer resources (eg, computing instances, containers, block storage volumes, databases, file storage) implemented in provider networks. Data plane traffic generally includes non-management operations such as transferring data to and from customer resources. Control plane components are typically implemented on a separate set of servers from the data plane servers, and control plane traffic and data plane traffic may be sent over separate/different networks. In some embodiments, control plane traffic and data plane traffic can be supported by different protocols. In some embodiments, messages (eg, packets) sent over the provider network include flags that indicate whether the traffic is control plane traffic or data plane traffic. In some embodiments, the payload of traffic can be inspected to determine its type (eg, control plane or data plane). Other techniques for distinguishing traffic types are possible.

Some customer applications are easily migrated to the provider network environment, while some customer workloads remain on-premises due to low latency, large amounts of data, data security, or other customer data processing demands ("on-premises"). Exemplary on-premise environments include customer data centers, robotic integration, field locations, colocation facilities, telecommunications facilities (eg, near cell towers), and the like. To satisfy customer demands, the present disclosure relates to deploying resources, such as substrates, on-premises. The term "provider substrate extension" (PSE) means that the customer can deploy on-premises (e.g., geographically remote from the provider network) but with the same or similar functionality (e.g., virtualized) as provided in the provider network. A collection of resources (e.g., hardware, software, firmware, configuration metadata, etc.) that provide computing resources). Such resources may be physically distributed as one or more computer systems or servers distributed in racks or cabinets such as those commonly found in on-premises locations. PSEs can offer customers a set of features and capabilities that can be deployed on-premises, similar to the provider network features described above. Effectively, from the provider network customer's perspective, a PSE represents a local extension of the provider network's capabilities that can be set up in any physical location (e.g., in terms of physical space, power, Internet access, etc.) that can accommodate a PSE. . From the perspective of the provider network itself, the PSE can be viewed as being physically located at a customer-selected deployment site while being virtually located in the same provider network data center as the core provider network board. In at least some embodiments, the customer physically hosting the PSE grants permission to its own customers (e.g., other users of the provider network) so that those users can launch instances and to host its respective workload within the PSE at its on-premises location, optionally allowing that workload to access the customer's network.

In at least some embodiments, the PSE is pre-configured, e.g., by the provider network operator, with an appropriate combination of hardware, software, and/or firmware elements to support various types of computing-related resources. and support in a manner that meets various local data processing needs without compromising the security of the provider network itself or any other customer of the provider network. In at least some embodiments, PSEs are generally governed by the same or similar set of interfaces that customers use to access computing-related resources within provider networks. For example, a customer can access the Provider Network via the same application programming interface (API) or console-based interface that it otherwise uses to provision, manage, and operate computing-related resources within the Provider Network. As such, computing-related resources can be provisioned, managed, and operated within an on-premise PSE or multiple PSEs at various deployment sites.

In at least some embodiments, provider network resources instantiate various networking components to ensure secure and reliable communication between the provider network and the PSE. Such components can establish one or more secure tunnels (eg, VPN) with the PSE. Such components can further split control plane traffic and data plane traffic and treat each type of traffic differently based on factors including the direction of the traffic (eg, direction to and from the PSE). In at least some embodiments, the control plane service dynamically provisions and configures these networking components for deployed PSEs. Such control plane services can monitor each PSE's network components and invoke self-healing or recovery mechanisms designed to prevent loss of communication with a PSE due to failures occurring within the provider network.

One service commonly offered to customers of provider networks is the block storage service, where instances are presented with a logical view of physical storage resources and thus can act as virtualized persistent disks for instances. The mapping of logical storage space to actual physical locations of storage is handled by the virtualization system. Volumes can be replicated one or more times to provide high availability and durability to customers, and replicas are typically stored on different servers. A customer can attach one or more block storage volumes to an instance, and clients supporting the instance can use the virtualized block storage volumes to perform block-based operations on the instance. For example, a customer may boot a particular instance from a given boot volume (e.g., the volume containing the operating system) and another attached volume to support data stored by customer applications run by the instance. can be specified to have To provide high flexibility in terms of available storage, provider networks decouple the physical storage devices supporting a given attached volume from the computing resources supporting a given compute instance. Traditionally, a fleet of block storage servers that support block storage services would divide the attached physical storage devices (eg, solid state drives, magnetic drives, etc.) into a number of logical volumes. A single block storage server can support hundreds or even thousands of instances of storage resources. These block storage servers are commonly run in "bare metal" configurations. The server software runs within an operating system environment that runs directly on dedicated server hardware, for example, instead of on virtual machines or containers.

This bare-metal block storage server design may not be well suited for substrate expansion, as the capacity of provider substrate extensions may be significantly limited compared to the availability zone capacity. For example, some board expansions may only have a single server (or another small number of servers), so it may not be possible to dedicate an entire server to a block of storage resources, thereby allowing the server are no longer available for Compute instances. To address this challenge, embodiments of the present disclosure virtualize block storage services so that they can be launched within compute instances, resulting in more flexible and efficient use of limited capacity. to For example, a single server can be configured to host both compute instances and instances that virtualize its attached volumes (e.g., block storage servers), making use of the PSE's finite set of resources provide high adaptability to Also, a single server may be divided into isolated failure domains to host multiple block storage servers (and even multiple replicas of volumes). A failure domain generally refers to a logical part of a system that can fail without affecting the rest of the system. When running on a bare metal system, the block storage system's failure domain generally corresponds to the entire bare metal computer system. By using virtualization to decouple failure domains from a per-server basis to a sub-server basis, we can take advantage of the underlying hardware redundancy used to support block storage server instances within the PSE. The number of fault domains can increase. As a result, block storage servers can individually manage small amounts of data when a failure occurs, reducing the workload to recover the data associated with the failure. These block storage instances can be created as virtual private clouds (VPCs), and instance clients that communicate with block storage volumes can also communicate in this same VPC. Beneficially, this allows VPC encryption to be leveraged to make communications in board extensions more secure.

However, virtualizing the block storage service with an extension on the provider board would allow the boot volume (also the block storage service machine image Specific technical challenges arise, including initializing the block storage service from the PSEs provide low-latency compute resources to on-premises facilities, but those resources experience increased latency when returning to the provider network. Although the PSE may rely on block storage servers within the domain of the provider network, such dependence suffers from increased latency and is therefore undesirable. As described in further detail below, embodiments of the present disclosure load the boot volume of a block storage server in the PSE's local storage from data stored in an area of the provider network, and then boot the block storage server. This issue is addressed by a local boot technique that allows the volume to be provided to other instances launched within the PSE. Thus, the disclosed local boot technology allows customers to use the block storage service machine image to launch instances in the substrate extension even when the block storage service itself does not yet reside in the substrate extension.

The disclosed systems and techniques also protect the provider network from potential security issues that can be enabled by connecting the PSE to the provider network. In some embodiments, a PSE may require a secure networking tunnel from the customer site where the PSE is installed to the provider network board (eg, the machine's physical network) in order to operate. These tunnels may include virtual infrastructure components hosted on both virtualized computing instances (eg, VMs) and substrates. Examples of tunnel components include containers running on VPC and proxy computing instances and/or computing instances. Each server in the PSE may use at least two tunnels, one for control plane traffic and one for data plane traffic. As described in more detail below, intermediate resources located along the network path between the provider network board and the PSE can securely manage traffic flowing between the board and the PSE.

In at least some embodiments, the provider network is a cloud provider network. A cloud provider network or "cloud" refers to a large pool of accessible virtualized computing resources, such as compute, storage and network resources, applications and services. The cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released upon customer command. These resources are dynamically provisioned and reconfigured to accommodate variable loads. Cloud computing therefore considers both applications delivered as services over publicly accessible networks (e.g., the Internet, cellular communication networks) and the hardware and software of the cloud provider data centers that provide those services. be able to.

A cloud provider network can be formed as a number of regions, where a region is a geographical area in which the cloud provider clusters its data centers. Each region may include two or more Availability Zones interconnected via a private high-speed network of fiber communications connections, for example. An Availability Zone refers to an isolated failure domain that includes one or more data center facilities with separate power sources, separate networks, and separate cooling from those in another Availability Zone. Preferably, Availability Zones within a region are located far enough from each other that the same natural disaster does not take more than one Availability Zone offline at the same time. A customer can connect to the cloud provider network's availability zone via a publicly accessible network (eg, the Internet, a cellular communication network). A PSE described herein can also connect to one or more Availability Zones via a publicly accessible network.

A cloud provider network may include a physical network called a substrate (eg, sheet metal boxes, cables). A cloud provider network may also include an overlay network of virtualized computing resources running on the substrate. Thus, network packets can be routed along the substrate network according to the configuration of the overlay network (eg, VPC, security groups). A mapping service can coordinate the routing of these network packets. The mapping service is a geographically distributed lookup service that maps a combination of overlay IP and network identifier to a substrate IP so that distributed substrate computing devices can look up a packet's destination.

For purposes of explanation, each physical host may have an IP address on the substrate network. Hardware virtualization technology may allow multiple operating systems to run simultaneously on a host computer, for example, as the host's virtual machines. A host's hypervisor or virtual machine monitor allocates the host's hardware resources among the host's various virtual machines and monitors the execution of the virtual machines. Each virtual machine may be provided with one or more IP addresses of the overlay network, and the host's virtual machine monitor may be aware of the IP addresses of the host's virtual machines. Virtual machine monitors (and/or other devices or processes on the network board) use encapsulation protocol techniques to transfer network packets (e.g., client IP packets) between virtualized resources of different hosts within the cloud provider network. can be encapsulated and routed on the network board. Encapsulation protocol techniques can be used in the network board to route encapsulated packets between network board endpoints over overlay network paths or routes. Encapsulation protocol technology can be viewed as providing a virtual network topology overlaid on a network substrate. The encapsulation protocol technology may include a mapping service that maintains a mapping directory that maps IP overlay addresses (public IP addresses) to substrate IP addresses (private IP addresses) for routing packets between endpoints. can be accessed by various processes across the cloud provider network.

As those skilled in the art will appreciate in light of this disclosure, certain embodiments may be capable of achieving various advantages, including some or all of (a)-(c) below. (a) where it uses a provider-managed infrastructure (e.g., PSE) at a site selected by the customer while still maintaining the scalability, security, availability, and other operational advantages enabled by the provider network; Allows customers of the provider network operator to deploy a variety of applications in an agnostic manner; (b) applications that need to be transported over long distances, such as through links between customer data centers and provider network data centers; (c) latency of applications where potentially large amounts of data can be consumed as input or produced as output by moving the application closer to the data source/destination; Overall responsiveness is improved and/or (d) security of sensitive application data is improved.

FIG. 1 is a block diagram illustrating an exemplary provider network extended by provider board extensions located within the network external to the provider network, in accordance with at least some embodiments. Within provider network 100 , customers can create one or more isolated virtual networks 102 . Customers can launch compute instances 101 within the IVN to run their applications. These compute instances 101 are hosted by board addressable devices (SADs) that are part of the provider network board (not shown). Similarly, a SAD that is part of the provider network board can host control plane services 104 . Exemplary control plane services 104 are instance management services (sometimes referred to as hardware virtualization services), object It includes object storage services that provide storage, block storage services that provide the ability to attach block storage devices to instances, database services that provide various database types, and so on.

Note that the components shown within provider network 100 can be treated as logical components. As mentioned above, these components are hosted by the SAD of the provider network board (not shown). For example, a provider network board can host instances 101 using containers or virtual machines running within an isolated virtual network (IVN). Such containers or virtual machines are executed by SAD. As another example, a provider network board can host one or more control plane services 104 using SAD in a bare metal configuration (eg, no virtualization). In at least some embodiments, the SAD is software (e.g., a server) executed by hardware that is addressable via a network address of the provider network rather than another network (e.g., customer network, IVN, etc.) point to In at least some embodiments, SAD can also refer to the underlying hardware (eg, computer system) that executes software.

As shown, provider network 100 communicates with provider substrate extension (PSE) 188 deployed within customer network 185 and PSE 198 deployed within customer network 195 . Each PSE includes one or more board addressable devices (SADs), such as SADs 189A-189N shown within PSE 188. FIG. Such SADs 189 facilitate provisioning of computing-related resources within the PSE. As in SAD 189A-189N, solid box-ellipse-dashed box combination illustrations of components are generally used to indicate that one or more of those components may be present in this and subsequent figures. used (whereas the corresponding textual references refer to the singular or plural form of the component, with or without the letter suffix). Customer gateway/router 186 provides connectivity between provider network 100 and PSE 188, as well as connectivity between PSE 188 and other customer resources 187 (e.g., other on-premise servers or on-premise services connected to customer network 185). offer. Similarly, customer gateway/router 196 provides connectivity between provider network 100 and PSE 198 as well as connectivity between PSE 198 and other customer resources 197 . Various connectivity options exist between provider network 100 and PSE 198, such as a public network such as the Internet as shown in PSE 188 or a direct connection as shown in PSE 198. FIG.

Within provider network 100, control plane traffic 106 is generally (but not always) directed to SADs, while data plane traffic 104 is generally (but not always) directed to instances. For example, some SADs may sell APIs that allow launching and terminating instances. The control plane service 104 can send a command to the API of such SAD via the control plane to launch a new instance on the IVN 102 .

An IVN, as suggested by its name, is a set of hosted (e.g., virtualized) resources that are logically separated or separated from other resources of a provider network (e.g., other IVNs). can contain. Control plane services can set up and configure IVNs, including assigning an identifier to each IVN to distinguish it from other IVNs. Provider networks can provide various ways to enable communication between IVNs by setting up peering relationships, etc. between IVNs (e.g., one IVN's gateway is configured to communicate with another ).

IVNs can be established for a variety of purposes. For example, an IVN can be set up for a particular customer by reserving a set of resources dedicated to the customer, with considerable flexibility regarding the network configuration of that set of resources offered to the customer. Within the IVN, the customer can set up subnets, assign desired private IP addresses to various resources, set up security rules governing incoming and outgoing traffic, and so on. In at least some embodiments, by default, a set of private network addresses set up within one IVN may be inaccessible from another IVN (or, more generally, from outside the IVN).

Tunneling technology facilitates traversal of IVN traffic between instances hosted by different SADs of provider network 100 . For example, a newly launched instance within IVN 102 may have IVN address A and be hosted by a SAD at board address X, while instance 101 may have IVN address B and be hosted by a SAD at board address Y. To facilitate communication between these compute instances, SAD X forwards packets sent from the newly launched instance to instance 101 (from IVN address A to IVN address B) to (from board address X to board address Y d) Encapsulate in the payload of a packet with the address information of the SAD that hosts each instance. Packets sent between SADs may also include an identifier for IVN 102 to indicate that the data is directed to IVN 102 as opposed to another IVN hosted by the SAD at board address Y. In some embodiments, the SAD also encrypts packets sent between instances within the payload of packets sent between SADs using the encryption key associated with the IVN. In at least some embodiments, encapsulation and encryption are performed by the software component of the SAD hosting instance.

In the case of PSEs, provider network 100 includes one or more networking components, thereby effectively extending provider network substrates outside of provider network 100 to PSEs connected to the customer's on-premises network. Such components can ensure that data plane and control plane operations targeting a PSE are communicated to the PSE securely, reliably, and transparently. In the illustrated embodiment, PSE interface 108 , PSESAD proxy 110 , and PSESAD anchor 112 facilitate data plane and control plane communications between provider network 100 and PSE 188 . Similarly, PSE interface 118 , PSESAD proxy 120 , and PSESAD anchor 122 facilitate data plane and control plane communications between provider network 100 and PSE 198 . As described herein, the PSE interface receives control plane traffic and data plane traffic from the provider network, sends such control plane traffic to the PSES AD proxy, and sends such data plane traffic to the PSE. The PSE interface also receives data plane traffic from the PSE and directs such data plane traffic to the appropriate destination within the provider network. The PSESAD Proxy receives control plane traffic from the PSE interface and sends such control plane traffic to the PSESAD Anchor. The PSESAD Anchor receives control plane traffic from the PSESAD Proxy and sends such control plane traffic to the PSE. The PSESAD Anchor also receives control plane traffic from the PSE and sends such control plane traffic to the PSESAD Proxy. The PSESAD Proxy also receives control plane traffic from the PSESAD Anchor and directs such control plane traffic to the appropriate destination within the provider network. Other embodiments may employ different combinations or configurations of network components to facilitate communication between the provider network 100 and the PSE (e.g., the PSE interface, the PSESAD Proxy functionality, and/or the PSESAD Anchor may Both PSE Interface and PSESAD Proxy, both PSESAD Proxy and PSESAD Anchor, may be combined in various ways by applications performing the operations of all three components, etc.).

As indicated above, each PSE has one or more substrate network addresses for SADs (eg, SAD189A-189N). Since their board addresses are not directly reachable via the provider network 100, the PSE interfaces 108, 118 masquerade with an attached virtual network address (VNA) that matches each PSE's board address. As shown, PSE interface 108 has attached VNA(s) 150 that match the SAD address(es) of PSE 188, and PSE interface 118 attaches VNA(s) 150 that match the SAD address(es) of PSE 198. 152 is attached. For example, traffic destined for a SAD with Internet Protocol (IP) address 192.168.0.10 in PSE 188 is sent to PSE interface 108 with attached virtual address 192.168.0.10 and Traffic destined for the SAD with IP address 192.168.1.10 is sent to PSE interface 118 with an attached virtual address of 192.168.1.10. Note that IPv4 or IPv6 addressing can be used. In at least some embodiments, a VNA is a logical construct that allows various network-related attributes, such as IP addresses, to be programmatically transferred between instances. Such transfers may be referred to as "attaching" the VNA to the instance and "detaching" the VNA from the instance.

At a high level, the PSE interface is essentially a packet forwarding component that routes traffic based on whether it is control plane traffic or data plane traffic. Note that given the board addressing and encapsulation techniques described above, both control and data plane traffic are directed to the SAD and are therefore both routed to the PSE interface. For control plane traffic, the PSE interface routes traffic to the PSES AD proxy based on the SAD address. For data plane traffic, the PSE interface may be one or more encrypted data plane traffic tunnels between provider network 100 and PSE (e.g., tunnel 191 between PSE interface 108 and PSE 188, PSE interface 118 and It establishes a tunnel 193) to and from PSE 198 and acts as an endpoint. For data plane traffic received from provider network 100, the PSE interface encrypts the traffic for transmission over the tunnel to the PSE. For data plane traffic received from a PSE, the PSE interface decrypts the traffic, optionally verifies the SAD addressing of the packets, and sends the traffic over the provider network 100 to the identified SAD destination. Note that if a PSE interface receives traffic from a PSE that does not conform to the expected format (e.g., protocol) used to transmit data plane traffic, the PSE interface can drop such traffic. Additionally, the PSE interface verifies the addressing of encapsulated packets so that the originator of the traffic (e.g., an instance hosted by the PSE within a particular IVN) is addressed to the destination (e.g., the same or a different IVN). Note that you can ensure that traffic is allowed to be sent to instances hosted by the provider network within the

Each SAD of a PSE has a corresponding group of one or more PSE interfaces, and each member of the group establishes one or more tunnels for data plane traffic with the PSE. For example, if a PSE with 4 SADs has 4 PSE interfaces, then each PSE interface establishes a secure tunnel (eg, 16 tunnels) with each data plane traffic endpoint of the SAD. Alternatively, a group of PSE interfaces can be shared by multiple SADs by attaching an associated VNA to each member of the group.

Each PSE has one or more PSESAD proxies and one or more PSESAD anchors that handle control plane traffic between the provider network 100 and the PSE's SADs. Control plane traffic generally has a command-response format or a request-response format. For example, a control plane service of provider network 100 can issue a command to PSESAD to start an instance. Control plane commands sent over secure tunnels should generally not originate from the PSE, as management of PSE resources is facilitated from the provider network. At a high level, the PSESAD proxy acts as a stateful security boundary between the provider network 100 and the PSE (such boundary is sometimes referred to as a data diode). To do so, the PSESAD proxy may employ one or more techniques such as applying various security policies or rules to the received control plane traffic. Other control plane services 104 indirectly or directly provide public APIs for PSE-hosted instances to issue commands to the provider network 100 via non-tunneled communications (eg, over a public network such as the Internet). Note that it may be possible to

For traffic originating within the provider network 100 and destined for the PSE, the PSES AD proxy can provide the control plane endpoint API of its corresponding SAD within the PSE. For example, a PSESAD proxy of a PSESAD that can host instances can provide APIs that match those that can receive control plane operations to launch, configure, and terminate instances. Depending on the API calls and associated parameters directed to the PSESAD and received by the PSESAD proxy, the PSESAD proxy can perform various operations. For some operations, the PSESAD proxy can pass the operation parameters and related parameters to the destination SAD without modification. In some embodiments, PSESAD proxies can verify that the parameters of API calls received from within provider network 100 are well-formed against the API before terminating their operation.

For some API calls or related parameters, PSESAD can act as an intermediary device to prevent sensitive information from being sent outside provider network 100 . Exemplary confidential information includes cryptographic information such as cryptographic keys, network certificates, and the like. For example, a PSESAD proxy can decrypt data using a confidential key and re-encrypt data using a key that can be exposed to the PSE. As another example, the PSESAD proxy terminates a first secure session originating from within provider network 100 (e.g., a Transport Layer Security (TLS) session) and uses a different certificate to establish a connection with the corresponding SAD. A new secure session can be created to prevent the provider network certificate from leaking to the PSE. Thus, the PSESAD proxy can receive certain API calls from within the provider network 100 containing sensitive information and issue alternative or replacement API calls to the PSESAD exchanging sensitive information.

For traffic originating from a PSE and destined for the provider network 100, the PSESAD proxy, for example, handles all control plane commands or requests originating from the PSE or directed to exposed control plane endpoints within the provider network. Only commands or requests that are not specified can be dropped.

In some embodiments, the PSESAD proxy can process responses to control plane operations depending on the nature of the expected response, if any. For example, for some responses, the PSESAD proxy may simply drop the response without sending any message to the originator of the corresponding command or request. As another example, for some responses, the PSESAD proxy may sanitize the response and send the corresponding command or request before sending the sanitized response to the originator of the corresponding command or request via control plane traffic 107 . Or you can ensure that the expected response format of the request is followed. As yet another example, the PSESAD proxy may generate a response (immediately or upon receipt of the actual response from the SAD) and send the generated response to the originator of the corresponding command or request via control plane traffic 107. can be sent using

As part of acting as a security boundary between the provider network 100 and the PSE, the PSESAD proxy can track the state of communication between the provider network's components (eg, control plane services 104) and each SAD of the PSE. State data may include session keys for the duration of the session, pending outbound API calls with associated sources and destinations to track outstanding responses, API calls received from within the provider network 100 and exchanged or It may contain the relationship with that API call issued to the SAD along with the replaced sensitive information and so on.

In some embodiments, the PSESAD proxy can provide stateful communications for network communications from other PSEs to providers in addition to control plane traffic. Such communications may include Domain Name System (DNS) traffic, Network Time Protocol (NTP) traffic, and operating system activation traffic (eg, for Windows activation).

In some embodiments, only certain components of the PSE are capable of functioning as endpoints of encrypted control plane traffic tunnels with provider network 100 . In order to provide redundancy and reliability to the connection between the provider network 100 and the PSE, a PSESAD anchor can serve as the provider network side endpoint of each of the PSE's available tunnel endpoints. As shown, PSESAD anchor(s) 112 serve to tunnel control plane traffic to PSE 188 via tunnel 190 and PSESAD anchor(s) 122 tunnel control plane traffic to PSE 1198 via tunnel 192 . work to

Various embodiments both use the techniques for handling traffic described above and isolate those network components that are exposed to traffic from the rest of the provider network 100. can limit the radial impact of any planned attack originating from outside the provider network (eg, from where the PSE is involved). Specifically, a network component can operate within one or more IVNs to limit the scope of an attacker's intrusion, thereby protecting the operations of both the provider network and other customers. Accordingly, various embodiments can instantiate the PSE Interface, PSESAD Proxy, and PSESAD Anchor as applications executed by virtual machines or containers running within one or more IVNs. In the illustrated embodiment, groups of PSE interfaces of different PSEs run in a multi-tenant IVN (eg, PSE interface IVN 132 of PSE 188 and PSE 198). In other embodiments, each group of PSE interfaces can be launched with a single-tenant IVN. In addition, each group of PSESAD proxies and each group of PSESAD anchors for a given PSE is a single-tenant IVN (e.g., PSE 188's PSESAD proxy IVN 134, PSE 188's PSESAD anchor IVN 136, PSE 198's PSESAD proxy IVN 138, and PSE 198's PSESAD proxy IVN 40). ).

The redundancy provided by running multiple instances of each of the network components (e.g., PSE Interface, PSESAD Proxy, and PSESAD Anchor) allows the provider network to: Note that it is possible to periodically recycle the instances that host those components. Recycling may include, for example, restarting an instance or launching a new instance and, for example, reconfiguring other instances using the recycled instance's address. Periodic recycling limits the time window in which an attacker can take advantage of a compromised network component if the network component is compromised.

PSE connection manager 180 manages the setup and configuration of network components that provide connectivity between provider network 100 and PSEs. As previously mentioned, the PSE interfaces 108, 118, PSESAD proxies 110, 120, and PSESAD anchors 112, 122 can be hosted as instances by provider network boards. The PSE connection manager 180 is responsible for connecting the PSE interface(s), PSESAD proxy/proxies when the PSE is shipped to the customer and/or when the PSE comes online and exchanges configuration data with the provider network. , and activation of the PSE's PSES AD anchor(s). In addition, the PSE connection manager 180 can further configure PSE interface(s), PSESAD proxy/proxies, and PSESAD anchor(s). For example, the PSE attach manager 180 attaches the VNA(s) corresponding to the PSE's SAD to the PSE interface(s) and provides the PSE interface(s) with the addresses of the PSESAD's proxy/proxies. The proxy/proxies of the PSESAD can be provided with the address of the PSESAD anchor(s) of the PSE's PSESAD. In addition, the PSE Connection Manager 180 may include, for example, various interfaces to enable communication between the PSE Interface IVN 132 and the PSE's PSESAD Proxy IVN and between the PSESAD Proxy IVN and the PSE's PSESAD Anchor IVN. You can configure the IVN for the component.

To facilitate the establishment of tunnels 190-193, tunnel endpoints are identified from outside each network (e.g., from outside the provider network for PSE interfaces and PSESAD anchors, from outside the customer network for PSE tunnel endpoints). ) may have one or more attached VNAs or assigned physical network addresses that can receive traffic. For example, PSE 188 may have a single outgoing network address and use port address translation (PAT) or multiple outgoing network addresses to manage communications to multiple SADs. Each of the PSESAD anchors 112, 122 can have an outgoing network address or can be shared (e.g., via PAT), and each of the PSE interfaces 108, 118 can have an outgoing accessible network address. can be obtained or shared (eg, via PAT).

FIG. 2 is a block diagram illustrating an exemplary provider board extension, according to at least some embodiments. In the illustrated embodiment, PSE 188 includes one or more PSE Frameworks 202 and one or more Hosts 220 . At a high level, each of the hosts 220 functionally (and possibly structurally) with at least a portion of a computer system forming part of a provider network substrate (e.g., its substrate resources hosting instances within the provider network). ), while the PSE framework(s) 202 are used to emulate provider network substrates within the PSE, as well as control plane traffic tunnels and data plane traffic tunnels (e.g., tunnels 190-193 of FIG. 1). ) to provide connectivity to provider networks.

In at least some embodiments, each of PSE frameworks 202 directs control plane traffic or data plane traffic to host 220 in a mesh-like architecture, as illustrated by PSE control plane traffic 240 and PSE data plane traffic 242 . , and vice versa. Such redundancy allows for the level of reliability that customers can expect from provider networks.

PSE framework 202 includes one or more control plane tunnel endpoints 204 that terminate encrypted tunnels (eg, tunnel 190, tunnel 192) that carry control plane traffic. In some embodiments, provider network 100 hosts a PSESAD anchor for each of control plane tunnel endpoints 204 . Back in the provider network, the PSESAD's proxy or proxies (e.g., proxy 110) can distribute control plane traffic to PSESAD anchors (e.g., anchors 112) and distribute the control plane traffic load to the PSE framework of PSE 188. 202 are effectively distributed. PSE framework 202 further includes one or more data plane tunnel endpoints 206 that terminate encrypted tunnels (e.g., tunnel 191, tunnel 193) that carry data plane traffic from the provider network's PSE interface and PSE frames. Works 202 may also be connected in a mesh-like architecture (eg, a given PSE interface 108 establishes a tunnel with each data plane tunnel endpoint 206 of PSE framework 202).

As indicated above, packets of control plane traffic and packets of data plane traffic may contain SADs as both sources and destinations. The latter (packets of data plane traffic) are encapsulated in packets with SAD-based addressing. As shown, PSE framework 202 is SAD 289 and host 220 is SAD 290 . The SADs in PSE 188 (e.g., SADs 289, 290) also provide secure session termination (e.g., Note that TLS termination) can be provided.

A SAD sells one or more control plane APIs to handle control plane operations directed to the SAD that manages the SAD's resources. For example, PSE manager 210 of PSE framework 202 may sell control plane APIs for managing components of PSE framework 202 . One such component is control plane traffic and data plane traffic, such as, for example, control plane traffic directed to SAD 289 for PSE manager 210 and control or data plane traffic directed to SAD 290 for host manager 222. / or PSE gateway 208 that routes data plane traffic to and from PSE 188 . PSE gateway 208 may also facilitate communication with customer networks, such as to or from other customer resources 187 accessible via a network of PSE deployment sites (eg, customer network 185).

The API of PSE Manager 210 may include one or more commands for configuring PSE Gateway 208 of PSE Framework 202 . Other components 212 of PSE framework 202 may include various applications or services involved in operating the PSE infrastructure for host 220, such as DNS, Dynamic Host Configuration Protocol (DHCP), and/or NTP services. .

Host manager 222 may sell control plane APIs for managing the components of host 220 . In the illustrated embodiment, host manager 222 includes instance manager 224 and network manager 226 . Instance manager 224 may handle API calls related to managing host 220 , including commands to launch, configure, and/or terminate instances hosted by host 220 . For example, an instance management service within a provider network (not shown) can issue a control plane command to instance manager 224 to launch an instance on host 220 . As shown, host 220 hosts customer instance 232 running inside customer IVN 233 , third party (3P) instance 234 running inside 3PIVN 235 , and service instance 236 running inside service IVN 237 . Note that each of these IVNs 233, 234, 235 can extend an existing IVN established within the provider network. Customer instance 232 may run some customer applications or workloads, and 3P instance 234 may run another party's applications or workloads that the customer is allowing to launch instances within PSE 188. , service instance 236 may run a provider network's services (eg, block storage services, database services, etc.) that it provides locally to PSE 188 .

Network manager 226 can process SAD-addressed data plane traffic received by host 220 . For such traffic, the network manager can perform any necessary decapsulation of the IVN packet before sending it to the addressed hosted instance. In addition, network manager 226 can handle routing of traffic sent by hosted instances. When a hosted instance attempts to send traffic to another locally hosted instance (eg, on the same host), network manager 226 can forward the traffic to the addressed instance. When a hosted instance attempts to send traffic to a non-local instance (e.g., not on the same host), network manager 226 finds the board address of the device hosting the non-local instance, encapsulates the corresponding packet, Optionally, encrypting into a SAD-addressed packet can be sent over the data plane (e.g., either to another host in the PSE, or back to the provider network via the PSE gateway 208). done). Note that the network manager 226 may contain or have access to various data that facilitates routing of data plane traffic (e.g., it may indicate the destination of packets received from hosted instances). to find the address of the SAD hosting the instance with the IVN network address).

FIG. 3 is a block diagram illustrating exemplary connections between provider networks and provider board extensions, according to at least some embodiments. Specifically, FIG. 3 shows an exemplary connection between the provider network and the PSE. Note that with respect to FIG. 3 and as shown at the top of the figure, the term “inbound” refers to traffic received from the PSE by the provider network and the term “outbound” refers to traffic sent to the PSE by the provider network. sea bream. Although not shown, this example assumes that the PSE includes two PSE frameworks 202 and two hosts 220 (four total) of SADs. The PSE framework provides tunnel endpoints 204A, 204B for control plane traffic and tunnel endpoints 206A, 206B for data plane traffic. Outbound traffic is decrypted and sent through PSE gateways 208A, 208B to destinations within the PSE board.

For each of the four SADs, the provider network includes a VNA, one or more PSE interfaces, and one or more PSESAD proxies. In this example, the provider network includes a PSESADVNA 304, two PSE interfaces 108A, 108B, and two PSESAD proxies 110A, 110B for a given PSESAD. Together, the PSE interface(s) and PSESAD proxies/proxies may be referred to as slices as shown, with each slice corresponding to a particular SAD within the PSE. In other embodiments, the PSE interface(s) may be shared by all VNAs for VPN rather than a single VNA for one of the SADs.

The PSESADVNA 304 acts as a front for a given PSE from which other components of the provider network can send traffic to and receive traffic from the PSE's corresponding SAD. A load balancer (not shown) can route outbound traffic sent to the PSESADVNA 304 to one of the PSE interfaces 108A, 108B. The illustrated PSE interfaces 108A, 108B for a given slice and those of other slices (not shown) operate within the PSE interface IVN 132 . The PSE interfaces 108A, 108B transmit control plane traffic to the PSEs by transmitting the data plane traffic to the PSEs over data plane traffic tunnels and forwarding the control plane traffic to the PSES AD proxies 110A, 110B of the slices. The PSE interfaces 108A, 108B provide the network addresses of the PSESAD's proxy/proxies of the associated SAD, the network addresses of the data plane tunnel endpoint(s), and communications with those endpoint(s). Stores (or has access to) one or more keys or associated keys for the PSE's data plane tunnel endpoint(s) to protect.

In at least some embodiments, PSE interfaces 110A, 110B establish secure tunnels for data plane traffic with data plane tunnel endpoints 206A, 206B, respectively, resulting in N data plane tunnels, where , where N is the number of PSE interfaces per SAD (assuming each SAD has the same number of interfaces) multiplied by the number of data plane tunnel endpoints multiplied by the number of SADs. In this example, 16 data plane tunnels are established between PSE interfaces and data plane tunnel endpoints (ie, 2 PSE interfaces x 2 data plane tunnel endpoints x 4 SADs per SAD).

The PSESAD proxies 110A, 110B receive control plane traffic from the PSE interfaces 108A, 108B, perform various operations described elsewhere herein, and pass through either of the two PSESAD anchors 112A, 112B. to send control plane traffic to the PSE. Similarly, the PSESAD proxies 110A, 110B receive control plane traffic from either of the two PSESAD anchors 112A, 112B and perform various operations described elsewhere herein to provide Control plane traffic 107 to the destination. The illustrated PSESAD proxies 110 A, 110 B for a given slice and proxies for other slices (not shown) operate within PSESAD proxy IVN 134 . The PSE interfaces 108A, 108B store (or have access to) the network address of the PSESAD anchor(s).

In at least some embodiments, PSESAD proxies can access shared data store 306 or otherwise exchange information. Such information exchange can be used for several reasons. For example, remember that a PSESAD proxy can sell an API interface to emulate the API interface of the associated SAD in the PSE. One PSESAD proxy may be routed by different PSESAD proxies because some communications may be stateful and various load balancing techniques may prevent the same PSESAD proxy from processing all communications for a given working set. It may be necessary to access previously processed communication state (eg, PSESAD proxy 110A sends control plane operations to the PSE and PSESAD proxy 110B receives responses to control plane operations from the PSE). For inbound control plane traffic, the PSESAD proxy checks if the inbound message matches the expected state, and if so, controls the control plane traffic as described elsewhere herein. Messages can be sent via traffic 107 . If not, the PSESAD proxy 110A, 110B can drop the traffic. As another example, recall that the PSESAD proxy can bridge a separate secure session (eg, a TLS session) so that provider network credentials are not sent to the PSE. Again, the PSESAD proxy handling the outbound message may be different from the PSESAD proxy handling the response to that message, so the PSESAD proxy handling the response message sends the secure response message through the control plane traffic 107. The same key established between the originator of the outbound message and the PSESAD proxy that processed the outbound message can be used to send to the originator.

In this example, each PSE framework provides a single control plane tunnel endpoint 204 . For each available control plane tunnel endpoint 204, the provider network includes a PSE anchor. In this example, the provider network includes two PSE anchors 112A, 112B. PSESAD Anchors 112A, 112B operate within PSESAD Anchor IVN 136 . PSE Anchor 112 receives control plane traffic from each of the eight PSES AD proxies (two per slice on each of the four SADs) and sends the traffic to the PSE. The PSE Anchor also receives control plane traffic from the PSE and sends the traffic from the PSE to one of the two PSES AD proxies associated with the SAD from which the traffic originated. The PSE Anchors 112A, 112B provide the network addresses of the proxy/proxies of the SAD's respective PSES ADs, the network addresses of the control plane tunnel endpoint(s), and communications with those endpoint(s). Stores (or has access to) one or more keys or associated keys for the PSE's control plane tunnel endpoint(s) to protect.

In at least some embodiments, network components or provider networks can employ load balancing techniques to distribute routing control and data plane traffic workloads between provider networks and PSEs. For example, traffic sent to PSESADVNA 304 can be distributed between PSE interfaces 108A, 108B. As another example, each of the PSE interfaces 108 can distribute traffic between the data plane tunnel endpoints 206A, 206B. As yet another example, each of the PSE interfaces 108 can distribute traffic between the PSES AD proxies 110A, 110B. As yet another example, each of PSESAD proxies 110 can distribute outbound traffic among PSESAD anchors 112A, 112B. As yet another example, PSESAD Anchor 112 can distribute inbound traffic among PSESAD Proxies 110A, 110B. In either case, such load balancing may be performed by a transmitting entity or load balancer (not shown). An exemplary load balancing technique employs a load balancer with a single VNA that distributes traffic to multiple components "behind" that address, and assigns multiple receiver addresses to each data sender. Including providing and distributing selected recipients at the application level.

1-3 show the establishment of separate tunnels for control plane traffic and data plane traffic, other embodiments show one or more tunnels for both control plane traffic and data plane traffic. Note that a tunnel of . For example, the PSE interface can route data plane traffic to the PSESAD anchor for transmission to the PSE over the shared tunnel, bypassing additional operations performed by the PSESAD proxy on control plane traffic.

4 and 5 are block diagrams illustrating examples of virtualized block storage systems, according to at least some embodiments. As shown in FIG. 4, provider network 400 or PSE 488 includes host 420A. Host 420 A is the host for block storage server instance 450 . In hosting the block storage server instance 450, the host 420A uses some of the host storage device(s) 421 as storage devices 447 (e.g., virtual drives or virtual disks) provisioned for the block storage server instance 450. You are provisioning a quantity. A block storage server instance 450 can partition provisioned storage devices 447 and create and host volumes on behalf of other instances. A provider network 400 instance management service (not shown), via instance manager 426 , can initiate the launch of such block storage server instances on either provider network 400 hosts or PSE 488 hosts. Instance manager 426 is a virtual machine manager (VMM), hypervisor, or other manager that can provision host resources (e.g., compute, memory, network, storage) for instances (e.g., virtual machines) and launch and terminate instances. It can be a virtualization program.

Host 420A is also the host of instance A 430 within instance IVN 432 . Block storage server instance 450 provides block storage volume A 434 to instance A 430 and block storage volume B 440 to another instance (not shown). The host 420A includes a block storage client 460A through which the hosted instance accesses block storage volumes hosted by the block storage server (eg, by providing a virtual block storage interface to the instance). Specifically, block storage client 460 intercepts or otherwise receives block storage operations issued to volumes “attached” to instances (eg, block storage volume A 434 attached to instance A 430). It can be a software program that

As shown in FIG. 5, provider network 400 or PSE 488 includes two hosts 420A and 420B. Again, host 420A is the host for block storage server instance 450 . Host 420B is the host of instance B 436 within instance IVN 438 . Block storage server instance 450 provides block storage volume B 440 to instance B 436 . The host 420B includes a block storage client 460B whose hosted instance accesses block storage volumes hosted by the block storage server.

When a volume is attached to an instance, the block storage client obtains a mapping that associates the volume with the block storage server instance(s) that host the replica. A block storage client can open a communication session (eg, via a network-based data transfer protocol) with each of the block storage server instances. Upon receiving a block storage operation from an instance, the block storage client issues the operation to the appropriate block storage server(s) to implement the operation for the instance. For example, referring to FIG. 4, instance A 430 can issue a read block of data at a given block address (eg, logical block address or LBA) from volume A 434 . Block storage client 460A receives the read operation and issues the operation to block storage server instance 450, including the requested block address, based on the mapping data of instance attached volume A 434 to block storage server instance 450. Upon receiving data from block storage server instance 450, block storage client 460A provides the data to instance A 430. A similar series of operations can be performed between instance B 436, block storage client 460B, block storage server instance 450, and volume B 440 of FIG.

4 and 5, block storage messaging between block storage clients and block storage server instances can be sent using a network-based data transfer protocol such as Global Network Block Device (GNDB). Such messages can be encapsulated as data plane traffic and routed between block storage clients 460 and block storage servers 450 by each host's network manager 424, as described elsewhere herein. . In some embodiments where the block storage server instance is hosted by the same host as the instance that issues block storage operations, the block storage client can issue related operations to the network manager via a network-based data transfer protocol, and then , the network manager can internally route operations to hosted block storage server instances. In other embodiments where the block storage server instance is hosted by the same host as the instance that issues block storage operations, the block storage client is a PCIe or other interconnect that resides between the block storage client and the hosted instance. Related operations can be issued over the connection, thereby avoiding the protocol packing associated with network-based transfers.

4 and 5, the system can deploy various levels of encryption to protect block storage data. To provide end-to-end encryption of block storage traffic, Block Storage Server Instances 452 run within Block Storage Service (BSS) IVN 452 and include any number of Block Storage Server Instances to provide PSEs from provider networks. can reach. Block storage client 460 has VNA 462 associated with block storage service IVN 452 . Thus, IVN level encryption using block storage service IVN encryption key 484 can be used to encrypt and decrypt traffic sent between block storage server instance 450 and block storage client 460 . Note that encryption and decryption of traffic carried over the block storage service IVN can be done by the network manager at the IVN's endpoints. For example, network manager 424B encrypts traffic sent from block storage client 460B over VNA 462B. As another example, network manager 424A decrypts traffic sent to instances hosted within IVN 452 of the BSS, including block storage server instance 450 . Block storage volumes may also have volume-level encryption to provide encryption at rest for data. For example, volume A 434 can be encrypted with volume A encryption key 480 and volume B 440 can be encrypted with volume B encryption key 482 . Using encryption key 480, block storage client 460A encrypts data written to volume A 434 by instance A 430 and decrypts data read from volume A 434 by instance A 430. Using encryption key 482, block storage client 460B can perform similar encryption and decryption operations between instance B 436 and volume B 440. Some embodiments prevent the block storage server instance that hosts a volume from being hosted on the same host as the instance attached to that volume, and prevent the volume encryption key from being hosted on the same host as the volume it encrypts. Note that the policy to prevent being hosted can be in-place (eg, volume A encryption key 480 and volume A 434 are no longer on host 420A).

In some embodiments, the instance and host manager are executed by different processor(s), as shown in FIGS. For example, block storage server instance 450 can be executed by a first processor (eg, processor 1710) and host manager 422A can be executed by a second processor (eg, processor 1775). The processor(s) running a block storage server instance access the host's storage device(s) via one or more interconnects. For example, processor(s) running block storage server instance 450 (eg, processor 1710) access storage device(s) 421 (eg, storage device 1780) via a PCIe interconnect between the two. can. As another example, the processor(s) running block storage server instance 450 may connect the storage device(s) to the processor or system-on-chip running host manager 422A (e.g., processor 1775) via a first PCIe bus. 421, which in turn bridges their communication to the storage device(s) 421 over a second PCIe bus.

Although shown as a single volume in FIGS. 4 and 5, each instance volume 434, 440 is mirrored across multiple replicas stored on multiple different block storage servers to provide high reliability to the system. obtain. For example, one replica may be referred to as the primary replica that handles reads from and writes to volumes (input and output operations, or "input/output (I/O)") and hosts that primary replica. can synchronously propagate writes to other secondary replicas. If the primary replica fails, one of the secondary replicas is selected to act as the primary replica (referred to as failover). Replicas can be subdivided (eg, striped) into partitions, with each partition of a given replica stored on a different server to facilitate parallel reads and writes. Replicas can also be encoded with redundancy (e.g., parity bits) so that when a block storage server becomes unavailable (e.g., due to server or disk failure, etc.), lost data is replaced with available data. can recover. As a result, volumes attached to an instance can be hosted by many block storage servers.

One challenge associated with hosting block storage servers in a virtualized context is booting those servers. How does a block storage server instance boot from a boot volume hosted by a block storage server instance if there is no operational block storage server instance? This is especially difficult in the context of PSEs, which may not include dedicated (or bare metal) block storage servers. Figures 6 and 7 illustrate examples of local boot techniques for booting a first or "seed" block storage server instance (a block storage server instance from which other instances can be started) within a PSE. Once started, other instances can be started from the seed block storage server instance.

FIG. 6 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using the first technique, according to at least some embodiments. At a high level, FIG. 6 illustrates an example of remotely booting a block storage server 650 within instance 690 via a proxy server hosted within provider network 600 . Block storage services (BSS) 606 of provider network 600 provide users with the ability to create block storage volumes and attach those volumes to instances. For example, a user (e.g., a customer of the provider network, another service of the provider network) submits commands via an API or console relayed to the BSS 606 to create, resize, or create volumes managed by the BSS. can be deleted and those volumes can be attached or detached from the instance. BSS 606 maintains a BSS object store 608 containing volume snapshots. A snapshot, like a file, can be considered a type of object that can include object data and metadata about the object (eg, creation time, etc.). A volume snapshot is a copy (or backup) of a volume at a given point in time, allowing re-creation of that volume on a physical or virtual drive. A snapshot can generally be divided into a boot volume and a non-boot volume, the former (boot volume) facilitating the booting of software running within an instance. A machine image may be a group of one or more snapshots of a given instance, including boot volumes.

In some embodiments, BSS 606 applies object-level encryption of objects in BSS object store 608 . To prevent leakage of encryption keys used within the provider network 600, the BSS 606 can also manage a PSE object store 610 and use different ciphers than those used to encrypt objects in the BSS object store 608. The key is used to re-encrypt objects sent to a given PSE. In this example, block storage server instance machine image 612A is used to boot block storage server 650 on PSE's host 620 . As shown in circle A, the BSS 606 decrypts the block storage server instance machine image 612A using a key held within the provider network 600 and sends it to the block storage server using a key 613 that can be sent to the PSE. Re-encrypt as instance machine image 612B.

An IVN can provide security for traffic between block storage server instances and block storage clients. In some embodiments, BSS 606 manages its IVN, shown as BSS IVN 652 . The host 620 on which the block storage server 650 runs contains a host manager 622 having a block storage client 660 with an associated VNA 662 of the BSS's IVN 652 . To facilitate communication between the block storage client 660 and the provider network 600 through the BSS's IVN 652 , the BSS 606 launches a proxy server instance 642 within the BSS's IVN 652 . Proxy server instance 642 may provide a block storage-like interface to block storage client 660, allowing client 660 to access block storage server instance machine image 612B. BSS 606 launches proxy server instance 642 as follows.

As shown in circle B, BSS 606 first sends a message to bare metal block storage server 614 to create a new logical volume to serve as the boot volume for proxy server instance 642 . The bare metal block storage server 614 creates a new logical volume 619 on its one or more storage devices 618 on the bare metal server 614 and loads the proxy server machine image 616 stored in the BSS object store 608 into the volume 619 .

As indicated by circle C, BSS 606 initiates activation of proxy server instance 642 using instance management service 640 . Instance management service 640 is another control plane service of provider network 600 that facilitates launching instances within provider network 600 or PSEs. In some embodiments, the instance management service 640 tracks host utilization metrics (i.e. metrics such as CPU utilization, memory utilization, and network utilization) that indicate how "hot" the host is. can or can access it. For example, instance management service 640 or another service may periodically poll hosts to obtain utilization metrics. By accessing metrics associated with pools of potential hosts (subject to some constraints on host location within a given PSE, etc.), Instance Management Service 640 identifies low utilization (e.g., thresholds or thresholds). is below the threshold) to identify the host hosting the instance in response to the launch request. In requesting activation of proxy server instance 642 , BSS 606 specifies that the instance should be activated from volume 619 within the BSS's IVN 652 . As shown in circle D, instance management service 640 identifies host 646 that hosts the instance and sends a message to host 646 to launch the instance. An instance manager (not shown) of host 646 provisions the host's resources for proxy server instance 642 and boots it from boot volume 619 .

BSS 606 also initiates booting of block storage server 650 via instance management service 640 . BSS 606 specifies that an instance should boot from a boot volume made available by proxy server instance 642 within the BSS's IVN 652 of a particular PSE (eg, using a PSE identifier). As shown in circle E, instance management service 640 identifies host 620 hosting block storage server instance 650 and sends a message to host 620 (eg, via a control plane traffic tunnel) to restore the instance. to start. As shown, a request to launch an instance is received by an instance manager 626, which can provision the instance's host resources and launch and terminate the instance. Here, instance manager 626 creates instance 690 to host the block storage server.

As part of configuring the instance 690, the instance manager 626 configures the basic input/output system (BIOS) 692 of the instance 690 and, via the proxy server instance 642, places the block storage server instance machine image 612B on the virtual boot volume 694. can be loaded. BIOS 692 contains block storage device drivers (eg, non-volatile memory express (NVMe) drivers) and can attach two block storage devices. A virtualized block storage device is presented by proxy server instance 642 through block storage client 660 and boot volume 694 . Although shown within instance 690 , boot volume 694 is allocated to instance 690 by instance manager 626 from a storage device (not shown) of host 620 accessible through block storage client 660 or another component of host manager 622 . Note that it matches the volume of the virtual drive you created. Other embodiments may include a non-BIOS interface (eg, Unified Extensible Firmware Interface (UEFI)) for connecting instance 690 to two block devices.

During execution, BIOS 692 (or other firmware such as UEFI) can load boot volume 694 from block storage server instance machine image 612B. When block storage server instance machine image 612B is encrypted using key 613, block storage client 660 decrypts block storage server instance machine image 612B during the loading process. Once the load operation is complete, the BIOS continues the boot process to boot instance 690 from boot volume D94 containing block storage server software. In some embodiments, BIOS 692 can detach the block storage device corresponding to proxy server instance 642 before booting block storage server software 650 .

Although shown and described using proxy server instance 642, other embodiments use a block storage server instance (not shown) launched from block storage server instance machine image 612A to provide block storage Devices can be provided to instances 690 via block storage clients 660 . The BSS 606 creates a volume on the virtual drive or virtual disk of the block storage server instance to be launched and loads the machine image 612A of the block storage server instance into the volume. During the load process, the block storage server instance decrypts the machine image using the provider network key, optionally using a volume-specific key (e.g., how key 480 encrypts volume B 440, etc.) encrypted and provided by BSS 606 to block storage client 660 .

FIG. 7 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using a second technology, according to at least some embodiments. At a high level, FIG. 7 illustrates an example of remotely booting instance 790 to execute block storage server software 750 via pre-boot software executed by instance 790 prior to executing block storage server software 750. show. Specifically, storage device 721 of PSE 720 is preloaded with pre-boot instance boot volume 723 (eg, prior to shipment of PSE). The pre-boot instance boot volume 723 contains software that allows an instance to boot during the block storage server pre-boot phase 798 to load block storage server software into another boot volume. The instance can then reboot and boot from the other boot volume to launch the block storage server instance during the block storage server boot phase 799 .

As shown in circle A, BSS 606 initiates the launch of a block storage server instance using instance management services 640, specifying that the instance should launch within the IVN of the BSS of a particular PSE. Again, the instance management service 640 identifies the host 620 of the PSE that hosts the block storage server instance 650 and sends a message to the host's host manager 722 (eg, via the control plane traffic tunnel) to launch the instance. to send. A request to launch an instance is received by an instance manager (not shown) of host manager 722 .

As shown in circle B, host manager 722 (or instance manager) provisions host resources for instance 790 running block storage server 750 . In addition, the host manager 722 is presented to the pre-boot instance boot volume 723 and the virtual boot volume 794 (such as the boot volume 694 described above) by two attached block storage devices (block storage clients (not shown)). block storage device interface) into the BIOS 792A (or UEFI) and boot from the boot volume 723 of the pre-boot instance.

BIOS 792A then enables booting of preboot software 796 from preboot instance boot volume 723, as indicated by circle C. FIG. If pre-boot instance boot volume 723 is based on a generic machine image, host manager 722 also configures instance 790 to facilitate communication between pre-boot software 796 and provider network 700, as shown in circle D. Configuration may need to be updated. Specifically, host manager 722 attaches a VNA to instance 790 , provides pre-boot software 796 with credentials to use when accessing PSE object store 610 , and stores machine images loaded into boot volume 794 . Can provide a key to decrypt. The VNA configuration, credentials, and keys may have been passed to host manager 722 as part of the request from instance management services 640 to launch the instance.

As shown in circle E, once instance 790 is configured and running pre-boot software 796, pre-boot software 796 copies block storage server instance machine image 612B from PSE object store 610 and loads it into boot volume 794. do. As part of loading boot volume 794, pre-boot software 796 can decrypt block storage server instance machine image 612B with key 613, assuming it was encrypted. Host manager 722 can detect that boot volume 794 has completed loading, or pre-boot software can signal its loading completion. For example, pre-boot software can initiate an instance reboot when boot volume 794 is loaded. As another example, network manager 724 can detect session termination between instance 790 and PSE object store 610 (eg, detect when a TCP session is closed).

As indicated by circle F, host manager 722 can update or otherwise reconfigure BIOS 792A to 792B either during data transfer or once data transfer is complete. Such reconfiguration includes changing the boot volume from pre-boot instance boot volume 723 to boot volume 794 and removing the block storage device containing pre-boot instance boot volume 723 . Prior to rebooting instance 790 , host manager 722 may also clear memory provisioned to instance 790 , which may contain residual data written by pre-boot software 796 . Upon rebooting instance 790 , BIOS 792 B boots block storage server 750 from boot volume 794 .

6 and 7 show exemplary techniques for booting a block storage server instance within a PSE, other techniques are possible. For example, an instance manager can create a volume on a PSE's storage device as part of the provisioning resources of a launching instance. Before launching an instance, an instance manager can send a message to a control plane service (eg, block storage service or instance management service) to request a bootable snapshot to load into a volume. A control plane service can send a bootable snapshot to the instance manager over the control plane traffic tunnel. Once loaded, the instance manager allows instance software to be launched.

FIG. 8 is a block diagram illustrating an exemplary system for booting additional compute instances of provider board extensions from a block storage server, according to at least some embodiments. For example, once a block storage server instance is launched using techniques such as those described with reference to FIGS. ) can provide boot volumes. This is particularly useful in the context of PSEs where seed block storage server instances are launched from machine images that are transferred or hosted over relatively slow connections between the provider network and the PSE compared to the PSE's substrate interconnect. obtain. In this example, instance 890 is launched on PSE 888 host 820A. Instance 890 is running block storage server software 850 and operates within IVN 844 of the BSS. In addition, block storage clients 860A and 860B have attached VNAs 862A and 862B that enable communication on the IVN 844 of the BSS.

As shown in circle A, instance management service 640 of provider network 800 can request that BSS 606 create a new volume from a specified snapshot. As shown in circle B, BSS 606 can command block storage server instance 890 to create a volume based on the specified snapshot. In some embodiments, instance management service 640 may provide a volume-specific encryption key (eg, key 680) to encrypt the volume.

As indicated by circle C, a block storage server instance 890 can create a volume by fetching a specified snapshot from an object store 810 such as PSE object store 610 . Although not shown, in other embodiments object store 810 may be an object store hosted by a BSS cache instance of PSE 888 for caching volume snapshots and/or machine images. BSS 606 may manage BSS cache instances, and a customer may specify to BSS 606 that a particular snapshot be loaded into cache prior to any request to activate an instance. In this way, block storage server instance 890 can create volumes from cached boot volume snapshots and PSE's by avoiding the delays associated with transferring data from provider network 800 object stores to block storage server instance 890 . Dramatically reduces instance boot time.

As shown in circle C, instance management service 640 of provider network 600 issues commands to hosts of PSE 888 to attach or detach volumes hosted by block storage server instance 890 to other instances hosted by PSE 888. can. For example, volume A 832 may be loaded with a bootable snapshot. Instance management service 640 can instruct host 820A to launch instance A 830 using the volume identifier associated with volume A 832 . In response, block storage client 860A can attach volume A 832 hosted by block storage server instance 890 to instance A 830 via BSS's IVN 844, thereby allowing instance A 830 to boot from volume A 832. become. As another example, Volume B 836 may be loaded with a non-bootable snapshot containing other data. Instance management service 640 can instruct host 820 B to attach volume B 836 to hosted instance 834 . In response, block storage client 860B can attach volume B 836 hosted by block storage server instance 890 to instance B 834 via IVN 844 of the BSS.

FIG. 9 is a block diagram illustrating an exemplary system for managing virtualized block storage servers, according to at least some embodiments. Running block storage servers within a virtualized environment provides several benefits, including allowing instance management services to automatically scale the number of running block storage server instances as needed. The Instance Management Service monitors the utilization of host resources (e.g., CPU, memory, network, etc.), rather than the BSS running on a defined pool of manually scaled bare metal servers, and the number of running block storage server instances. number can be adjusted automatically. For example, if two hosts of block storage server instances are reporting high resource utilization, the instance management service can launch additional block storage server instances on hosts with low resource utilization. The block storage service can then create volumes on newly launched instances, thereby avoiding increased workload and possible performance degradation of running block storage server instances.

Additionally, running a block storage server as an instance allows the number of fault domains to exceed the number of host computer systems by decoupling fault domains from the overall computer system. As the number of failure domains increases, more block storage servers run on a fixed set of hardware, and as the number of block storage servers increases, data managed by any one particular server increases. reduces the overall footprint of For example, assume that the system includes nine host computer systems that are not subdivided into smaller failure domains (ie, one computer system has one failure domain). To avoid data loss, each host or fault domain runs a single block storage server. If those nine block storage servers host 90 terabytes (TB) of data (including encoding that enables data recovery) across the instance's volume, each block storage server would store approximately 10TB of data. . If one of the hosts fails, it will need to recover its 10TB (eg, from another 80TB). Such data recovery has a cost of computing, transferring and storing data. If each of those 9 hosts is subdivided into 2 failure domains and the number of block storage servers is increased to 18, each block storage server stores about 5TB of data and 1 component of the failure domain fails. approximately halves the amount of data (and corresponding cost) that would need to be recovered.

In this example, PSE 988 or provider network 900 includes three hosts 920A, 920B, and 920C. Depending on the hardware design of the host, the host can either be treated as a single fault domain or subdivided into two or more fault domains. Here, hosts 920 each contain two failure domains. Host 920A includes fault domain 922 and fault domain 924 so that components in one fault domain can continue to operate even if components in another fault domain fail. For example, fault domain 922 corresponds to a first processor in a multiprocessor system connected to a first bank of memory (e.g., RAM) and using a first set of one or more storage drives (e.g., SSDs). While it may, failure domain 924 may correspond to a second processor in a system connected to a second memory bank and using a second set of one or more storage drives. Note that some components may be shared across fault domains, such as power supplies, but again, this is subject to redundancy in the hardware design and how fault domains are overlaid on the hardware. sea bream.

Host 920A is running block storage server instance 950A in failure domain 922 and block storage server instance 950B in failure domain 924 . Host 920B is running block storage server instance 950C in failure domain 926 and block storage server instance 950F in failure domain 928 . Volume A includes a primary replica 660 provided by block storage server instance 950A and a secondary replica 662 provided by block storage server instance 950B. Volume B includes a primary replica 664 provided by block storage server instance 950C and a secondary replica 666 provided by block storage server instance 950A. Volume C includes a primary replica 668 provided by block storage server instance 950B and a secondary replica 670 provided by block storage server instance 950C. Although shown as having two replicas per volume, in practice each volume may have fewer or more replicas, with each replica having many different blocks (e.g., by striping). It can be split between storage server instances.

As shown in Circle A, Instance Management Service 940 monitors utilization of host physical resources, such as processor utilization, memory utilization, storage utilization, and network utilization, which are aggregated across hosts. may be isolated by fault domain, or may be per resource (e.g., instance management service 940 may specify that the average CPU utilization is 50%, that the CPU utilization of processor(s) in fault domain 922 is 50%, or a metric may be received from host 920A indicating that the CPU utilization of a particular processor supporting fault domain 922 is 50%). The instance management service 940 can track the host's resource utilization in a database (not shown). Note that in addition to block storage server instances, the indicated fault domains can host other instances (eg, customer instances) that contribute to the utilization of the host's physical resources. The Instance Management Service 940 periodically updates the BSS 906 with resource utilization metrics so that the BSS 906 selects less utilized block storage server instances when creating new volumes, as shown in circle B. can enable

As indicated by circle C, the instance management service 940 can launch new block storage server instances when the resource utilization of the host supporting the block storage server instances exceeds one or more thresholds. The threshold can be some combination of resources across hosts (e.g., storage usage rate is greater than 80% and processor utilization is greater than 50%), or defined in various ways such as on an individual resource and/or host basis. To avoid starting a block storage server instance on a fault domain that already hosts block storage server instances, the instance management service 940 determines which fault domain is occupied (e.g., in a database containing resource usage metrics). and you can track which ones are available. At this point, instance management service 940 has exceeded a certain threshold of resource utilization of hosts 920A, 920B, and/or 920C, causing new It was determined that block storage server instances 950D and 950E were being started. Failure domain 928 includes block storage server 950F that is not hosting any volumes, but the resource utilization of other instances hosted within that failure domain or the total resource utilization of host 920B may be affected by another block storage server instance. Get too high to support.

As shown in circle D, instance management service 940 can update BSS 906 to provide updated identification information for operational block storage instances 950 (here, including block storage servers 950D and 950E). Based on the resource utilization metrics received from the identified block storage instance 950 and instance management service 940, the BSS 906 can create a new volume on host 920C, as shown in circle E. Now, assuming that host 920C has shown low resource utilization as reported by instance management service 940, BSS 906 determines primary replica 670 provided by block storage server instance 950D and block storage server instance create a new volume D containing the secondary replica 672 provided by 950E.

FIG. 10 is a block diagram illustrating an exemplary system for providing volume mappings to block storage clients, according to at least some embodiments. As alluded to above, a single volume may be associated with multiple replicas, such as a primary replica and several secondary replicas. Each replica may be distributed across several block storage servers. The association between a volume, its replicas, and the servers that host those replicas (or parts of replicas) prevents data from being transferred between servers in the event of a hardware failure or as part of background load balancing operations. As you migrate, you can change over time. When deploying block storage servers in PSE, high data availability (e.g. primary replica becomes secondary replica in case of failure), high durability (e.g. process to recreate lost data in case of server failure) ), it is preferable to be able to change their association even when PSEs are disconnected from the provider network or the provider network is unreachable. .

As shown, the PSE 1088 includes components that track how volumes are distributed (or mapped) across block storage server instances so that when a block storage client needs to access a volume, a block A storage client can find the block storage server that hosts the volume. The exemplary volume A mapping data 1098 includes several items for each entry. The items include the server identifier (eg, a unique identifier associated with the instance or host hardware), the server address (eg, IP address), and the volume type of volume A (eg, primary or secondary). Mapping data may include different or additional items such as block identifiers (eg, if replicas are split or striped across multiple block storage server instances). Exemplary volume A mapping data 1098 shows that volume A has a primary replica 1052 provided by block storage server instance 1050A and two secondary replicas 1054 and 1056 provided by block storage server instances 950B and 1050C, respectively. indicates that it contains Note that PSE 1088 can and possibly hosts many other block storage server instances (not shown).

To reliably store volume mapping data 1098 , distributed data store 1064 can store volume mapping data 1098 . In some embodiments, each distributed data store corresponds to a cluster of nodes that separately maintain the state of volume mapping(s). Each node in the cluster exchanges messages with other nodes in the cluster to update its state based on the state seen by other cluster nodes. One of the nodes of the cluster may be designated as the leader or primary node to which changes to volume mappings are proposed. Nodes of a cluster may implement a consensus protocol, such as the Paxos protocol, to propose and agree on changes to volume mapping data for a given volume. A cluster may track volume mapping data for one or more volumes. As shown, cluster 1066 tracks volume mapping data 1098 for volume A, while other clusters 1067 track volume mapping data for other volumes.

In some embodiments, the nodes of the cluster are instances run by hosts of the provider network. Such instances persist individual views of the volume mapping data to the host's non-volatile storage (eg, via the block storage client to volumes hosted by the block storage server instance). In other embodiments, the nodes of the cluster are part of block storage server software executed by block storage server instances. As shown in exemplary node software environment 1090 , nodes can run as containers 1082 hosted within container engine processes included in block storage server software 1080 . Such nodes can persist views of volume mapping data directly to volumes provided by block storage server software. Preferably, the nodes of the cluster are hosted by separate instances or within separate failure domains.

Like any other software run by the PSE's host, the node is subject to hardware failures. If so, the remaining nodes (or the provider network's block storage service) will detect node loss, create new nodes, replace the lost nodes, and use the other nodes' consensus view of the volume mapping data. to update the volume mapping data for the new node. As a result, not only can the volume mapping data be changed, but the ID of the instance hosting the nodes of the cluster can also be changed. Cluster mapping data can be used to track clusters. Exemplary cluster mapping data 1099 includes several items such as node identifier (eg, VOL_A_NODE1), node address (eg, IP address), and node type (eg, leader node). In this example, the cluster is formed with 5 nodes.

Cluster mapping data can be determined and maintained by cluster discovery service 1062 . As shown in circle A, the cluster discovery service 1062 can monitor the node locations of clusters of various volumes hosted by block storage server instances within the PSE. The cluster discovery service 1062 can monitor node locations in a variety of ways. For example, in embodiments where nodes run in environment 1090, cluster discovery service 1062 can periodically poll all block storage server instances 1050 hosted by PSE 1088 to obtain the identity of any resident nodes. . As another example, the PSE 1088 host's network manager may be configured to route special broadcast messages to any of the hosted cluster nodes (e.g., directly or indirectly hosted by a block storage server instance, etc.). be done). Cluster Discovery Service 1062 can periodically broadcast queries to obtain the identity of any of the hosted cluster nodes.

In some embodiments, Cluster Discovery Service 1062 is an instance hosted by one of PSE's 1088 hosts. Such an instance may have a VNA with a reserved IP address in the BSS's IVN 1052 so that it can be reached even if it needs to change hosts due to a hardware failure. In other embodiments, the cluster discovery service 1062 can be integrated into the PSE's DNS service. For example, a volume cluster can be associated with a domain, and a DNS service can resolve name resolution requests for that name to the IP addresses of one or more nodes of the cluster.

As shown in circle B, an instance management service 1040 can send a message to a particular host's block storage client 1060 to attach a volume to a hosted instance (not shown). For example, instance management service 1040 can send a message to the host that includes the instance identifier and volume identifier for Volume A. As shown in circle C, block storage client 1060 can query cluster discovery service 1062 to obtain the identity of one or more nodes of volume A's cluster 1066 . In some embodiments, block storage client 1060 can cache cluster mapping data in cluster mapping data cache 1066 . In some embodiments, cluster discovery service 1062 may be omitted, and block storage client 1060 queries block storage server instances of PSE 1088 (eg, via the broadcast mechanism described above) to determine cluster 1066 of Volume A. Note that we configure it to identify the node.

As indicated by circle D, block storage client 1060 obtains a current view of volume mapping data for volume A from cluster 1066, and based on the volume mapping data, assigns volume A to host volume A, as indicated by circle E. can be connected to a block storage server 1050 that Although not shown, in some embodiments, upon receiving a connection request from a client, the block storage server sends a message indicating whether the block storage server still hosts the volume (or at least part of the volume). can be sent to the volume cluster. Despite receiving a connection request from a block storage client, the block storage server cannot host the volume for various reasons. For example, recent changes to the set of servers hosting the volume may not have yet propagated to or through the volume cluster, or the block storage client sending the connection request may have a stale cache. May rely on volume mapping data. A block storage server that receives a connection request from a client can obtain the identity of one or more nodes of the volume from the cluster discovery service 1062, regardless of whether the block storage server hosts the volume. If a block storage server no longer hosts a volume, the block storage server can suggest updates to the datastores maintained by the cluster to remove the block storage server from the volume mapping data. Additionally, the block storage server may send a response to the block storage client that initiated the request, indicating that the connection attempt failed, and optionally indicating that the volume is no longer hosted by the server. If the block storage server is still hosting the volume, the block storage server can send an acknowledgment to the cluster to indicate to the cluster that at least that portion of the mapping data is still valid.

FIG. 11 is a block diagram illustrating an exemplary system for tracking volume mappings, according to at least some embodiments. As explained above, connectivity between the PSE and the provider network cannot be guaranteed. To meet desired levels of data availability and data durability, the PSE includes facilities that allow changing the mapping between a given volume and the block storage server instance that hosts that volume. As described with reference to Figure 10, the volume of a given volume can be tracked using a cluster that implements a distributed data store. FIG. 11 illustrates a stepwise approach to volume placement, or the process of selecting block storage server instances to host replicas or portions of replicas of volumes. Specifically, the BSS Volume Placement Service 1107 of the BSS 1106 makes initial placement decisions and associated volume mappings at volume creation time, and the PSE Volume Placement Service 1108 manages subsequent changes to volume mapping over the life of the volume. do. PSE volume placement service 1108 may be implemented in various ways, such as by an instance hosted by PSE 1188 integrated as a component of the PSE framework (eg, PSE framework 202).

As shown in circle A, PSE volume placement service 1108 monitors the status of block storage server instances hosted by PSE 1188 . For example, the PSE volume placement service 1108 can periodically poll the hosted block storage server instance to check for its response and/or (eg, as described with reference to FIG. 9) ) resource usage of the hosted block storage server instance, its failure domains, and/or metrics related to the host. As shown, the PSE Volume Placement Service 1108 can send the collected server status to the BSS Volume Placement Service 1107 as shown in circle B. Note that in other embodiments, the BSS Volume Placement Service 1107 may obtain metrics related to resource utilization from Instance Management Services (as described with reference to FIG. 9).

Upon receiving a request to create a new block storage volume for an instance hosted by PSE 1188 , BSS 1106 can request volume placement recommendations from BSS placement service 1107 . Depending on the profile of the new volume (eg, number of replicas, whether replicas as stripes, etc.), the BSS placement service 1107 can provide identification of recommended block storage server instances. In this example, BSS placement service 1107 recommends that block storage server instances 1150A and 1150B host new volume A with two replicas. Adopting the recommended block storage server instance, BSS 1106 creates a new cluster 1166 to track volume mapping data for volume A as shown in circle C. The mapping data first identifies block storage server instance 1150A as hosting the primary replica block and storage server instance 1150B as hosting the secondary replica. In addition, the BSS 1106 sends one or more messages to the identified block storage server(s) to indicate the storage volume (e.g., the storage volume hosted by block storage server instance 1150A and the block storage server(s), as shown in circle D). Create a storage volume hosted by instance 1150B). Storage volumes can be backed by the underlying host storage device's capacity provisioned to each block storage server instance. A storage volume created in block storage server instance 1150A can host a primary replica 1152 of volume A, and a storage volume created in block storage server instance 1150B can host a secondary replica 1154A of volume A. In some embodiments, newly created storage volumes can be loaded from volume snapshots or machine images as described with reference to FIG. In this example, an instance (not shown) attached to volume A performs block storage operations for some time to propagate writes from primary replica 1152 to secondary replica 1154B (as shown, for example, in circle E). block storage server instance 1150A communicates with block storage server instance 1150B.

At some point, block storage server instance 1150B may experience a problem, as indicated by circle F. For example, block storage server instance 1150B may lag or become unresponsive (eg, due to memory leaks, hardware failures, etc.). Problem detection can occur in a variety of ways. In some embodiments, block storage server instance 1150A detects problems such as failures to acknowledge propagated writes. As such, block storage server instance 1150A may include a policy that includes one or more actions to take in response to the detected problem. For example, block storage server instance 1150A may wait for a number of consecutive propagated writes to go unacknowledged, or wait for a period of time. At that point, block storage server instance 1150 A can request a replacement block storage server instance for secondary replica 1154 from PSE volume placement service 1108 . In other embodiments, the PSE volume placement service 1108 detects problems (eg, based on collected metrics or responsiveness) during monitoring as described above with reference to Circle A. Again, the PSE volume placement service 1108 may include policies that include one or more actions to take in response to detected problems, including initiating a replacement of the block storage server instance of the secondary replica 1154. . Regardless of the detector, PSE volume placement service 1108 provides block storage server instance 1150A with the identity of the replacement block storage server instance.

In this example, the PSE volume placement service 1108 identifies block storage server instance 1150C to block storage server instance 1150A, as indicated by circle G. A message is sent to block storage server instance 1150C to create a storage volume to back up the replica data being relocated. For example, identifying the block storage server instance 1150C allows the PSE volume placement service 1108 to send a message to the block storage server instance 1150C to create the storage volume. As another example, block storage server instance 1150A can send a message to create a storage volume upon receiving identification information from PSE volume placement service 1108 . Once created, block storage server instance 1150A can begin mirroring replica 1152 to block storage server instance 1150C as replica 1154B, as shown in circle H. Note that if block storage server instance 1150B is responsive (but, for example, exhibits poor performance), the mirroring operation can be performed by copying replica 1154A to replica 1154B. Re-mirroring from either replica 1154 or replica 1152 is feasible in this scenario as those replicas are not distributed across storage servers, but in other embodiments the various storage servers where the replicas are distributed. to recreate or otherwise generate lost data by using redundancy, etc. (e.g., parity bits, error correction codes, etc.) encoded in the stored data . For example, if replica data is encoded and distributed across ten block storage server instances and one of the ten is lost, reading the data associated with the replica from the remaining nine block storage server instances can recreate the lost data. As another example, if 10 different block storage servers host different replicas of a volume using the same distribution pattern, the lost data is transferred from the block storage server instance hosting the corresponding portion of the other replicas. can be copied.

As shown in circle I, block storage server instance 1150A submits a request to cluster 1166 to update the volume mapping for volume A, causing block storage server instance 1150B to become block storage server instance 1150C as the host of the secondary replica. can be exchanged for Block storage server instance 1150A may submit a request to cluster 1166 when it initiates or completes a remirror to block storage server instance 1150C. In other embodiments, another entity (such as PSE volume placement service 1108 or block storage server instance 1150C) may submit a request to cluster 1166 to update the volume mapping of volume A.

FIG. 12 is a flow diagram illustrating operations of a method for launching a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes or variations and/or combinations thereof described herein) may be performed under the control of one or more computer systems made up of executable instructions. , hardware or a combination thereof, implemented as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of the extension of the provider network. An extension of a provider network includes one or more physical computing devices or systems located remotely from the provider network's data center (eg, outside the data center network), such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data center. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems of the provider network. In some embodiments, one or more (or all) of the operations are performed by components of hosts in other figures (eg, host 420A).

At block 1205, operations include receiving, by a computer system of an extension of the provider network, a first request to launch a first virtual machine to host a block storage server application, the extension of the provider network. communicates with the provider network through at least a third party network. When providing a block storage device to a PSE-hosted instance, the virtualization provided by the PSE's host can be used to host the block storage server. For example, the provider network's block storage service can initiate the launch of a block storage server virtual machine by the instance management service, which in turn initiates the launch via a secure communication channel between the provider network and the PSE. A request can be issued to the host manager of the selected host of the PSE.

The operations further include provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as provisioned storage devices at block 1210 . As part of starting a virtual machine, the host system's host manager can allocate or provision a portion of the host system's computing resources to the virtual machine. Such resources may include, for example, the storage capacity of the host system's storage device (eg, SSD), memory capacity (eg, RAM), processors or processor cores, and the like.

At block 1215, operations further include executing the block storage server application on the first virtual machine. As part of executing the block storage server application, at block 1220, the operation also creates a logical volume on the provisioned storage device in response to a second request from the block storage service of the provider network, Including creating volumes. For example, a provider network's block storage service can send one or more messages to create a volume (using provisioned storage capacity) that can be attached to other instances so that the instance can use the block storage interface You can access the volume via

As part of executing the block storage server application, at block 1225 operations are further performed by receiving a third request to perform an I/O operation on the logical volume and, at block 1230, on the requested logical volume. and performing input/output operations. For example, referring to FIG. 4, instance A 430 can issue a command to read a block of data from a block address of a virtual block device attached to the instance and backed by block storage volume A 434. The block storage server instance 450 can receive the command (eg, via the BSS's IVN 452) and process it to block storage volume A 434. For example, referring to FIG. 5, Instance B 436 can issue a command to write a block of data to a block address of a virtual block device attached to the instance and backed by Block Storage Volume B 440. The block storage server instance 450 can receive the command (eg, via the BSS's IVN 452) and process it to block storage volume A 440.

FIG. 13 is a flow diagram illustrating operations of a method for using a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes or variations and/or combinations thereof described herein) may be performed under the control of one or more computer systems made up of executable instructions. , hardware or a combination thereof, implemented as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of the extension of the provider network. An extension of a provider network includes one or more physical computing devices or systems located remotely from the provider network's data center (eg, outside the data center network), such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data center. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems of the provider network. In some embodiments, one or more (or all) operations are performed by a host in another figure (eg, host 420A).

At block 1305, operations include executing, with the computer system, a first block storage server virtual machine to host a first volume using one or more storage devices of the computer system. As shown in FIG. 4, for example, host 420A hosts a virtual machine (ie, block storage server instance 450). At block 1310, operations further include executing, with the computer system, a second virtual machine that can access the virtual block storage device. Host 420A also hosts another virtual machine (ie, instance 430). At block 1315, operations further include executing a block storage client with the computer system. Host 420A includes block storage client 460A, which can facilitate attachment of block storage devices to hosted virtual machines. As part of running the block storage client, at block 1320, the operation further includes receiving from the second virtual machine a first block storage operation to perform on the virtual block storage device and, at block 1325, sending a message to the virtual block storage device. sending to the first block storage server virtual machine to cause the first block storage server virtual machine to perform a first block storage operation on the first volume. A virtual machine can issue block storage operations (eg, block read, write, burst read, write, etc.) that block attached storage devices through a block storage client. Block storage clients, for example, can relay their block storage operations across a network to block storage servers that host volumes containing block addresses.

FIG. 14 is a flow diagram illustrating operations of a method for managing virtualized block storage servers in a provider board extension, according to at least some embodiments. Some or all of the operations (or other processes or variations and/or combinations thereof described herein) may be performed under the control of one or more computer systems made up of executable instructions. , hardware or a combination thereof, implemented as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of the extension of the provider network. An extension of a provider network includes one or more physical computing devices or systems located remotely from the provider network's data center (eg, outside the data center network), such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data center. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems of the provider network. In some embodiments, one or more (or all) of the operations are performed by components of PSEs in other figures (eg, PSE 1088, PSE 1188).

At block 1405, operations receive a first request by a first block storage server instance to create a first storage volume to store a first portion of a first logical volume; At 1410, receiving a second request by a second block storage server instance to create a second storage volume to store a second portion of the first logical volume. As shown in FIG. 11, for example, initial placement of volumes (eg, replicas, striped replicas, etc.) can originate from the provider network's block storage service 1106 and be received by the PSE 1188 component. In the example shown in FIG. 11, volume A is initially stored using block storage server instances 1150A and 1150C on hosts of PSE 1188. In the example shown in FIG.

At block 1415, the operation further sends a third request to a third block storage server instance to create a third storage volume and store a second portion of the first logical volume. include. At some point, you may need to change the block storage server instance that hosts part of your volume. As described herein, the PSE volume placement service 1108 or one of the other block storage server instances hosting the volume sends a message to another block storage server instance to replace the instance that is being changed. can.

At block 1420, the operations further include storing, by the third block storage server instance, the second portion of the first logical volume to the third storage volume. At block 1425, the operation further updates the datastore containing identification information for each block storage server instance that hosts a portion of the first logical volume to remove identification information for the second block storage server instance. , adding the identification information of the third block storage server instance. To track migration of volumes across instances, PSE 1188 can host a datastore that maps volumes (including replicas, striping across servers, etc., if any). Such data stores may be clusters as described with reference to FIG.

FIG. 15 illustrates an exemplary provider network (or “service provider system”) environment, according to at least some embodiments. Provider network 1500 may provide resource virtualization to customers via one or more virtualization services 1510, which allow customers to use one or more provider networks in one or more data centers or It allows purchasing, renting, or acquiring instances 1512 of virtualized resources (including but not limited to computing and storage resources) implemented on devices in multiple provider networks. A local Internet Protocol (IP) address 1516 may be associated with resource instance 1512 . The local IP address is the internal network address of resource instance 1512 of provider network 1500 . In some embodiments, provider network 1500 also includes a public IP address 1514 and/or public IP address ranges (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6)) that customers may obtain from provider 1500 . ) address).

Traditionally, the provider network 1500 allows customers of the service provider (eg, customers operating one or more client networks 1550A-1550C including one or more customer device(s) 1552) to , may enable dynamically associating at least some public IP addresses 1514 assigned or allocated to a customer with particular resource instances 1512 assigned to the customer. The provider network 1500 also allows a customer to transfer a public IP address 1514 previously mapped to one virtualized computing resource instance 1512 assigned to the customer to another virtualized computing resource instance 1512 similarly assigned to the customer. It may be possible to remap to instance 1512 . Using virtualized computing resource instances 1512 and public IP addresses 1514 provided by the service provider, customers of the service provider, such as operators of customer network(s) 1550A-1550C, implement customer-specific applications, for example. , and may present the customer's application on an intermediate network 1540 such as the Internet. Other network entities 1520 in the intermediate network 1540 may then generate traffic to the destination public IP address 1514 published by the customer network(s) 1550A-1550C. The traffic is routed to the service provider data center, where it is routed through the network board to the local IP address 1516 of the virtualized computing resource instance 1512 currently mapped to the destination public IP address 1514 . Similarly, response traffic from virtualized computing resource instance 1512 may be routed through network board to intermediate network 1540 and back to source entity 1520 .

As used herein, a local IP address refers to the internal or "private" network address of a resource instance, eg, in a provider network. The local IP address may be in the address block reserved by Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and/or may be of the address format specified by IETF RFC 4193; May be changeable within the network. Network traffic originating outside the provider network is not routed directly to local IP addresses. Instead, traffic uses a public IP address that maps to the local IP address of the resource instance. The provider network may include network devices or appliances that provide network address translation (NAT) or similar functions to perform the mapping from public IP addresses to local IP addresses (or vice versa).

A public IP address is an Internet variable network address assigned to a resource instance by either a service provider or a customer. Traffic routed to a public IP address is translated by, for example, a 1:1 NAT and forwarded to the local IP address of each resource instance.

Some public IP addresses may be assigned to specific resource instances by the provider network infrastructure. These public IP addresses may be referred to as standard public IP addresses or simply standard IP addresses. In some embodiments, the mapping of standard IP addresses to resource instance local IP addresses is the default launch configuration for all resource instance types.

At least some public IP addresses may be assigned to or obtained by customers of provider network 1500 . The customer can then assign the assigned public IP address to the specific resource instance assigned to the customer. These public IP addresses may be referred to as customer public IP addresses or simply customer IP addresses. Instead of being assigned to resource instances by the provider network 1500 as is the case with standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, eg, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and can be remapped to other resource instances by each customer as needed or desired. A customer IP address is associated with the customer's account, rather than a specific resource instance, and controls that IP address until the customer chooses to release that IP address. Unlike traditional static IP addresses, customer IP addresses allow customers to mask resource instance or Availability Zone failures by remapping the customer's public IP address to one of the resource instances associated with the customer's account. make it possible to For example, a customer IP address can be remapped to a replacement resource instance to work around a customer resource instance or software problem.

FIG. 16 is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers, according to at least some embodiments. A hardware virtualization service 1620 provides a plurality of computing resources 1624 (eg, VMs) to customers. Computing resources 1624 may, for example, be rented or leased to customers of provider network 1600 (eg, customers implementing customer network 1650). Each of computing resources 1624 may be provided with one or more local IP addresses. Provider network 1600 may be configured to route packets from local IP addresses of computing resources 1624 to public Internet destinations and from public Internet sources to local IP addresses of computing resources 1624 .

Provider network 1600 connects, for example, customer network 1650 coupled to intermediate network 1640 via local network 1656 and virtual computing system 1692 via hardware virtualization service 1620 coupled to intermediate network 1640 and provider network 1600 . provide the ability to implement In some embodiments, hardware virtualization service 1620 may provide one or more APIs 1602, e.g., web service interfaces, through which customer network 1650, e.g., console 1694 (e.g., web-based applications, stand-alone applications, mobile applications, etc.) may access the functionality provided by the hardware virtualization service 1620. In some embodiments, in provider network 1600 , each virtual computing system 1692 in customer network 1650 may correspond to a computing resource 1624 that is leased, rented, or otherwise provided to customer network 1650 .

From an instance of virtual computing system 1692 and/or another customer device 1690 (e.g., via console 1694), the customer can access functionality of storage service 1610, e.g., via one or more APIs 1602, Access data from and store data in storage resources 1618A-1618N of virtual datastores 1616 (eg, folders or “buckets,” virtualized volumes, databases, etc.) provided by provider network 1600. obtain. In some embodiments, a virtualized data store gateway (not shown) may be provided to customer network 1650, which stores at least some data, e.g., frequently accessed or critical data, locally. , communicates with Storage Service 1610 over one or more of a number of communication channels, and may upload new or modified data from the local cache, resulting in a primary store of data (virtualized data store 1616) is maintained. In some embodiments, users can mount and mount volumes of virtual datastores 1616 via storage service 1610 acting as a storage virtualization service via virtual computing system 1692 and/or on another customer device 1690 . These volumes can be accessed and viewed as local (virtualized) storage 1698 to the user.

Although not shown in FIG. 16, virtualization service(s) may also be accessed from resource instances within provider network 1600 via API(s) 1602 . For example, a customer, appliance service provider, or other entity may access a virtualized service from within each virtual network in provider network 1600 via API 1602 to access one or more resources within the virtual network or within another virtual network. May request allocation of an instance.

Figure 17 is a block diagram illustrating an exemplary computing device that may be used with at least some embodiments. In at least some embodiments, such computer systems include control plane components and/or data plane components, various virtualization components (virtual machine, container, etc.) and/or a server that implements one or more of the SEDs. Such computer systems may include general-purpose or special-purpose computer systems that include or are configured to access one or more computer-accessible media. In at least some embodiments, such computer systems can also be used to implement provider boards and/or components outside of provider board extensions (e.g., customer gateways/routers 186, other customer resources 187, etc.). . In the illustrated embodiment of the computer system, computer system 1700 includes one or more processors 1710 coupled to system memory 1720 via input/output (I/O) interface 1730 . Computing device 1700 also includes a network interface 1740 coupled to I/O interface 1730 . FIG. 17 illustrates computer system 1700 as a single computing device, and in various embodiments computer system 1700 is one computing device or computer configured to operate together as single computer system 1700. It may contain any number of computing devices.

In various embodiments, computer system 1700 is a uniprocessor system including one processor 1710 or a multiprocessor system including several processors 1710 (eg, two, four, eight, or another suitable number). can be a system. Processor 1710 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2010 is a general purpose processor implementing any of a variety of instruction set architectures (ISAs) such as x86, ARM, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. or may be an embedded processor. In a multiprocessor system, each of processors 1710 may typically, but not necessarily, implement the same ISA.

System memory 1720 may store instructions and data accessible by processor(s) 1710 . In various embodiments, system memory 1720 is either random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash type memory, or any other type of memory, or the like. can be implemented using any suitable memory technology. In the illustrated embodiment, program instructions and data that implement one or more desired functions, such as those methods, techniques, and data described above, are stored in system memory 1720 as code 1725 and data 1726. is shown.

In one embodiment, I/O interface 1730 is configured to coordinate I/O traffic between any of the peripheral devices including processor 1710, system memory 1720, and network interface 1740 or other peripheral interface. obtain. In some embodiments, I/O interface 1730 performs any necessary protocol, timing, or other data conversion to transfer data signals from one component (eg, system memory 1720) to another component. into a form suitable for use by (eg, processor 1710). In some embodiments, I/O interface 1730 provides support for devices attached by various types of peripheral buses, such as, for example, variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. can include In some embodiments, the functionality of I/O interface 1730 may be split into two or more separate components, such as, for example, northbridge and southbridge. Also, in some embodiments, some or all of the functionality of I/O interface 1730 , such as an interface to system memory 1720 , may be incorporated directly into processor 1710 .

Network interface 1740 exchanges data between computer system 1700 and other devices 1760 attached to a network or networks 1750 (eg, other computer systems or devices such as those shown in FIG. 1). can be configured to allow In various embodiments, network interface 1740 may support communication over any suitable wired or wireless general data network, such as, for example, Ethernet network types. Additionally, network interface 1740 may be via a telecommunications/telephony network such as an analog voice network or a digital fiber communications network, via a storage area network (SAN) such as a Fiber Channel SAN, or any other suitable type of network. Communication may be supported via networks and/or protocols.

In some embodiments, computer system 1700 includes an I/O interface 1730 (eg, Peripheral Component Interconnect Express (PCI-E) standard or another such as QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). one or more offload cards 1770 (including one or more processors 1775, and possibly one or more network interfaces 1740) connected using a bus implementing a version of the interconnect. For example, in some embodiments, computer system 1700 may act as a host electronic device (e.g., operating as part of a hardware virtualization service) that hosts compute instances and one or more offload cards 1770 runs a virtualization manager that can manage compute instances running on the host electronic device. By way of example, in some embodiments, the offload card(s) 1770 may be used to suspend and/or unsuspend compute instances, launch and/or terminate compute instances, perform memory transfers/copies. Can execute instance management operations, etc. These management operations, in some embodiments, are executed by hypervisor computer system 1700 other processors 1710A-1710N in coordination with the hypervisor (eg, in response to a request from the hypervisor). may be performed by (s) 1770; However, in some embodiments, the virtualization manager implemented by the offload card(s) 1770 can serve requests from other entities (eg, from the compute instance itself) and any other It cannot cooperate with (or serve) a hypervisor. Referring to FIG. 2, in at least some embodiments, at least a portion of the functionality of PSE framework 202 and host manager 222 executes on one or more processors 1775 of offload card 1770, while instances (e.g., , 232 , 234 , 236 ) are executed on one or more processors 1710 .

In some embodiments, computer system 1700 includes one or more storage devices (SD) 1780 . Exemplary storage devices 1780 include solid state drives (eg, with various types of flash or other memory) and magnetic drives. Processor(s) 1710 may access SD 1780 via interface(s) 1730 or possibly via offload card(s) 1770 . For example, offload card(s) 1770 may include a system-on-chip (SoC) that includes multiple interconnect interfaces, bridging interface 1730 with an interface to SD 1780 (e.g., performing PCIe-to-PCIe bridging). ).

In some embodiments, system memory 1720 may be one embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and/or data may be received, transmitted, or stored on different types of computer-accessible media. Generally speaking, computer-accessible media refers to non-transitory storage media such as magnetic or optical media (e.g., disks or DVD/CDs) that are coupled to computing device 1700 through I/O interface 1730; A memory medium may be included. Non-transitory computer-accessible storage media may also be included in some embodiments of computing device 1700, as system memory 1720 or another type of memory such as RAM (e.g., SDRAM, double data rate (DDR), It can include any volatile or non-volatile media such as SDRAM, SRAM, etc.), read only memory (ROM), and the like. Additionally, computer-accessible media can include electrical, electromagnetic, or digital signals conveyed over a transmission medium or communication medium, such as a network and/or wireless link, such as can be implemented through network interface 1740. signal.

Various embodiments as described or suggested herein can be implemented in various operating environments, which in some cases include one or more user computers that can be used to run any number of applications; It may include a computing device or processing device. User or client devices include desktop or laptop computers that run standard operating systems, as well as mobile devices that run software for mobile phones and are capable of supporting several networking and messaging protocols; It may include any of a number of general purpose personal computers such as wireless devices and handheld devices. Such a system may also include several workstations running any of a variety of commercially available operating systems as well as other known applications for purposes such as development and database management. These devices may also include other electronic devices such as dummy terminals, thin clients, game consoles, and other devices capable of communicating over a network.

Most embodiments use Transmission Control Protocol/Internet Protocol (TCP/IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), utilizes at least one network that will be familiar to those skilled in the art to support communications using any of a variety of commercially available protocols such as AppleTalk. The network(s) may be, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network. , and any combination thereof.

In embodiments utilizing a web server, the web server may include HTTP servers, file transfer protocol (FTP) servers, common gateway interface (CGI) servers, data servers, Java servers, business application servers, and the like. can start either a simple server or a middle-tier application. The server(s) may also be in any programming language such as Java, C, C# or C++, or any scripting language such as Perl, Python or TCL, and combinations thereof. It may be possible to execute programs or scripts in response to requests from user devices by executing one or more web applications, etc., which may be implemented as one or more scripts or programs written in . The server(s) may also include database servers, including, but not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM®, and the like. Database servers can be relational or non-relational (eg, “NoSQL”), distributed or non-distributed, and the like.

The environment may include various data stores as described above as well as other memory and storage media. They may reside in various locations, such as storage media, local to (and/or residing on) one or more computers, or remote from any or all of the computers across a network. In a particular set of embodiments, information may reside in a storage area network (“SAN”) familiar to those skilled in the art. Similarly, any necessary files to perform functions residing on a computer, server, or other network device may be stored locally and/or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically coupled via a bus, the hardware elements being, for example, at least one central processing unit ("CPU ”), at least one input device (eg, mouse, keyboard, controller, touch screen, or keypad), and/or at least one output device (eg, display device, printer, or speakers). Such systems may also include one or more of disk drives, optical storage devices and solid state storage devices such as random access memory (RAM) or read only memory (ROM), as well as removable storage devices, memory cards, flash cards and the like. May include storage devices.

Such devices may also include computer-readable storage medium readers, communication devices (eg, modems, network cards (wireless or wired), infrared communication devices, etc.), and working memories as described above. A computer-readable storage medium reader is connectable to or configured to receive a computer-readable storage medium and includes remote, local, fixed and/or removable storage devices and computer-readable information. Represents a storage medium for permanently and/or more permanently containing, storing, transmitting, and retrieving. Also, the system and various devices generally include some software application, module, service, or other element located within at least one working memory device, and may be used by an operating system, such as a client application or web browser. and application programs. It should be appreciated that alternate embodiments may have many variations from those described above. For example, customized hardware may also be used and/or particular elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices, such as network input/output devices, may be employed.

Storage media and computer readable media for containing code or portions of code include, but are not limited to, storage media and communication media, such as volatile and nonvolatile media, removable and non-removable media, and are known in the art. It can include any suitable media known or used for storing and/or transmitting information such as computer readable instructions, data structures, program modules or other data. Any method or technology embodied in RAM, ROM, electrically erasable programmable read-only memory (“EEPROM”), flash memory, or other memory technology, compact disc read-only memory (“CD-ROM”), digital multimedia A disk (DVD), or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device, or any other device that can be used to store desired information and that can be accessed by a system device Including other media. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other means and/or methods for implementing the various embodiments.

Various embodiments are described in the foregoing description. For purposes of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Additionally, well-known features may be omitted or simplified so as not to obscure the described embodiments.

Herein, the option to use bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dashed lines, and dotted lines) to add additional characteristics to some embodiments to explain the operation of However, such notations should not be interpreted to mean that they are the only options or optional actions and/or that blocks with solid borders are not optional in particular embodiments.

Reference numbers with trailing letters (e.g., 101A, 102A, etc.) may be used to indicate that there may be more than one instance of the referenced entity in various embodiments; When present, each need not be identical, but instead may share some common characteristics or serve in a common way. Further, any particular suffix used is not meant to imply that any particular amount of the entity is present, unless specifically indicated to the contrary. Thus, two entities using the same or different trailing characters may or may not have the same number of instances in various embodiments.

References to "one embodiment," "an embodiment," "exemplary embodiment," and the like are used, although the described embodiments may include specific features, structures, or features, but all embodiments may include Indicates that a particular property, structure, or characteristic may not necessarily be included. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or feature is described in connection with an embodiment, such feature, structure, or feature is described in connection with other embodiments, whether explicitly stated or not. Influencing is considered to be within the knowledge of those skilled in the art.

Further, in the various embodiments described above, unless otherwise noted, disjunctive language such as the phrase "at least one of A, B, or C" refers to either A, B, or C, or It is intended to be understood to mean any combination thereof (eg, A, B, and/or C). Thus, such disjunctive language is not intended to imply that a given embodiment requires the presence of at least one of A, at least one of B, or at least one of C, respectively. Also should not be understood.

At least some embodiments of the disclosed technology can be described in view of the following examples.
1. A computer-implemented method comprising:
A block storage server virtual machine is executed by a first one or more processors of a first computer system of a provider network to store a first storage using one or more storage devices of said first computer system. hosting the volume;
running a customer virtual machine with access to a virtual block storage device by the first one or more processors;
executing a block storage client by a second one or more processors of the first computer system;
The block storage client
receiving from the customer virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the block storage server virtual machine to cause the block storage server virtual machine to perform the first block storage operation on a first storage volume;
The computer-implemented method.
2. 2. The computer-implemented method of clause 1, wherein the message is sent over the secure virtual network using a key to encrypt and decrypt traffic sent over the virtual network.
3. Clause 3. The computer-implemented method of clause 2, wherein a first virtual network address of the virtual network is associated with the block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client.
4. A computer-implemented method comprising:
executing, by a computer system, a first block storage server virtual machine to host a first volume using one or more storage devices of said computer system;
running, with the computer system, a second virtual machine that can access a virtual block storage device;
executing a block storage client by the computer system;
Executing the block storage client includes:
receiving from the second virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the block storage operation on the first volume;
The computer-implemented method.
5. 5. The computer-implemented method of clause 4, wherein the message is sent over the secure virtual network using a key to encrypt and decrypt traffic sent over the virtual network.
6. 6. The computer implementation of clause 5, wherein a first virtual network address of the virtual network is associated with the first block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client. Method.
7. The first block storage server virtual machine and the second virtual machine are executed by a first one or more processors of the computer system, and the block storage client is executed by a second one or more processors of the computer system. 7. The computer-implemented method of any one of clauses 4-6, executed by a processor.
8. Clauses 4-, wherein said first block storage server virtual machine is a first virtual machine hosted by said computer system and said second virtual machine is a second virtual machine hosted by said computer system; 8. The computer-implemented method of any one of clause 7.
9. further comprising executing, with said computer system, a second block storage server virtual machine to host a second volume using said one or more storage devices of said computer system; is a replica associated with the first volume.
10. The first block storage server virtual machine is run using a first physical component of the computer system and the second block storage server virtual machine is a second block storage server virtual machine of the computer system different from the first physical component. 10. The computer-implemented method of clause 9, performed using two physical components.
11. The first block storage operation is writing a block of data, the block storage client further encrypts the block of data using an encryption key associated with the virtual block storage device, and writes a block of data. generating an encrypted block, the message sent to the first block storage server virtual machine includes the encrypted block of the data, and sending the encrypted block of the data to the first block storage server virtual machine; 11. The computer-implemented method of any one of clauses 4-10, causing the first volume to be written.
12. said computer system being included in an extension of a provider network, said extension of said provider network communicating with said provider network through at least a third party network;
12. The computer of any one of clauses 4-11, wherein the provider network instance management service initiates the execution of the first block storage server virtual machine and the second virtual machine by the computer system. Implementation method.
13. one or more storage devices of the host computer system;
A first one or more processors of the host computer system executing a first block storage server application and a second application capable of accessing a virtual block storage device, the first block storage server application executing said first one or more processors, sometimes including instructions to cause said first block storage server application to host a first volume using said one or more storage devices;
A first one or more processors of the host computer system executing a block storage client application, the block storage client application comprising instructions which, when executed, cause the block storage client application to:
receiving from the second application a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server application to cause the first block storage server application to perform the first block storage operation on the first volume; one or more processors of 1;
system, including
14. 14. The system of clause 13, wherein the message is sent over the virtual network that is secure by encrypting and decrypting traffic sent over the virtual network using a key.
15. Clause 15. The system of Clause 14, wherein a first virtual network address of said virtual network is associated with said first block storage server application and a second virtual network address of said virtual network is associated with said block storage application.
16. The first block storage server application executes within a first virtual machine hosted by the host computer system and the second application executes within a second virtual machine hosted by the host computer system. 16. The system according to any one of clauses 13-15.
17. The first one or more processors of the host computer system further execute a second block storage server application which, when executed, causes the second block storage server to: Clause 13, comprising instructions to cause an application to use said one or more storage devices of said host computer system to host a second volume, said second volume being a replica associated with said first volume. The system described in .
18. The first block storage server application is executed using a first physical component of the computer system and the second block storage server application is executed on a second physical component of the computer system different from the first physical component. 18. The system of Clause 17, implemented using physical components.
19. The first block storage operation is writing a block of data, and the block storage client application further, at runtime, instructs the block storage client application to use an encryption key associated with the virtual block storage device. , the message sent to the first block storage server application includes instructions to cause the block of data to be encrypted and to generate an encrypted block of data, the message sent to the first block storage server application including the encrypted block of data; 19. The system of any one of clauses 13-18, wherein a block storage server application writes encrypted blocks of said data to said first volume.
20. said host computer system being included in an extension of a provider network, said extension of said provider network communicating with said provider network through at least a third party network;
20. The system of any one of clauses 13-19, wherein the provider network instance management service initiates the execution of the first block storage server application and the second application by the host computer system.
21. A computer-implemented method comprising:
receiving a first request to launch a first virtual machine at a host computer system of an extension of a provider network to host a block storage server application, the extension of the provider network comprising at least: said hosting in communication with said provider network via a third party network;
provisioning at least a portion of the storage capacity of one or more physical storage devices of the host computer system to the first virtual machine as a provisioned storage device;
running the block storage server application on the first virtual machine;
Running the block storage server application includes:
creating the logical volume on the provisioned storage device in response to a second request from the provider network block storage service to create a logical volume;
receiving a third request from a block storage client application over the virtual network to perform an input/output operation on the logical volume;
performing the requested I/O operation on the logical volume;
The computer-implemented method.
22. Before executing the block storage server application on the first virtual machine, the method further comprises:
running another application in the first virtual machine to load a boot volume from a machine image obtained from a datastore of the provider network into a boot volume, the boot volume being the provisioned storage; said loading being another logical volume of the device;
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
22. The computer-implemented method of Clause 21, comprising:
23. Before executing the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device. 22. The computer-implemented method of clause 21, wherein the boot volume is another logical volume of the provisioned storage device and the first block storage device is a virtual block storage device.
24. A computer-implemented method comprising:
Receiving a first request to launch a first virtual machine by a computer system of an extension of a provider network to host a block storage server application, the extension of the provider network comprising at least a third said hosting in communication with said provider network via a party network;
provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as a provisioned storage device;
running the block storage server application on the first virtual machine;
Running the block storage server application includes:
creating the logical volume on the provisioned storage device in response to a second request from the provider network block storage service to create a logical volume;
receiving a third request to perform an I/O operation on the logical volume;
performing the requested I/O operation on the logical volume;
The computer-implemented method.
25. Before executing the block storage server application on the first virtual machine, the method further comprises:
running another application in the first virtual machine and loading a boot volume from a machine image obtained from a datastore of the provider network, the boot volume being another of the provisioned storage devices; said loading, which is a logical volume of
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
25. The computer-implemented method of Clause 24, comprising:
26. 26. The computer-implemented method of clause 25, further comprising clearing memory of the first virtual machine prior to executing the block storage server application.
27. Before executing the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device. 25. The computer-implemented method of clause 24, wherein the boot volume is another logical volume of the provisioned storage device and the first block storage device is a virtual block storage device.
28. 28. The computer-implemented method of clause 27, wherein the loading is performed by at least one of a basic input/output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine.
29. the third request is received from a block storage client application via a virtual network;
29. The computer-implemented method of any one of Clauses 24-28, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.
30. The first virtual machine is one of a plurality of virtual machines, each of the plurality of virtual machines running a block storage server application, the method further comprising:
determining that resource utilization associated with one or more of the plurality of virtual machines exceeds a threshold;
provisioning at least a portion of the storage capacity of one or more storage devices of another host computer system to a second virtual machine;
running another block storage server application in the second virtual machine;
30. The computer-implemented method of any one of Clauses 24-29, comprising:
31. provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system to a second virtual machine as another provisioned storage device;
running another block storage server application in said second virtual machine, said first virtual machine being run using a first physical component of said computer system; is performed using a second physical component of the computer system that is different from the first physical component;
31. The computer-implemented method of any one of Clauses 24-30, further comprising:
32. The input/output operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested input/output logical volume on the logical volume writes the block of data to the logical volume. wherein the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network comprises writing to the block address of the 32. The computer-implemented method of clause 31, comprising one or more physical computing devices located outside a data center of a provider network and on the premises of a customer of said provider network.
33. one or more storage devices of a host computer system of extensions of a provider network, wherein said extensions of said provider network communicate with said provider network through at least a third party network; ,
one or more processors of the host computer system for executing the host manager application, the host manager application including instructions which, when executed, cause the host manager application to:
receiving a first request to launch a first virtual machine to host a block storage server application;
provisioning at least a portion of the storage capacity of the one or more storage devices to the first virtual machine as a provisioned storage device;
executing the block storage server application on the first virtual machine, the block storage server application comprising instructions, the instructions, when executed, causing the block storage server application to:
creating the logical volume on the provisioned storage device in response to a second request from the provider network block storage service to create a logical volume;
receiving a third request to perform an I/O operation on the logical volume;
performing the requested I/O operation on the logical volume;
system, including
34. The host manager application includes further instructions that, when executed, before executing the block storage server application in the first virtual machine, cause the host manager application to:
running another application in the first virtual machine and loading a boot volume from a machine image obtained from a datastore of the provider network, the boot volume being another of the provisioned storage devices; said loading, which is a logical volume of
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
34. The system of Clause 33, which causes the
35. 35. The method of clause 34, wherein the host manager application, when executed, includes further instructions that cause the host manager application to clear memory of the first virtual machine before executing the block storage server application. system.
36. Components of the first virtual machine, at run time, cause the component to copy the boot volume of the first virtual machine to the first virtual machine before executing the block storage server application in the first virtual machine. comprising instructions to load from a machine image stored by a block storage device, wherein said boot volume is another logical volume of said provisioned storage device and said first block storage device is a virtual block storage device. 34. The system according to 33.
37. 37. The system of clause 36, wherein the component is at least one of a basic input/output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine.
38. the third request is received from a block storage client application via a virtual network;
38. The system of any one of clauses 33-37, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.
39. The host manager application includes further instructions which, when executed, cause the host manager application to:
provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system to a second virtual machine as another provisioned storage device;
running another block storage server application in the second virtual machine, the first virtual machine running using a first physical component of the computer system, the second virtual machine said performing performed using a second physical component of said computer system different from said first physical component;
39. The system according to any one of clauses 33-38, which causes the
40. The input/output operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested input/output logical volume on the logical volume writes the block of data to the logical volume. wherein the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network comprises writing to the block address of the 40. The system of clause 39, comprising one or more physical computing devices located outside a data center of a provider network and on customer premises of said provider network.
41. A computer-implemented method comprising:
receiving, by a first block storage server instance, a first request from a block storage service of a provider network, said first request creating a first storage volume and said first logical volume; said receiving, storing a first portion of
Receiving, by a second block storage server instance, a second request from the block storage service, the second request to create a second storage volume and to create a first logical volume of the first logical volume. storing the part of 2;
determining that the second block storage server instance is not responding;
sending a third request to a third block storage server instance to create a third storage volume to store said second portion of said first logical volume;
storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume;
updating a datastore containing identification information for each block storage server instance that hosts a portion of the first logical volume to remove identification information for the second block storage server instance and the third block storage adding identification information for the server instance;
The first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems in extensions of the provider network and The computer-implemented method, wherein the extension communicates with the provider network through at least a third party network.
42. receiving, by a block storage client, from the datastore the identification of each block storage server instance hosting a portion of the first logical volume;
establishing a connection by the block storage client to at least one block storage server instance included in the identification;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; When,
sending, by the block storage client, a message over the connection to the at least one block storage server instance to instruct the at least one block storage server instance to transfer the block of data to the block of the first storage volume; to write to an address; and
42. The computer-implemented method of Clause 41, further comprising:
43. 43. The computer-implemented method of any one of Clauses 41-42, further comprising polling for responses of the first block storage server instance and the second block storage server instance.
44. A computer-implemented method comprising:
receiving, by the first block storage server instance, the first request to create a first storage volume and store a first portion of the first logical volume;
receiving, by a second block storage server instance, a second request to create a second storage volume to store a second portion of the first logical volume;
sending a third request to a third block storage server instance to create a third storage volume to store said second portion of said first logical volume;
storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume;
updating a datastore containing identification information for each block storage server instance that hosts a portion of the first logical volume to remove identification information for the second block storage server instance and the third block storage adding identification information for the server instance;
The computer-implemented method, comprising:
45. receiving, by a block storage client, from the datastore the identification of each block storage server instance hosting a portion of the first logical volume;
establishing a connection by the block storage client to at least one block storage server instance included in the identification;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; When,
sending, by the block storage client, a message over the connection to the at least one block storage server instance to instruct the at least one block storage server instance to transfer the block of data to the block of the first storage volume; to write to an address; and
45. The computer-implemented method of Clause 44, further comprising:
46. 46. The computer-implemented method of clause 45, wherein the data store is a distributed data store including multiple nodes that independently run a consensus protocol to update the identity.
47. 47. The computer-implemented method of clause 46, further comprising obtaining, by the block storage client, identification information of each node of the plurality of nodes from a service.
48. 47. The computer-implemented method of clause 46, wherein one node of the plurality of nodes runs within a container hosted by the first block storage server instance.
49. 49. The computer-implemented method of clause 48, further comprising, by said block storage client, broadcasting a request for identification information of any of said plurality of nodes to a plurality of block storage server instances.
50. polling for responses of the first block storage server instance and the second block storage server instance, wherein the third request is responsive to a determination that the second block storage server instance does not respond 49. The computer-implemented method of any one of Clauses 44-49, transmitted as a
51. Clauses 44-50, wherein the first storage volume is striped across multiple block storage server instances, each of the multiple block storage server instances being included in the identification information.
52. The first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems in extensions of a provider network; and said second request is sent from a block storage service of said provider network, and said extension of said provider network communicates with said provider network via at least a third party network. A computer-implemented method as described in .
53. one or more computing devices in extensions of a provider network, said extensions of said provider network communicating with said provider network over at least a third party network, said one or more computing devices The device comprises instructions which, when executed on one or more processors, cause the one or more computing devices to:
receiving, by the first block storage server instance, the first request to create a first storage volume and store a first portion of the first logical volume;
receiving, by a second block storage server instance, a second request to create a second storage volume to store a second portion of the first logical volume;
sending a third request to a third block storage server instance to create a third storage volume to store said second portion of said first logical volume;
storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume;
updating a datastore containing identification information for each block storage server instance that hosts a portion of the first logical volume to remove identification information for the second block storage server instance and the third block storage adding identification information for the server instance;
system, including
54. The one or more computing devices comprise further instructions which, when executed on the one or more processors, cause the one or more computing devices to:
receiving, by a block storage client, from the datastore the identification of each block storage server instance hosting a portion of the first logical volume;
establishing a connection by the block storage client to at least one block storage server instance included in the identification;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; When,
sending, by the block storage client, a message over the connection to the at least one block storage server instance to instruct the at least one block storage server instance to transfer the block of data to the block of the first storage volume; to write to an address; and
54. The system of Clause 53, causing the
55. 55. The system of clause 54, wherein the data store is a distributed data store including multiple nodes that independently run a consensus protocol to update the identity.
56. The one or more computing devices obtain identification information of each node of the plurality of nodes from a service by the block storage client to the one or more computing devices when executing on the one or more processors. 56. The system of clause 55, including further instructions to cause the
57. 56. The system of clause 55, wherein one node of the plurality of nodes runs within a container hosted by the first block storage server instance.
58. The one or more computing devices, when executed on the one or more processors, request the one or more computing devices, by the block storage client, for identification information of any of the plurality of nodes. to the plurality of block storage server instances.
59. The one or more computing devices, when executed on the one or more processors, provide the one or more computing devices with responses of the first block storage server instance and the second block storage server instance. 59. The system of any one of clauses 53-58, comprising further instructions to poll, wherein said third request is sent in response to determining that said second block storage server instance does not respond.
60. 60. The system of any one of clauses 53-59, wherein the first storage volume is striped across multiple block storage server instances, each of the multiple block storage server instances being included in the identification information.

The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made therein without departing from the broader spirit and scope of the disclosure as set forth in the claims.

Claims (15)

A computer-implemented method comprising:
executing, by a computer system, a first block storage server virtual machine to host a first volume using one or more storage devices of said computer system;
running, with the computer system, a second virtual machine that can access a virtual block storage device;
executing, by said computer system, a block storage client;
Executing the block storage client includes:
receiving from the second virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the first block storage operation on the first volume;
Computer-implemented method. 2. The computer-implemented method of claim 1, wherein the message is sent over the virtual network that is secure by using a key to encrypt and decrypt traffic sent over the virtual network. 3. The computer of claim 2, wherein a first virtual network address of said virtual network is associated with said first block storage server virtual machine and a second virtual network address of said virtual network is associated with said block storage client. Implementation method. The first block storage server virtual machine and the second virtual machine are executed by a first one or more processors of the computer system, and the block storage client is executed by a second one or more processors of the computer system. The computer-implemented method of any one of claims 1-3, executed by a processor. 2. The first block storage server virtual machine is a first virtual machine hosted by the computer system, and the second virtual machine is a second virtual machine hosted by the computer system. 5. The computer-implemented method of any one of claims 1-4. further comprising executing, with said computer system, a second block storage server virtual machine to host a second volume using said one or more storage devices of said computer system; 2. The computer-implemented method of claim 1, wherein is a replica associated with the first volume. The first block storage server virtual machine is run using a first physical component of the computer system and the second block storage server virtual machine is a second block storage server virtual machine of the computer system different from the first physical component. 7. The computer-implemented method of claim 6, performed using two physical components. The first block storage operation is writing a block of data, the block storage client further encrypts the block of data using an encryption key associated with the virtual block storage device, and writes a block of data. generating an encrypted block, the message sent to the first block storage server virtual machine includes the encrypted block of the data, and sending the encrypted block of the data to the first block storage server virtual machine; The computer-implemented method of any one of claims 1-7, wherein the first volume is written to. said computer system being included in an extension of a provider network, said extension of said provider network communicating with said provider network through at least a third party network;
9. The provider network instance management service of any one of claims 1-8, wherein the computer system initiates the execution of the first block storage server virtual machine and the second virtual machine. Computer-implemented method. one or more storage devices of the host computer system;
A first one or more processors of the host computer system executing a first block storage server application and a second application capable of accessing a virtual block storage device, the first block storage server application executing said first one or more processors, sometimes including instructions to cause said first block storage server application to host a first volume using said one or more storage devices;
a first one or more processors of the host computer system executing a block storage client application;
The block storage client application includes instructions;
The instructions, when executed, cause the block storage client application to:
receiving from the second application a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server application to cause the first block storage server application to perform the first block storage operation on the first volume;
system. 11. The system of claim 10, wherein the message is sent over the virtual network that is secure by encrypting and decrypting traffic sent over the virtual network using a key. 12. The system of claim 11, wherein a first virtual network address of said virtual network is associated with said first block storage server application and a second virtual network address of said virtual network is associated with said block storage client application. . The first block storage server application executes within a first virtual machine hosted by the host computer system and the second application executes within a second virtual machine hosted by the host computer system. The system according to any one of claims 10-12, wherein The first one or more processors of the host computer system further execute a second block storage server application which, when executed, causes the second block storage server to: 3. comprising instructions for causing an application to use said one or more storage devices of said host computer system to host a second volume, said second volume being a replica associated with said first volume. 11. The system according to 10. The first block storage server application is executed using a first physical component of the host computer system, and the second block storage server application is executed on a different physical component of the host computer system than the first physical component. 15. The system of claim 14, implemented using two physical components.

Images