JP7440195B2 - Virtualized block storage server in cloud provider board expansion - Google Patents

Description

Many businesses and other organizations operate computer networks that interconnect many computing systems to support their operations; ) co-located, or alternatively, located at multiple different geographic locations (e.g., connected via one or more private or public intermediate networks). For example, data centers housing a significant number of interconnected computing systems have become commonplace, such as private data centers operated by and on behalf of a single organization. , and public data centers operated by an entity as a business to provide computing resources to customers. Some public data center operators provide network access, power, and secure installation facilities for hardware owned by various customers, while other public data center operators provide network access, power, and secure installation facilities for use by their customers. It offers "full service" facilities, including available hardware resources. However, as the size and scope of typical data centers increases, the tasks of provisioning, supervising, and managing physical computing resources become increasingly complex.

The advent of general-purpose hardware virtualization technology offers benefits in managing large-scale computing resources for many customers with diverse needs, allowing various computing resources to be efficiently and efficiently managed by multiple customers. Enable secure sharing. For example, virtualization technology allows a single physical computing machine to be shared among multiple users by providing each user with one or more virtual machines hosted by the single physical computing machine. It can be possible. Each such virtual machine is a software simulation that functions as a unique logical computing system that provides the user with the illusion of being the sole operator and manager of a given hardware computing resource, while at the same time It also provides application isolation and security between machines. Additionally, some virtualization technologies are capable of providing virtual resources that span two or more physical resources, such as a single virtual machine with multiple virtual processors that span multiple unique physical computing systems. be. As another example, virtualization technology may enable data storage hardware to be shared among multiple users by providing each user with a virtualized data store that may be distributed across multiple data storage devices; Each such virtualized data store functions as a unique logical data store that provides the user with the illusion that the user is the sole operator and administrator of the data storage resource.

Various virtual machine types, optimized for different types of applications such as compute-intensive applications, memory-intensive applications, etc., are set up in data centers of several cloud computing provider networks as required by clients. can be done. In addition, high-level services that rely on such provider network virtual computing services, such as some database services where database instances are instantiated using virtual computing service virtual machines, are also available to provider network clients. It could be possible. However, for some types of applications, such as applications that process fairly large amounts of data that need to be stored on customer premises outside the provider network, virtualization using hardware located in the provider network's data center Services that are limited to providing customized resources may not be optimal, for example, for latency-related and/or other reasons.

Various embodiments according to the present disclosure will be described with reference to the following drawings.

FIG. 2 is a block diagram illustrating an example provider network extended with provider board extensions located in a network external to the provider network, in accordance with at least some embodiments. FIG. 2 is a block diagram illustrating an example provider board extension, in accordance with at least some embodiments. FIG. 2 is a block diagram illustrating an example connection between a provider network and a provider board extension, according to at least some embodiments. 1 is a block diagram illustrating an example virtualized block storage system, according to at least some embodiments. FIG. 1 is a block diagram illustrating an example virtualized block storage system, according to at least some embodiments. FIG. 1 is a block diagram illustrating an example system for booting a virtualized block storage server using a first technique, in accordance with at least some embodiments. FIG. 1 is a block diagram illustrating an example system for booting a virtualized block storage server using a second technique, in accordance with at least some embodiments. FIG. FIG. 2 is a block diagram illustrating an example system for booting additional compute instances of a provider board extension from a block storage server, in accordance with at least some embodiments. 1 is a block diagram illustrating an example system for managing virtualized block storage servers, according to at least some embodiments. FIG. 1 is a block diagram illustrating an example system for providing volume mapping to block storage clients, according to at least some embodiments. FIG. 1 is a block diagram illustrating an example system for tracking volume mapping, according to at least some embodiments. FIG. FIG. 2 is a flow diagram illustrating the operation of a method for starting a virtualized block storage server, according to at least some embodiments. FIG. 2 is a flow diagram illustrating the operation of a method for using a virtualized block storage server, according to at least some embodiments. FIG. 2 is a flow diagram illustrating the operation of a method for managing virtualized block storage servers in provider board expansion, according to at least some embodiments. 1 illustrates an example provider network environment, in accordance with at least some embodiments. 1 is a block diagram of an example provider network that provides storage services and hardware virtualization services to customers, according to at least some embodiments. FIG. 1 is a block diagram illustrating an example computing device that may be used in at least some embodiments. FIG.

This disclosure provides a method for configuring a provider board extension to communicate with a network external to a provider network and for providing virtualized resources on a board extension that are the same or similar to resources available in the provider network. , apparatus, system, and non-transitory computer-readable storage medium. A provider network operator (or provider) provides its users (or customers) with the ability to utilize one or more of various types of computing-related resources, such as computing resources (e.g. running virtual machines (VMs) and/or containers, running batch jobs, running code without server provisioning), data/storage resources (e.g., object storage, block-level storage, data archive storage, databases and tables, etc.), network-related resources (e.g., virtual network configurations including groups of compute resources, content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build/deployment services), Examples include access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other compute resources may be provided as services.

Provider network operators often offer these and other computing resources as services that rely on virtualization technology. For example, a Compute instance (e.g., a VM using a guest operating system (OS) that runs using a hypervisor that may or may not additionally run on top of the underlying host OS) using virtualization technology; It may provide users with the ability to control or utilize containers that may or may not run in VMs (instances that can run on "bare metal" hardware without an underlying hypervisor), using a single electronic device. can implement one or more Compute instances. Accordingly, users may directly utilize compute instances provided by instance management services (sometimes referred to as hardware virtualization services) hosted by provider networks to perform various computing tasks. Additionally or alternatively, a user may utilize a Compute instance indirectly by submitting code to be executed by a provider network (e.g., utilized by an on-demand code execution service), and then to execute code (typically without the user having any control over or knowledge of the associated underlying Compute instance(s)).

Resources that support both the services that provide computing-related resources to users and the computing-related resources that are provisioned to users may generally be referred to as provider network substrates. Such resources generally include hardware and software in the form of many networked computer systems. Provider network board traffic and operations may be broadly subdivided into two categories in various embodiments: control plane traffic carried on the logical control plane and data plane operations carried on the logical data plane. The data plane represents the movement of user data through the distributed computing system, whereas the control plane represents the movement of control signals through the distributed computing system. The control plane generally includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic generally involves establishing isolated virtual networks for various customers, monitoring resource usage and status, identifying specific hosts or servers on which requested Compute instances are launched, and optionally Includes administrative operations such as provisioning additional hardware. The data plane includes customer resources (e.g., compute instances, containers, block storage volumes, databases, file storage) that are implemented in the provider network. Data plane traffic generally includes non-managed operations such as transferring data to and from customer resources. Control plane components are typically implemented on a separate set of servers from the data plane servers, and control plane traffic and data plane traffic may be sent through separate/different networks. In some embodiments, control plane traffic and data plane traffic can be supported by different protocols. In some embodiments, messages (eg, packets) sent through the provider network include a flag indicating whether the traffic is control plane traffic or data plane traffic. In some embodiments, the payload of traffic can be inspected to determine its type (eg, control plane or data plane). Other techniques for differentiating traffic types are possible.

While some customer applications migrate easily to provider network environments, some customer workloads remain on-premises due to low latency, large amounts of data, data security, or other customer data processing requirements. (“on-premises”). Exemplary on-premises environments include customer data centers, robotic integration, field locations, colocation facilities, telecommunications facilities (eg, near mobile phone base stations), and the like. To satisfy customer requirements, the present disclosure relates to the deployment of resources such as substrates on-premises. The term "provider board extension" (PSE) refers to the term "provider board extension" (PSE), which a customer can deploy on-premises (e.g., geographically remote from the provider network), but with the same or similar functionality provided by the provider network (e.g., virtualized). A collection of resources (e.g., hardware, software, firmware, configuration metadata, etc.) that provides Such resources may be physically delivered as one or more computer systems or servers delivered in racks or cabinets such as those commonly found in on-premises locations. A PSE can provide customers with a set of features and capabilities that can be deployed on-premises, similar to those of provider networks described above. Effectively, from the perspective of a provider network customer, a PSE represents a local extension of the provider network's capabilities that can be set up at any physical location (e.g., in terms of physical space, power, Internet access, etc.) that can accommodate the PSE. . From the perspective of the provider network itself, the PSE may be considered to be virtually located in the same provider network data center as the core provider network board, while being physically located at the customer's chosen deployment site. In at least some embodiments, a customer physically hosting a PSE may grant permission to its own customers (e.g., other users of the provider network) to launch instances and host their respective workloads within the PSE at an on-premises location and, in some cases, allow that workload to access the customer's network.

In at least some embodiments, the PSE is preconfigured with appropriate combinations of hardware, software, and/or firmware elements, e.g., by a provider network operator, to support various types of computing-related resources. and may support in a manner that satisfies various local data processing demands without compromising the security of the provider network itself or any other customers of the provider network. In at least some embodiments, the PSE is generally managed by the same or similar set of interfaces that customers use to access computing-related resources within the provider network. For example, customers can use the same application programming interfaces (APIs) or console-based interfaces that they otherwise use to provision, manage, and operate computing-related resources within a provider network to to provision, manage, and operate computing-related resources within an on-premise PSE or multiple PSEs at various deployment sites.

In at least some embodiments, provider network resources instantiate various networking components to ensure secure and reliable communications between the provider network and the PSE. Such components can establish one or more secure tunnels (eg, VPNs) with the PSE. Such components can further partition control plane traffic and data plane traffic and treat each type of traffic differently based on factors including the direction of the traffic (eg, to and from the PSE). In at least some embodiments, control plane services dynamically provision and configure these networking components for deployed PSEs. Such control plane services may monitor the network components of each PSE and invoke self-healing or repair mechanisms designed to prevent loss of communication with the PSE due to failures occurring within the provider network.

One of the services commonly offered to provider network customers is block storage services, which allow instances to act as virtualized persistent disks for instances as they are presented with a logical view of physical storage resources. Mapping the logical storage space to the actual physical location of storage is handled by the virtualization system. Volumes may be replicated one or more times to provide high availability and durability to customers, and replicas are typically stored on different servers. A customer can attach one or more block storage volumes to an instance, and clients that support the instance can use the virtualized block storage volumes to perform block-based operations on the instance. For example, a customer may boot a particular instance from a given boot volume (e.g., the volume containing the operating system) and create another attached volume to support data stored by customer applications run by the instance. It can be specified to have. To provide high flexibility with respect to available storage, provider networks separate the physical storage devices that support a given attached volume from the computing resources that support a given compute instance. Traditionally, a fleet of block storage servers supporting block storage services will partition the attached physical storage devices (eg, solid state drives, magnetic drives, etc.) into many logical volumes. One block storage server may support hundreds or even thousands of instances of storage resources. These block storage servers typically run in a "bare metal" configuration. Server software runs within an operating system environment that boots directly on dedicated server hardware, for example, instead of on a virtual machine or within a container.

This bare metal block storage server design may not be well suited for board expansion because the provider board's expansion capability capacity may be significantly limited compared to the availability zone capacity. For example, because some board expansions may have only a single server (or another few servers), it may not be possible to dedicate an entire server to a block of storage resources, thereby is no longer available for Compute instances. To address this challenge, embodiments of the present disclosure virtualize block storage services so that they can be launched within Compute instances, thereby enabling more adaptive and efficient use of limited capacity. Make it. For example, a single server can be configured to host both a Compute instance and an instance that has virtualized its attached volumes (e.g., a block storage server), making use of the PSE's finite set of resources. provides high adaptability. Also, a single server may be partitioned into isolated fault domains to host multiple block storage servers (and even multiple replicas of volumes). A fault domain generally refers to a logical part of a system that can fail without affecting other parts of the system. When running on a bare metal system, a block storage system's fault domain typically corresponds to the entire bare metal computer system. By using virtualization to separate fault domains from a per-server to a sub-server basis, you can take advantage of redundancy in the underlying hardware used to support block storage server instances within a PSE. The number of failure domains may increase. As a result, block storage servers can manage small amounts of data individually so that when a failure occurs, the workload for recovering data associated with the failure is reduced. These block storage instances can be created as virtual private clouds (VPCs), and instance clients that communicate with block storage volumes can also communicate in this same VPC. Beneficially, this allows the use of VPC encryption to make communications in the board extension more secure.

However, when virtualizing a block storage service with a provider board extension, it would be possible for the block storage service to store the boot volume (also the block storage service machine image) before it is started on the board extension. Particular technical challenges arise, including initializing block storage services from Although PSEs provide low-latency compute resources to on-premises facilities, those resources are subject to increased latency when returning to the provider network. Although the PSE may rely on block storage servers within the realm of the provider network, such dependence is undesirable because it suffers from increased latency. As described in further detail below, embodiments of the present disclosure load the block storage server boot volume of the PSE's local storage from data stored in an area of the provider network and then boot the block storage server. This challenge is addressed by local boot techniques that can provide volumes to other instances launched within the PSE. Accordingly, the disclosed local boot technique enables a customer to launch an instance on a board extension using a block storage service machine image, even when the block storage service itself is not yet present on the board extension.

The disclosed systems and techniques also protect provider networks from potential security issues that may be made possible by connecting a PSE to a provider network. In some embodiments, a PSE may require a secure networking tunnel from the customer site where the PSE is installed to the provider network board (eg, the machine's physical network) in order to operate. These tunnels may include virtual infrastructure components hosted on both virtualized computing instances (eg, VMs) and substrates. Examples of tunnel components include VPCs and proxy computing instances and/or containers running on computing instances. Each server within a PSE may use at least two tunnels, one for control plane traffic and one for data plane traffic. As described in further detail below, intermediate resources located along the network path between the provider network board and the PSE can securely manage traffic flowing between the board and the PSE.

In at least some embodiments, the provider network is a cloud provider network. A cloud provider network or "cloud" refers to a large pool of accessible virtualized computing resources, such as compute, storage, and network resources, applications, and services. The cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be provisioned and released programmatically in response to customer commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads. Cloud computing therefore refers to both applications delivered as services over publicly accessible networks (e.g., the Internet, cellular communications networks) and the hardware and software of cloud provider data centers that provide those services. be able to.

A cloud provider network can be formed as a number of regions, where a region is a geographical area in which a cloud provider clusters its data centers. Each region may include two or more availability zones interconnected via, for example, a private high speed network of fiber communication connections. Availability Zone refers to an isolated failure domain that includes one or more data center facilities with separate power supplies, separate networks, and separate cooling than those in another Availability Zone. Preferably, the availability zones within a region are located sufficiently far apart from each other such that the same natural disaster does not take more than one availability zone offline at the same time. Customers can connect to the cloud provider network's availability zones via publicly accessible networks (eg, the Internet, cellular communications networks). The PSEs described herein can also connect to one or more availability zones via a publicly accessible network.

A cloud provider network may include a physical network called a substrate (eg, sheet metal box, cable). The cloud provider network may also include an overlay network of virtualized computing resources running on the board. Thus, network packets can be routed along the substrate network according to the configuration of the overlay network (eg, VPC, security groups). A mapping service can coordinate the routing of these network packets. The mapping service is a geographically distributed lookup service that maps the overlay IP and network identifier combination to the substrate IP so that the distributed substrate computing device can locate the destination of the packet.

To illustrate, each physical host may have an IP address on the substrate network. Hardware virtualization techniques may allow multiple operating systems to run simultaneously on a host computer, for example, as host virtual machines. A host's hypervisor or virtual machine monitor allocates the host's hardware resources among the host's various virtual machines and monitors the execution of the virtual machines. Each virtual machine may be provided with one or more IP addresses of the overlay network, and the host's virtual machine monitor may recognize the host's virtual machine's IP address. The virtual machine monitor (and/or other devices or processes on the network board) uses encapsulation protocol techniques to transfer network packets (e.g., client IP packets) between virtualized resources of different hosts within the cloud provider network. can be encapsulated and routed onto the network board. Encapsulation protocol techniques can be used in the network board to route encapsulated packets between endpoints of the network board via an overlay network path or route. Encapsulation protocol technology may be viewed as providing a virtual network topology overlaid on the network substrate. The encapsulation protocol technique may include a mapping service that maintains a mapping directory that maps IP overlay addresses (public IP addresses) to substrate IP addresses (private IP addresses), the mapping service for routing packets between endpoints. can be accessed by various processes such as cloud provider networks.

As those skilled in the art will appreciate in light of this disclosure, certain embodiments may be capable of achieving various advantages, including some or all of the following (a)-(c). (a) where the provider managed infrastructure (e.g., PSE) is used at sites selected by the customer while still maintaining the scalability, security, availability, and other operational benefits enabled by the provider network; (b) applications that need to be transferred over long distances, such as over links between customer data centers and provider network data centers; (c) reduce the amount of data and the resulting amount of data; (c) by moving the application closer to the data source/destination, the latency and overall responsiveness is improved; and/or (d) security of sensitive application data is improved.

FIG. 1 is a block diagram illustrating an example provider network extended with provider board extensions located in a network external to the provider network, in accordance with at least some embodiments. Within provider network 100, a customer can create one or more isolated virtual networks 102. A customer can launch a compute instance 101 within the IVN to run its applications. These compute instances 101 are hosted by substrate addressable devices (SADs) that are part of a provider network substrate (not shown). Similarly, a SAD that is part of a provider network board can host control plane services 104. Exemplary control plane services 104 include instance management services (sometimes referred to as hardware virtualization services) that enable customers or other control plane services to launch and configure instances and/or IVNs, object These include object storage services that provide storage, block storage services that provide the ability to attach block storage devices to instances, database services that provide various database types, and so on.

Note that the components shown within provider network 100 can be treated as logical components. As mentioned above, these components are hosted by the SAD of the provider network board (not shown). For example, the provider network board can host instance 101 using a container or virtual machine running within an isolated virtual network (IVN). Such containers or virtual machines are executed by SAD. As another example, a provider network board can host one or more control plane services 104 using SAD in a bare metal configuration (eg, without virtualization). In at least some embodiments, SAD is software executed by hardware (e.g., a server) that is addressable via the network address of the provider network rather than another network (e.g., customer network, IVN, etc.). refers to In at least some embodiments, SAD may also refer to the underlying hardware (eg, computer system) that executes the software.

As shown, the provider network 100 communicates with a provider board extension (PSE) 188 located within a customer network 185 and a PSE 198 located within a customer network 195. Each PSE includes one or more substrate addressable devices (SADs), such as SADs 189A-189N shown within PSE 188. Such SAD 189 facilitates the provisioning of computing related resources within the PSE. As in SAD 189A-189N, solid box-ellipse-dashed box combination illustrations of components are generally used to indicate that more than one of those components may be present in this and subsequent drawings. (note that the corresponding textual references refer to the singular or plural form of the component, with or without a trailing letter). Customer gateway/router 186 provides connections between provider network 100 and PSE 188 as well as connections between PSE 188 and other customer resources 187 (e.g., other on-premise servers or on-premise services connected to customer network 185). provide. Similarly, customer gateway/router 196 provides connectivity between provider network 100 and PSE 198 as well as between PSE 198 and other customer resources 197. Various connectivity options exist between provider network 100 and PSE 198, such as a public network such as the Internet as shown in PSE 188 or a direct connection as shown in PSE 198.

Within the provider network 100, control plane traffic 106 is generally (but not always) directed to the SAD, while data plane traffic 104 is generally (but not always) directed to the instances. For example, some SADs may sell APIs that allow launching and terminating instances. The control plane service 104 can launch a new instance on the IVN 102 by sending commands to the API of the relevant SAD via the control plane.

An IVN, as suggested by its name, is a set of hosted (e.g., virtualized) resources that are logically separated or remote from other resources in a provider network (e.g., other IVNs). may be included. Control plane services can set up and configure IVNs, including assigning an identifier to each IVN to distinguish it from other IVNs. A provider network can provide various ways to enable communication between IVNs, such as by setting up peering relationships between IVNs (e.g., one IVN's gateway is configured to communicate with another IVN's gateway). ).

IVNs can be established for various purposes. For example, an IVN may be set up for a particular customer by reserving a set of resources dedicated to the customer, and there is considerable flexibility regarding the network configuration of that set of resources provided to the customer. Within the IVN, customers can set up subnets, assign desired private IP addresses to various resources, set up security rules governing incoming and outgoing traffic, and so on. In at least some embodiments, by default, the set of private network addresses set up within one IVN may be inaccessible from another IVN (or more generally, from outside the IVN).

Tunneling techniques facilitate traversal of IVN traffic between instances hosted by different SADs of provider network 100. For example, a newly launched instance within IVN 102 may have IVN address A and be hosted by a SAD with substrate address X, while instance 101 may have IVN address B and be hosted by a SAD with substrate address Y. To facilitate communication between these compute instances, SAD ) Each instance is encapsulated within the payload of the packet with address information of the hosting SAD. Packets sent between SADs may further include an identifier of IVN 102 to indicate that the data is directed to IVN 102, as opposed to another IVN hosted by the SAD at substrate address Y. In some embodiments, the SAD further encrypts packets sent between instances within the payload of the packets sent between the SADs using an encryption key associated with the IVN. In at least some embodiments, encapsulation and encryption is performed by a software component of the SAD hosting the instance.

In the case of a PSE, provider network 100 includes one or more networking components, thereby effectively extending the provider network board outside of provider network 100 to a PSE connected to a customer's on-premises network. Such components can ensure that data plane and control plane operations targeting the PSE are securely, reliably, and transparently communicated to the PSE. In the illustrated embodiment, PSE interface 108, PSESAD proxy 110, and PSESAD anchor 112 facilitate data plane and control plane communications between provider network 100 and PSE 188. Similarly, PSE interface 118, PSESAD proxy 120, and PSESAD anchor 122 facilitate data plane and control plane communications between provider network 100 and PSE 198. As described herein, the PSE interface receives control plane traffic and data plane traffic from the provider network, sends such control plane traffic to the PSESAD proxy, and sends such data plane traffic to the PSE. The PSE interface also receives data plane traffic from the PSE and transmits such data plane traffic to appropriate destinations within the provider network. The PSESAD proxy receives control plane traffic from the PSE interface and sends such control plane traffic to the PSESAD anchor. The PSESAD anchor receives control plane traffic from the PSESAD proxy and sends such control plane traffic to the PSE. The PSESAD anchor also receives control plane traffic from the PSE and sends such control plane traffic to the PSESAD proxy. The PSESAD proxy also receives control plane traffic from the PSESAD anchor and sends such control plane traffic to the appropriate destination within the provider network. Other embodiments may employ different combinations or configurations of network components to facilitate communication between provider network 100 and a PSE (e.g., a PSE interface, a PSESAD proxy function, and/or a PSESAD anchor). Both PSE interfaces and PSESAD proxies, both PSESAD proxies and PSESAD anchors, applications that perform the operations of all three components, etc. may be combined in various ways).

As indicated above, each PSE has one or more substrate network addresses for SADs (eg, SADs 189A-189N). Since their board addresses are not directly reachable via the provider network 100, the PSE interfaces 108, 118 masquerade with an attached virtual network address (VNA) that matches the board address of each PSE. As shown, PSE interface 108 has attached VNA(s) 150 that match the SAD address(es) of PSE 188 and PSE interface 118 has attached VNA(s) 150 that match the SAD address(es) of PSE 198. (Multiple possible) 152 are attached. For example, traffic destined for a SAD with Internet Protocol (IP) address 192.168.0.10 within PSE 188 is sent to PSE interface 108 with an attached virtual address 192.168.0.10 and Traffic directed to the SAD with IP address 192.168.1.10 is sent to PSE interface 118 with an attached virtual address of 192.168.1.10. Note that IPv4 or IPv6 addressing may be used. In at least some embodiments, a VNA is a logical construct that allows various network-related attributes, such as IP addresses, to be programmatically transferred between instances. Such transfers may be referred to as "attaching" the VNA to the instance and "detaching" the VNA from the instance.

At a high level, the PSE interface is essentially a packet forwarding component that routes traffic based on whether it is control plane traffic or data plane traffic. Note that given the substrate addressing and encapsulation techniques described above, since both control traffic and data plane traffic are directed to the SAD, they are both routed to the PSE interface. For control plane traffic, the PSE interface routes traffic to the PSESAD proxy based on the SAD address. For data plane traffic, the PSE interface includes one or more encrypted data plane traffic tunnels between provider network 100 and the PSE (e.g., tunnel 191 between PSE interface 108 and PSE 188, tunnel 191 between PSE interface 108 and PSE 188, It establishes a tunnel 193) with the PSE 198 and functions as an endpoint. For data plane traffic received from provider network 100, the PSE interface encrypts the traffic for transmission to the PSE through the tunnel. For data plane traffic received from a PSE, the PSE interface decodes the traffic, optionally verifies the SAD addressing of the packet, and sends the traffic to the identified SAD destination via provider network 100. Note that if a PSE interface receives traffic from a PSE that does not conform to the expected format (e.g., protocol) used to transmit data plane traffic, the PSE interface can drop such traffic. In addition, the PSE interface verifies the addressing of the encapsulated packets so that the originator of the traffic (e.g., an instance hosted by the PSE within a particular IVN) is connected to the addressed destination (e.g., in the same or a different IVN). Note that you can ensure that you are allowed to send traffic to (instances hosted by the provider network within).

Each SAD of a PSE has a corresponding group of one or more PSE interfaces, and each member of the group establishes one or more tunnels for data plane traffic with the PSE. For example, if there are four PSE interfaces in a PSE with four SADs, each PSE interface establishes a secure tunnel (eg, 16 tunnels) with a respective data plane traffic endpoint of the SAD. Alternatively, a group of PSE interfaces may be shared by multiple SADs by attaching an associated VNA to each member of the group.

Each PSE has one or more PSESAD proxies and one or more PSESAD anchors that handle control plane traffic between provider network 100 and the PSE's SAD. Control plane traffic typically has a command-response or request-response format. For example, a control plane service of provider network 100 can issue a command to PSESAD to start an instance. Control plane commands sent over a secure tunnel generally should not originate from the PSE, as management of PSE resources is facilitated from the provider network. At a high level, the PSESAD proxy acts as a stateful security boundary between the provider network 100 and the PSE (such a boundary is sometimes referred to as a data diode). To that end, the PSESAD proxy may employ one or more techniques, such as applying various security policies or rules to the received control plane traffic. Other control plane services 104 indirectly or directly provide public APIs for PSE-hosted instances to issue commands to provider network 100 via non-tunneled communications (e.g., over a public network such as the Internet). Note that it may be possible to

For traffic originating from within the provider network 100 and directed to the PSE, the PSESAD proxy may provide the control plane endpoint API of its corresponding SAD within the PSE. For example, a PSESAD proxy for a PSESAD that can host an instance can provide an API that matches the API that can receive control plane operations to launch, configure, and terminate the instance. Depending on the API calls and related parameters directed to and received by the PSESAD proxy, the PSESAD proxy may perform various operations. For some operations, the PSESAD proxy may pass the operation parameters and related parameters to the destination SAD without modification. In some embodiments, PSESAD proxies may verify that the parameters of API calls received from within provider network 100 are eligible for the API before completing their operations.

For some API calls or related parameters, PSESAD can act as an intermediary to prevent sensitive information from being sent outside of provider network 100. Exemplary sensitive information includes cryptographic information such as encryption keys, network certificates, etc. For example, the PSESAD proxy can decrypt the data using a private key and re-encrypt the data using a key that can be exposed to the PSE. As another example, the PSESAD proxy may terminate a first secure session (e.g., a Transport Layer Security (TLS) session) originating from within the provider network 100 and connect it to the corresponding SAD using a different certificate. A new secure session can be created and provider network certificates can be prevented from being leaked to the PSE. Accordingly, the PSESAD proxy can receive certain API calls from within the provider network 100 that include sensitive information and issue alternative or replacement API calls to the PSESAD that exchanges the sensitive information.

With respect to traffic originating from a PSE and destined for the provider network 100, the PSESAD proxy, for example, Only undefined commands or requests can be dropped.

In some embodiments, the PSESAD proxy can process responses to control plane operations depending on the nature of the expected response, if any. For example, for some responses, the PSESAD proxy may simply drop the response without sending any message to the originator of the corresponding command or request. As another example, for some responses, the PSESAD proxy sanitizes the response and sends the response to the corresponding command or request before sending the sanitized response to the originator of the corresponding command or request via control plane traffic 107. or can ensure that the expected response format of the request is followed. As yet another example, the PSESAD proxy generates a response (either immediately or upon receipt of an actual response from the SAD) and passes the generated response to the originator of the corresponding command or request via control plane traffic 107. can be sent.

As part of serving as a security boundary between the provider network 100 and the PSE, the PSESAD proxy can track the state of communications between components of the provider network (eg, control plane services 104) and each SAD of the PSE. State data is exchanged or exchanged with session keys during the session, pending outbound API calls with associated sources and destinations to track outstanding responses, and API calls received from within provider network 100. It may include the relationship to that API call issued to the SAD, along with replaced sensitive information, etc.

In some embodiments, the PSESAD proxy can provide stateful communication for network communication from other PSEs to providers in addition to control plane traffic. Such communications may include Domain Name System (DNS) traffic, Network Time Protocol (NTP) traffic, and operating system activation traffic (eg, for Windows activation).

In some embodiments, only certain components of the PSE may serve as endpoints for encrypted control plane traffic tunnels with provider network 100. To provide redundancy and reliability for the connection between the provider network 100 and the PSE, the PSESAD anchor can serve as a provider network side endpoint for each of the PSE's available tunnel endpoints. As shown, PSESAD anchor(s) 112 serve to tunnel control plane traffic to PSE 188 via tunnel 190 and PSESAD anchor(s) 122 serve to tunnel control plane traffic to PSE 1198 via tunnel 192. work to do.

Various embodiments both use the techniques for processing traffic described above and isolate those network components that are exposed to traffic from other parts of provider network 100. This can limit the radial impact of any planned attack originating from outside the provider network (e.g., from a situation where the PSE is included). Specifically, network components can operate within one or more IVNs to limit the scope to which an attacker can penetrate, thereby protecting the operations of both the provider network and other customers. Accordingly, various embodiments may instantiate the PSE interface, PSESAD proxy, and PSESAD anchor as applications executed by virtual machines or containers running within one or more IVNs. In the illustrated embodiment, a group of PSE interfaces of different PSEs are activated within a multi-tenant IVN (eg, PSE interface IVN 132 of PSE 188 and PSE 198). In other embodiments, each group of PSE interfaces can be launched with a single tenant IVN. Additionally, each group of PSESAD proxies and each group of PSESAD anchors for a given PSE is a single-tenant IVN (e.g., PSESAD Proxy IVN 134 of PSE 188, PSESAD Anchor IVN 136 of PSE 188, PSESAD Proxy IVN 138 of PSE 198, and PSESAD Proxy IVN 138 of PSE 198). N40 ).

The redundancy provided by running multiple instances of each of the network components (e.g., PSE interfaces, PSESAD proxies, and PSESAD anchors) allows provider networks to Note that it is possible to periodically recycle the instances hosting those components. Recycling may include, for example, restarting an instance or launching a new instance and reconfiguring other instances using, for example, the address of the recycled instance. Periodic recycling limits the time window in which an attacker can take advantage of a compromised network component if the network component is compromised.

PSE connection manager 180 manages the setup and configuration of network components that provide connectivity between provider network 100 and PSEs. As previously discussed, PSE interfaces 108, 118, PSESAD proxies 110, 120, and PSESAD anchors 112, 122 may be hosted as instances by the provider network board. The PSE connectivity manager 180 connects the PSE interface(s) to the PSESAD proxy/proxy(s) when the PSE is shipped to a customer and/or when the PSE comes online and exchanges configuration data with the provider network. , and activation of the PSE's PSESAD anchor(s). In addition, PSE connection manager 180 can further configure PSE interface(s), PSESAD proxy/proxies, and PSESAD anchor(s). For example, the PSE attach manager 180 attaches the VNA(s) corresponding to the PSE's SAD to the PSE interface(s) and assigns the address of the PSESAD's proxy/proxies to the PSE interface(s). and provide the PSESAD's proxy/proxies with the address of the PSESAD anchor(s) of the PSE's PSESAD. Additionally, the PSE connection manager 180 may be configured to provide a variety of connections, such as to enable communication between the PSE interface IVN 132 and the PSE's PSESAD proxy IVN, and between the PSESAD proxy IVN and the PSE's PSESAD anchor IVN. You can configure IVNs for components.

To facilitate the establishment of the tunnels 190-193, the tunnel endpoints are connected from outside the respective networks (e.g., from outside the provider network for the PSE interface and PSESAD anchor, from outside the customer network for the PSE's tunnel endpoints). ) Note that it may have one or more attached VNAs or assigned physical network addresses that can receive traffic. For example, the PSE 188 may have a single outgoing network address and manage communications to multiple SADs using port address translation (PAT) or multiple outgoing network addresses. Each of the PSESAD anchors 112, 122 may have or be shared (e.g., via PAT) with an outward facing network address, and each of the PSE interfaces 108, 118 may have an outward accessible network address. can be obtained or shared (eg, via PAT).

FIG. 2 is a block diagram illustrating an example provider board extension, in accordance with at least some embodiments. In the illustrated embodiment, PSE 188 includes one or more PSE frameworks 202 and one or more hosts 220. At a high level, each of the hosts 220 is functionally (and in some cases structurally) connected to at least a portion of a computer system that forms part of a provider network board (e.g., its board resources that host instances within the provider network). ), while the PSE framework(s) 202 may be similar to ) to provide the support infrastructure for providing connectivity to provider networks.

In at least some embodiments, each of the PSE frameworks 202 transmits control plane traffic or data plane traffic to the hosts 220 in a mesh-like architecture, as illustrated by PSE control plane traffic 240 and PSE data plane traffic 242. and vice versa. Such redundancy allows for the level of reliability that customers can expect from provider networks.

PSE framework 202 includes one or more control plane tunnel endpoints 204 that terminate encrypted tunnels (eg, tunnel 190, tunnel 192) that carry control plane traffic. In some embodiments, provider network 100 hosts a PSESAD anchor for each of the control plane tunnel endpoints 204. Returning to the provider network, the PSESAD proxy or proxies (e.g., proxy 110) can distribute control plane traffic to the PSESAD anchors (e.g., anchor 112) and load the control plane traffic to the PSE framework of PSE 188. effectively distributed over 202. The PSE framework 202 further includes one or more data plane tunnel endpoints 206 that terminate encrypted tunnels (e.g., tunnel 191, tunnel 193) that carry data plane traffic from the provider network's PSE interfaces to PSE frames. Works 202 may further be connected in a mesh-like architecture (eg, a given PSE interface 108 establishes a tunnel with a respective data plane tunnel endpoint 206 of PSE framework 202).

As indicated above, control plane traffic packets and data plane traffic packets may include SADs as both sources and destinations. The latter (packets of data plane traffic) are encapsulated into packets with SAD-based addressing. As shown, PSE framework 202 is SAD 289 and host 220 is SAD 290. A SAD within a PSE 188 (e.g., SADs 289, 290) may also provide secure session termination (e.g., Note that TLS termination) can be provided.

The SAD sells one or more control plane APIs to handle control plane operations directed to the SAD that manage the SAD's resources. For example, PSE manager 210 of PSE framework 202 can sell control plane APIs for managing components of PSE framework 202. One such component may include control plane traffic and data plane traffic, such as, for example, control plane traffic directed to SAD 289 for PSE manager 210 and control plane traffic or data plane traffic directed to SAD 290 for host manager 222. and/or a PSE gateway 208 that routes data plane traffic to and from PSE 188. PSE gateway 208 may further facilitate communication with the customer network, such as to or from other customer resources 187 accessible via the PSE deployment site's network (eg, customer network 185).

The PSE manager 210 API may include one or more commands for configuring the PSE gateway 208 of the PSE framework 202. Other components 212 of PSE framework 202 may include various applications or services involved in the operation of the PSE substrate for host 220, such as DNS, Dynamic Host Configuration Protocol (DHCP), and/or NTP services. .

Host manager 222 may sell control plane APIs for managing components of host 220. In the illustrated embodiment, host manager 222 includes instance manager 224 and network manager 226. Instance manager 224 can handle API calls related to managing host 220, including commands to start, configure, and/or terminate instances hosted by host 220. For example, an instance management service within a provider network (not shown) can issue control plane commands to instance manager 224 to launch an instance on host 220. As shown, host 220 hosts a customer instance 232 running inside a customer IVN 233 , a third party (3P) instance 234 running inside a 3PIVN 235 , and a service instance 236 running inside a service IVN 237 . Note that each of these IVNs 233, 234, 235 can extend an existing IVN established within a provider network. Customer instance 232 may run several customer applications or workloads, and 3P instances 234 may run applications or workloads of another party that the customer has authorized to launch instances within PSE 188. , service instance 236 may execute provider network services (eg, block storage services, database services, etc.) that are provided locally to PSE 188.

Network manager 226 can process SAD addressed data plane traffic received by host 220. For such traffic, the network manager may perform the necessary decapsulation of the IVN packets before sending them to the addressed hosted instance. Additionally, network manager 226 can handle the routing of traffic sent by hosted instances. When a hosted instance attempts to send traffic to another locally hosted instance (eg, on the same host), network manager 226 can forward the traffic to the addressed instance. When a hosted instance attempts to send traffic to a non-local instance (e.g., not on the same host), network manager 226 finds the board address of the device hosting the non-local instance, encapsulates the corresponding packet, Optionally, the packet can be encrypted into a SAD-addressed packet and sent across the data plane (e.g., either to another host within the PSE or back to the provider network via the PSE gateway 208). carried out). Note that the network manager 226 may contain or have access to various data that facilitates the routing of data plane traffic (e.g., it may determine the destination of packets received from hosted instances). (to find the address of the SAD hosting the instance with the IVN network address).

FIG. 3 is a block diagram illustrating example connections between a provider network and provider board extensions, according to at least some embodiments. Specifically, FIG. 3 shows an example connection between a provider network and a PSE. Note with respect to FIG. 3 and as shown at the top of the figure, the term "inbound" refers to traffic received from the PSE by the provider network, and the term "outbound" refers to traffic sent by the provider network to the PSE. sea bream. Although not shown, this example assumes that the PSE includes two SAD PSE frameworks 202 and two hosts 220 (four total). The PSE framework provides tunnel endpoints 204A, 204B for control plane traffic and tunnel endpoints 206A, 206B for data plane traffic. Outbound traffic is decrypted and sent via PSE gateways 208A, 208B to a destination within the PSE board.

For each of the four SADs, the provider network includes a VNA, one or more PSE interfaces, and one or more PSESAD proxies. In this example, the provider network includes a PSESAD VNA 304, two PSE interfaces 108A, 108B, and two PSESAD proxies 110A, 110B for a given PSESAD. Together, the PSE interface(s) and the proxy/proxies of the PSESAD may be referred to as slices as shown, with each slice corresponding to a particular SAD within the PSE. In other embodiments, the PSE interface(s) may be shared by all VNAs for the VPN rather than a single VNA for one of the SADs.

PSESAD VNA 304 acts as a front for a given PSE through which other components of the provider network can send traffic to and receive traffic from the PSE's corresponding SAD. A load balancer (not shown) can route outbound traffic sent to PSESAD VNA 304 to one of PSE interfaces 108A, 108B. The illustrated PSE interfaces 108A, 108B for a given slice and those of other slices (not shown) operate within PSE interface IVN 132. The PSE interfaces 108A, 108B send control plane traffic to the PSE by sending data plane traffic to the PSE via a data plane traffic tunnel and forwarding the control plane traffic to the slice's PSESAD proxy 110A, 110B. The PSE interfaces 108A, 108B provide the network address of the PSESAD proxy/proxies of the associated SAD, the network address of the data plane tunnel endpoint(s), and communication with those endpoint(s). Stores (or has access to) one or more keys or associated keys of the data plane tunnel endpoint(s) of the PSE to protect.

In at least some embodiments, the PSE interfaces 110A, 110B establish secure tunnels for data plane traffic with each of the data plane tunnel endpoints 206A, 206B, resulting in N data plane tunnels, where where N is the number of PSE interfaces per SAD (assuming each SAD has the same number of interfaces) multiplied by the number of data plane tunnel endpoints, multiplied by the number of SADs. In this example, 16 data plane tunnels are established between the PSE interfaces and the data plane tunnel endpoints (ie, 2 PSE interfaces per SAD x 2 data plane tunnel endpoints x 4 SADs).

The PSESAD proxies 110A, 110B receive control plane traffic from the PSE interfaces 108A, 108B to perform various operations described elsewhere herein and route the traffic through either of the two PSESAD anchors 112A, 112B. and send control plane traffic to the PSE. Similarly, the PSESAD proxies 110A, 110B receive control plane traffic from either of the two PSESAD anchors 112A, 112B to perform various operations described elsewhere herein and to Control plane traffic 107 to the destination. The illustrated PSESAD proxies 110A, 110B for a given slice and the proxies for other slices (not shown) operate within the PSESAD proxy IVN 134. The PSE interfaces 108A, 108B store (or have access to) the network address of the PSESAD anchor(s).

In at least some embodiments, the PSESAD proxies may have access to or otherwise exchange information with the shared data store 306. Such information exchange can be used for several reasons. For example, recall that a PSESAD proxy can sell an API interface to emulate the API interface of an associated SAD within a PSE. One PSESAD proxy may not be supported by different PSESAD proxies because some communications may be stateful and various load balancing techniques may prevent the same PSESAD proxy from handling all communications for a given set of operations. There may be a need to access previously processed communication state (eg, PSESAD proxy 110A sends control plane operations to the PSE, and PSESAD proxy 110B receives responses to the control plane operations from the PSE). For inbound control plane traffic, the PSESAD proxy checks whether the inbound message matches the expected state and, if so, sends the control plane Messages can be sent via traffic 107. If there is no match, PSESAD proxies 110A, 110B can drop the traffic. As another example, recall that a PSESAD proxy can bridge separate secure sessions (eg, TLS sessions) such that provider network credentials are not sent to the PSE. Again, the PSESAD proxy that handles the outbound message may be different from the PSESAD proxy that handles the response to that message, so the PSESAD proxy that handles the response message sends the secure response message via the control plane traffic 107. The same key established between the originator of the outbound message and the PSESAD proxy that processed the outbound message can be used to send it to the originator.

In this example, each PSE framework provides a single control plane tunnel endpoint 204. For each available control plane tunnel endpoint 204, the provider network includes a PSE anchor. In this example, the provider network includes two PSE anchors 112A, 112B. PSESAD anchors 112A, 112B operate within PSESAD anchor IVN 136. PSE anchor 112 receives control plane traffic from each of the eight PSESAD proxies (two per slice in each of the four SADs) and sends the traffic to the PSE. The PSE anchor also receives control plane traffic from the PSE and sends the traffic from the PSE to one of two PSESAD proxies associated with the SAD from which the traffic originates. The PSE anchors 112A, 112B provide the network address of the SAD's respective PSESAD proxy/proxies, the network address of the control plane tunnel endpoint(s), and the communication with those endpoint(s). Stores (or has access to) one or more keys or associated keys of the PSE's control plane tunnel endpoint(s) to protect.

In at least some embodiments, a network component or provider network may employ load balancing techniques to distribute the workload of routing control and data plane traffic between the provider network and the PSEs. For example, traffic sent to PSESA DVNA 304 may be distributed between PSE interfaces 108A, 108B. As another example, each of the PSE interfaces 108 can distribute traffic between data plane tunnel endpoints 206A, 206B. As yet another example, each of the PSE interfaces 108 can distribute traffic between PSESAD proxies 110A, 110B. As yet another example, each of PSESAD proxies 110 can distribute outbound traffic between PSESAD anchors 112A, 112B. As yet another example, PSESAD anchor 112 can distribute inbound traffic between PSESAD proxies 110A, 110B. In either case, such load balancing may be performed by a transmitting entity or a load balancer (not shown). An exemplary load balancing technique is to employ a load balancer with a single VNA that distributes traffic to multiple components "behind" that address, and to assign multiple recipient addresses to each data sender. distribution of selected recipients at the application level, etc.

The embodiments shown in FIGS. 1-3 depict the establishment of separate tunnels for control plane traffic and data plane traffic; other embodiments include one or more tunnels for both control plane traffic and data plane traffic. Note that tunnels may be employed. For example, the PSE interface can route data plane traffic to the PSESAD anchor for transmission to the PSE over a shared tunnel, bypassing the additional operations performed by the PSESAD proxy on the control plane traffic.

4 and 5 are block diagrams illustrating example virtualized block storage systems, according to at least some embodiments. As shown in FIG. 4, provider network 400 or PSE 488 includes host 420A. Host 420A is the host for block storage server instance 450. In hosting a block storage server instance 450, the host 420A may use some of the host storage device(s) 421 as storage devices 447 (e.g., virtual drives or virtual disks) provisioned for the block storage server instance 450. Provisioning amount. Block storage server instance 450 can partition provisioned storage device 447 and create and host volumes on behalf of other instances. An instance management service (not shown) of provider network 400 can initiate the startup of such block storage server instances on either provider network 400 hosts or PSE 488 hosts via instance manager 426 . Instance manager 426 is a virtual machine manager (VMM), hypervisor, or other device that can provision host resources (e.g., compute, memory, network, storage) for instances (e.g., virtual machines), and start and terminate instances. It can be a virtualization program.

Host 420A is also the host of instance A 430 within instance IVN 432. Block storage server instance 450 provides block storage volume A 434 to instance A 430 and block storage volume B 440 to another instance (not shown). Host 420A includes a block storage client 460A through which hosted instances access block storage volumes hosted by block storage servers (eg, by providing virtual block storage interfaces to the instances). Specifically, block storage client 460 intercepts or otherwise receives block storage operations issued to volumes “attached” to an instance (e.g., block storage volume A 434 attached to instance A 430). It can be a software program that

As shown in FIG. 5, provider network 400 or PSE 488 includes two hosts 420A and 420B. Again, host 420A is the host of block storage server instance 450. Host 420B is the host of instance B 436 within instance IVN 438. Block storage server instance 450 provides block storage volume B 440 to instance B 436. Host 420B includes a block storage client 460B whose hosted instances access block storage volumes hosted by block storage servers.

When a volume is attached to an instance, the block storage client obtains a mapping that associates the volume with the block storage server instance(s) that hosts the replica. A block storage client may open a communication session (eg, via a network-based data transfer protocol) with each of the block storage server instances. Upon receiving a block storage operation from an instance, the block storage client issues the operation to the appropriate block storage server(s) to implement the operation on the instance. For example, referring to FIG. 4, instance A 430 may issue a read of a block of data at a given block address (eg, logical block address or LBA) from volume A 434. Block storage client 460A receives the read operation and issues an operation to block storage server instance 450 that includes the requested block address based on the mapping data of instance attached volume A 434 to block storage server instance 450. Upon receiving data from block storage server instance 450, block storage client 460A provides the data to instance A 430. A similar set of operations can be performed between instance B 436, block storage client 460B, block storage server instance 450, and volume B 440 of FIG. 5.

Referring to FIGS. 4 and 5, block storage messaging between block storage clients and block storage server instances may be sent using a network-based data transfer protocol such as Global Network Block Device (GNDB). Such messages can be encapsulated as data plane traffic and routed between block storage client 460 and block storage server 450 by network manager 424 of each host, as described elsewhere herein. . In some embodiments where the block storage server instance is hosted by the same host as the instance that issues the block storage operation, the block storage client can issue the related operation to the network manager via a network-based data transfer protocol, and then , the network manager can internally route operations to hosted block storage server instances. In other embodiments where the block storage server instance is hosted by the same host as the instance that issues block storage operations, the block storage client may have a PCIe or other interconnect that resides between the block storage client and the hosted instance. Related operations can be issued over the connection, thereby avoiding protocol packing associated with network-based transport.

Referring to FIGS. 4 and 5, the system can deploy various levels of encryption to protect block storage data. To provide end-to-end encryption of block storage traffic, a block storage server instance 452 runs within a block storage service (BSS) IVN 452 that includes any number of block storage server instances and provides access to the PSE from a provider network. It can reach up to Block storage client 460 has a VNA 462 associated with block storage service IVN 452. In this manner, IVN level encryption using block storage service IVN encryption key 484 may be used to encrypt and decrypt traffic sent between block storage server instance 450 and block storage client 460. Note that encryption and decryption of traffic carried through the block storage service IVN may be performed by a network manager at the endpoints of the IVN. For example, network manager 424B encrypts traffic sent from block storage client 460B via VNA 462B. As another example, network manager 424A decrypts traffic sent to instances hosted within BSS's IVN 452, including block storage server instance 450. To provide encryption at rest of data, block storage volumes may further have volume-level encryption. For example, volume A 434 may be encrypted with volume A encryption key 480 and volume B 440 may be encrypted with volume B encryption key 482. Using encryption key 480, block storage client 460A encrypts data written to volume A 434 by instance A 430 and decrypts data read from volume A 434 by instance A 430. Using encryption key 482, block storage client 460B can perform similar encryption and decryption operations between instance B 436 and volume B 440. Some embodiments prevent the block storage server instance that hosts a volume from being hosted on the same host as the instance attached to that volume, and ensure that the volume encryption key is not hosted on the same host as the volume it encrypts. Note that the policy that prevents being hosted may be in-place (eg, Volume A encryption key 480 and Volume A 434 are no longer on host 420A).

In some embodiments, the instance and host managers are executed by different processor(s), as shown in FIGS. 4 and 5. For example, block storage server instance 450 may be executed by a first processor (eg, processor 1710) and host manager 422A may be executed by a second processor (eg, processor 1775). Processor(s) running a block storage server instance accesses host storage device(s) via one or more interconnects. For example, processor(s) (e.g., processor 1710) running block storage server instance 450 access storage device(s) 421 (e.g., storage device 1780) via a PCIe interconnect between the two. It is possible. As another example, the processor(s) running the block storage server instance 450 connects the storage device(s) to the processor or system-on-chip running the host manager 422A (e.g., processor 1775) via a first PCIe bus. ) 421, which in turn bridges those communications to the storage device(s) 421 via the second PCIe bus.

Although shown as a single volume in FIGS. 4 and 5, each instance volume 434, 440 is mirrored across multiple replicas stored on multiple different block storage servers to provide high reliability to the system. obtain. For example, one replica may be referred to as a primary replica that handles reads from and writes to a volume (input and output operations, or "input/output (I/O)"), and a replica that hosts the primary replica. server can synchronously propagate writes to other secondary replicas. If a primary replica fails, one of the secondary replicas is selected to act as the primary replica (referred to as failover). Replicas may be subdivided (eg, striped) into partitions, with each partition of a given replica stored on a different server to facilitate parallel reads and writes. Replicas may also be encoded with redundancy (e.g., parity bits) so that when a block storage server becomes unavailable (e.g., due to server or disk failure, etc.), lost data can be replaced with available data. I can recover. As a result, volumes attached to an instance may be hosted by many block storage servers.

One challenge associated with hosting block storage servers in a virtualized context is booting those servers. If there are no operational block storage server instances, how does a block storage server instance boot from a boot volume hosted by a block storage server instance? This is particularly difficult in the context of PSEs, which may not include dedicated (or bare metal) block storage servers. 6 and 7 illustrate examples of local boot techniques for booting a first or "seed" block storage server instance (a block storage server instance that can launch other instances) within a PSE. Once launched, you can launch other instances from the seed block storage server instance.

FIG. 6 is a block diagram illustrating an example system for booting a virtualized block storage server using a first technique, in accordance with at least some embodiments. At a high level, FIG. 6 illustrates an example of remotely booting block storage server 650 in instance 690 via a proxy server hosted within provider network 600. A block storage service (BSS) 606 of provider network 600 provides users with the ability to create block storage volumes and attach those volumes to instances. For example, a user (e.g., a customer of a provider network, another service of a provider network) can submit commands via an API or console relayed to the BSS 606 to create, resize, or create volumes managed by the BSS. and attach or detach those volumes to an instance. BSS 606 maintains a BSS object store 608 that includes volume snapshots. A snapshot, like a file, can be considered a type of object that can include object data and metadata about the object (eg, creation time, etc.). A volume snapshot is a copy (or backup) of a volume at a given point in time, allowing it to be recreated on a physical or virtual drive. A snapshot can generally be divided into a boot volume and a non-boot volume, the former (boot volume) facilitating booting of software running within the instance. A machine image may be a group of one or more snapshots of a given instance, including the boot volume.

In some embodiments, BSS 606 applies object-level encryption of objects within BSS object store 608. To prevent compromise of the encryption keys used within the provider network 600, the BSS 606 can further manage the PSE object store 610 and use encryption different from that used to encrypt objects in the BSS object store 608. The key is used to re-encrypt objects sent to a given PSE. In this example, block storage server instance machine image 612A is used to boot block storage server 650 on PSE host 620. As shown in circle A, the BSS 606 decrypts the block storage server instance machine image 612A using a key held within the provider network 600 and sends it to the block storage server using a key 613 that can be sent to the PSE. It is re-encrypted as an instance machine image 612B.

An IVN can provide security for traffic between block storage server instances and block storage clients. In some embodiments, the BSS 606 manages its IVN, designated as BSS IVN 652. The host 620 on which the block storage server 650 is started includes a host manager 622 that has a block storage client 660 with an associated VNA 662 of a BSS's IVN 652. To facilitate communication between the block storage client 660 and the provider network 600 through the BSS's IVN 652, the BSS 606 launches a proxy server instance 642 within the BSS's IVN 652. Proxy server instance 642 may provide a block storage-like interface to block storage client 660 and allow client 660 to access machine image 612B of the block storage server instance. BSS 606 launches proxy server instance 642 as follows.

As shown in circle B, BSS 606 first sends a message to bare metal block storage server 614 to create a new logical volume to serve as the boot volume for proxy server instance 642. Bare metal block storage server 614 creates a new logical volume 619 on its one or more storage devices 618 of bare metal server 614 and loads proxy server machine image 616 stored in BSS object store 608 into volume 619.

As shown in circle C, BSS 606 initiates startup of proxy server instance 642 using instance management service 640 . Instance management service 640 is another control plane service in provider network 600 that facilitates launching instances within provider network 600 or a PSE. In some embodiments, instance management service 640 tracks host usage metrics that indicate how "hot" the host is (i.e., metrics such as CPU usage, memory usage, and network usage). be able to do or have access to it. For example, instance management service 640 or another service may periodically poll hosts to obtain utilization metrics. Instance management service 640 determines whether the host is underutilized (subject to some constraint regarding the location of the host within a given PSE, etc.) by accessing metrics associated with a pool of potential hosts (e.g., at some threshold or multiple (below a threshold), the host that hosts the instance can be identified in response to the launch request. In requesting the startup of proxy server instance 642, BSS 606 specifies that the instance should be launched from volume 619 within BSS's IVN 652. As shown in circle D, instance management service 640 identifies the host 646 that hosts the instance and sends a message to host 646 to start the instance. An instance manager (not shown) of host 646 provisions the host's resources for proxy server instance 642 and boots it from boot volume 619 .

BSS 606 also initiates startup of block storage server 650 via instance management service 640. BSS 606 specifies that the instance should boot from a boot volume made available by proxy server instance 642 within the BSS's IVN 652 of a particular PSE (eg, using a PSE identifier). As shown in circle E, instance management service 640 identifies host 620 that hosts block storage server instance 650 and sends a message to host 620 (e.g., via a control plane traffic tunnel) to host the instance. to start. As shown, a request to launch an instance is received by an instance manager 626, which can provision host resources for the instance and launch and terminate the instance. Here, instance manager 626 creates instance 690 to host the block storage server.

As part of configuring instance 690, instance manager 626 configures instance 690's basic input/output system (BIOS) 692 and installs block storage server instance machine image 612B on virtual boot volume 694 via proxy server instance 642. Can be loaded. BIOS 692 includes a block storage device driver (eg, a Non-Volatile Memory Express (NVMe) driver) and can attach two block storage devices. Virtualized block storage devices are presented by proxy server instance 642 via block storage client 660 and boot volume 694. Although shown within instance 690 , a boot volume 694 is allocated to instance 690 by instance manager 626 from a storage device (not shown) of host 620 that is accessible via block storage client 660 or another component of host manager 622 . Note that it matches the volume of the virtual drive you created. Other embodiments may include an interface other than BIOS (eg, a Unified Extensible Firmware Interface (UEFI)) to connect instance 690 to two block devices.

During execution, BIOS 692 (or other firmware such as UEFI) can load boot volume 694 from block storage server instance machine image 612B. When block storage server instance machine image 612B is encrypted using key 613, block storage client 660 decrypts block storage server instance machine image 612B during the load process. Once the load operation is complete, the BIOS continues the boot process to boot instance 690 from boot volume D94 containing block storage server software. In some embodiments, BIOS 692 can detach the block storage device corresponding to proxy server instance 642 before booting block storage server software 650.

Although shown and described using proxy server instance 642, other embodiments use a block storage server instance (not shown) launched from block storage server instance machine image 612A to Devices can be provided to instance 690 via block storage client 660. The BSS 606 creates a volume on the virtual drive or virtual disk of the block storage server instance being started and loads the block storage server instance machine image 612A into the volume. During the loading process, the block storage server instance uses the provider network key to decrypt the machine image and optionally uses a volume-specific key (for example, how key 480 encrypts volume B 440) to encrypt the volume. It can be encrypted and provided by BSS 606 to block storage client 660 .

FIG. 7 is a block diagram illustrating an example system for booting a virtualized block storage server using a second technique, in accordance with at least some embodiments. At a high level, FIG. 7 illustrates an example of remotely booting an instance 790 to run block storage server software 750 via pre-boot software executed by instance 790 prior to running block storage server software 750. show. Specifically, the storage device 721 of the PSE 720 is preloaded with a pre-boot instance boot volume 723 (eg, before shipping the PSE). Pre-boot instance boot volume 723 contains software that can be booted into the instance during block storage server pre-boot phase 798 to load block storage server software into a separate boot volume. The instance can then restart and boot from the other boot volume to start the block storage server instance during the block storage server boot phase 799.

As shown in circle A, the BSS 606 initiates the startup of a block storage server instance using the instance management service 640 and specifies that the instance should be launched within the BSS's IVN of a particular PSE. Again, the instance management service 640 identifies the PSE's host 620 that hosts the block storage server instance 650 and sends a message to the host's host manager 722 (e.g., via a control plane traffic tunnel) to start the instance. Send. A request to launch an instance is received by an instance manager (not shown) of host manager 722.

As shown in circle B, host manager 722 (or instance manager) provisions host resources for instance 790 running block storage server 750. In addition, the host manager 722 is presented with two attached block storage devices, a pre-boot instance boot volume 723 and a virtual boot volume 794 (such as the boot volume 694 described above) by a block storage client (not shown). block storage device interface) to BIOS 792A (or UEFI) and can boot from the boot volume 723 of the pre-boot instance.

As shown in circle C, BIOS 792A then enables booting of pre-boot software 796 from pre-boot instance boot volume 723. If the pre-boot instance boot volume 723 is based on a generic machine image, the host manager 722 further configures the instance 790 to facilitate communication between the pre-boot software 796 and the provider network 700, as shown in circle D. You may need to update your configuration. Specifically, host manager 722 attaches a VNA to instance 790, provides pre-boot software 796 with credentials to use when accessing PSE object store 610, and installs a machine image to be loaded into boot volume 794. You can provide the key to decrypt it. The VNA configuration, credentials, and keys may have been passed to host manager 722 as part of the request from instance management service 640 to launch an instance.

As shown in circle E, once instance 790 is configured and runs pre-boot software 796, pre-boot software 796 copies block storage server instance machine image 612B from PSE object store 610 and loads it into boot volume 794. do. As part of loading boot volume 794, pre-boot software 796 can decrypt block storage server instance machine image 612B with key 613, assuming it was encrypted. The host manager 722 can detect when the boot volume 794 has been loaded, or the pre-boot software can signal the completion of the load. For example, pre-boot software can initiate a reboot of the instance while boot volume 794 is being loaded. As another example, network manager 724 can detect termination of a session between instance 790 and PSE object store 610 (eg, detect when a TCP session is closed).

As shown in circle F, host manager 722 can update or otherwise reconfigure BIOS 792A to 792B either during the data transfer or once the data transfer is complete. Such reconfiguration includes changing the boot volume from pre-boot instance boot volume 723 to boot volume 794 and removing the block storage device containing pre-boot instance boot volume 723. Prior to restarting instance 790, host manager 722 may also clear the memory provisioned to instance 790, which may contain residual data written by pre-boot software 796. When instance 790 is restarted, BIOS 792B boots block storage server 750 from boot volume 794.

Although FIGS. 6 and 7 illustrate example techniques for booting block storage server instances within a PSE, other techniques are possible. For example, the instance manager can create a volume on the PSE's storage device as part of the provisioning resources for the instance it launches. Before launching an instance, the instance manager can send a message to a control plane service (eg, block storage service or instance management service) to request a bootable snapshot to load into a volume. Control plane services can send bootable snapshots to instance managers via control plane traffic tunnels. Once loaded, the instance manager allows instance software to be started.

FIG. 8 is a block diagram illustrating an example system for booting additional compute instances of a provider board extension from a block storage server, in accordance with at least some embodiments. For example, once a block storage server instance is launched using techniques such as those described with reference to FIG. 6 or FIG. ). This is particularly useful in the context of a PSE where the seed block storage server instance is launched from a machine image that is transferred or hosted over a relatively slow connection between the provider network and the PSE compared to the PSE's board interconnect. obtain. In this example, instance 890 is started on host 820A of PSE 888. Instance 890 is running block storage server software 850 and operates within the BSS's IVN 844. Additionally, block storage clients 860A and 860B have attached VNAs 862A and 862B that enable communication in the BSS's IVN 844.

As shown in circle A, the instance management service 640 of the provider network 800 can request that the BSS 606 create a new volume from a specified snapshot. As shown in circle B, BSS 606 can instruct block storage server instance 890 to create a volume based on the specified snapshot. In some embodiments, instance management service 640 may provide a volume-specific encryption key (eg, key 680) to encrypt the volume.

As shown in circle C, block storage server instance 890 can create a volume by fetching a specified snapshot from an object store 810, such as PSE object store 610. Although not shown, in other embodiments, object store 810 may be an object store hosted by a BSS cache instance of PSE 888 for caching volume snapshots and/or machine images. BSS 606 may manage BSS cache instances, and customers may specify to BSS 606 that particular snapshots are to be loaded into the cache prior to any request to launch an instance. In this way, block storage server instance 890 can create volumes from cached boot volume snapshots, avoiding the delays associated with transferring data from the provider network 800 object store to block storage server instance 890 . Dramatically reduces instance boot time.

As shown in circle C, the instance management service 640 of the provider network 600 issues commands to the hosts of the PSE 888 to attach or detach volumes hosted by the block storage server instance 890 to other instances hosted by the PSE 888. can. For example, volume A 832 may be loaded with a bootable snapshot. Instance management service 640 can instruct host 820A to launch instance A 830 using the volume identifier associated with volume A 832. In response, block storage client 860A can attach volume A 832 hosted by block storage server instance 890 to instance A 830 via BSS's IVN 844, thereby allowing instance A 830 to boot from volume A 832. become. As another example, volume B 836 may be loaded with a non-bootable snapshot that includes other data. Instance management service 640 can instruct host 820B to attach volume B 836 to hosted instance 834. In response, block storage client 860B can attach volume B 836 hosted by block storage server instance 890 to instance B 834 via BSS's IVN 844.

FIG. 9 is a block diagram illustrating an example system for managing virtualized block storage servers, according to at least some embodiments. Running a block storage server within a virtualized environment provides several benefits, including allowing the instance management service to automatically scale the number of running block storage server instances as needed. The Instance Management Service is not a BSS that operates on a predefined pool of manually expanded bare metal servers, but instead monitors the utilization of host resources (e.g., CPU, memory, network, etc.) and manages the usage of running block storage server instances. The number can be adjusted automatically. For example, if two hosts of block storage server instances are reporting high resource utilization, the instance management service can launch additional block storage server instances on the hosts with low resource utilization. The block storage service can then create volumes on newly launched instances, thereby avoiding increased workload and possible performance degradation of running block storage server instances.

Additionally, running a block storage server as an instance allows the number of failure domains to exceed the number of host computer systems by isolating failure domains from the overall computer system. An increase in the number of fault domains allows for an increase in the number of block storage servers running on a fixed set of hardware, and an increase in the number of block storage servers reduces the amount of data managed by any particular server. ’s overall footprint is reduced. For example, assume that the system includes nine host computer systems that are not subdivided into smaller fault domains (ie, there is one fault domain per computer system). To avoid data loss, each host or failure domain runs a single block storage server. If those nine block storage servers host 90 terabytes (TB) of data (including encoding to enable data recovery) across the instance's volumes, each block storage server will store approximately 10 TB of data. . If one of the hosts fails, that 10TB will need to be recovered (eg, from the other 80TB). Such data recovery involves data computation, transfer, and storage costs. If those nine hosts are each subdivided into two fault domains and the number of block storage servers increases to 18, each block storage server will store approximately 5TB of data and one component of the fault domain will fail. In this case, the amount of data (and corresponding cost) that would need to be recovered would be roughly halved.

In this example, PSE 988 or provider network 900 includes three hosts 920A, 920B, and 920C. Depending on the host's hardware design, the host can either be treated as a single fault domain or subdivided into two or more fault domains. Here, each host 920 includes two failure domains. Host 920A includes a fault domain 922 and a fault domain 924 such that components in one fault domain can continue to operate even if a component in another fault domain fails. For example, failure domain 922 corresponds to a first processor of a multiprocessor system connected to a first memory bank (e.g., RAM) and using a first set of one or more storage drives (e.g., SSD). While failure domain 924 may correspond to a second processor of the system connected to a second memory bank and using a second set of one or more storage drives. Note that some components may be shared across fault domains, such as power supplies, but again this is subject to redundancy in the hardware design and how the fault domains are overlaid on the hardware. sea bream.

Host 920A is running block storage server instance 950A in failure domain 922 and block storage server instance 950B in failure domain 924. Host 920B is running block storage server instance 950C in failure domain 926 and block storage server instance 950F in failure domain 928. Volume A includes a primary replica 660 provided by block storage server instance 950A and a secondary replica 662 provided by block storage server instance 950B. Volume B includes a primary replica 664 provided by block storage server instance 950C and a secondary replica 666 provided by block storage server instance 950A. Volume C includes a primary replica 668 provided by block storage server instance 950B and a secondary replica 670 provided by block storage server instance 950C. Although shown as having two replicas per volume, in reality each volume may have fewer or more replicas, and each replica may have many different blocks (e.g., by striping). Can be split between storage server instances.

As shown in circle A, the instance management service 940 monitors the host's physical resource usage, such as processor usage, memory usage, storage usage, and network usage, which are aggregated across hosts. may be separated by fault domain, or may be per resource (e.g., the instance management service 940 may have an average CPU usage of 50%, the CPU usage of the processor(s) in the fault domain 922 may be (or a metric may be received from host 920A indicating that the CPU utilization of a particular processor supporting fault domain 922 is 50%). Instance management service 940 can track host resource usage in a database (not shown). Note that in addition to block storage server instances, the illustrated fault domain can host other instances (eg, customer instances) that contribute to the utilization of the host's physical resources. Instance management service 940 periodically updates BSS 906 with resource utilization metrics to ensure that BSS 906 selects a less utilized block storage server instance when creating a new volume, as shown in circle B. can be made possible.

As shown in circle C, instance management service 940 can launch a new block storage server instance when the resource utilization of the host supporting the block storage server instance exceeds one or more thresholds. Thresholds can be set for each resource aggregated across hosts (e.g., average processor utilization of all processors running a block storage server is 50%) and for some combination of resources across hosts (e.g., storage usage processor utilization is greater than 80%, processor utilization is greater than 50%), or on an individual resource and/or host basis. To avoid starting a block storage server instance in a fault domain that already hosts a block storage server instance, the instance management service 940 determines which fault domains are occupied (e.g., in a database containing resource usage metrics). and can track which ones are available. Here, the instance management service 940 determines that the resource utilization of the hosts 920A, 920B, and/or 920C exceeds a certain threshold and that a new It was determined that block storage server instances 950D and 950E were running. Although fault domain 928 includes a block storage server 950F that is not hosting any volumes, the resource utilization of other instances hosted within that fault domain or the total resource utilization of host 920B may exceed that of another block storage server instance. May be too expensive to support.

As shown in circle D, instance management service 940 can update BSS 906 to provide updated identification information for operational block storage instances 950 (here including block storage servers 950D and 950E). Based on the identified block storage instance 950 and the resource utilization metrics received from the instance management service 940, the BSS 906 can create a new volume on the host 920C, as shown in circle E. Now, assuming that the host 920C has shown low resource utilization as reported by the instance management service 940, the BSS 906 determines that the primary replica 670 provided by the block storage server instance 950D and the block storage server instance Create a new volume D containing the secondary replica 672 provided by the 950E.

FIG. 10 is a block diagram illustrating an example system for providing volume mapping to block storage clients, according to at least some embodiments. As alluded to above, a single volume may be associated with multiple replicas, such as a primary replica and several secondary replicas. Each replica may be distributed across several block storage servers. The association between a volume, its replica, and the server that hosts that replica (or part of a replica) can occur when a hardware failure occurs or when data is transferred between servers as part of a background load-balancing operation. As you migrate, you can change it over time. When deploying block storage servers on PSEs, you need to ensure high data availability (e.g., the primary replica becomes the secondary replica in the event of a failure), high durability (e.g., the process of recreating lost data in the event of a server failure). It is preferable that it is possible to change their association even when PSEs are disconnected from the provider network or it is impossible to reach the provider network, in order to maintain .

As illustrated, the PSE 1088 includes components that track how volumes are distributed (or mapped) across block storage server instances, so that when a block storage client needs to access a volume, it A storage client can find a block storage server that hosts its volumes. The exemplary volume A mapping data 1098 includes several items for each entry. The items include a server identifier (eg, a unique identifier associated with an instance or host hardware), a server address (eg, an IP address), and a volume type of volume A (eg, primary or secondary). The mapping data may include different or additional items such as block identifiers (eg, if replicas are split or striped across multiple block storage server instances). Mapping data 1098 for example volume A shows that volume A has a primary replica 1052 provided by block storage server instance 1050A and two secondary replicas 1054 and 1056 provided by block storage server instances 950B and 1050C, respectively. Indicates that it includes. Note that PSE 1088 can and likely hosts many other block storage server instances (not shown).

To reliably store volume mapping data 1098, distributed data store 1064 can store volume mapping data 1098. In some embodiments, each distributed data store corresponds to a cluster of nodes that separately maintain the state of the volume mapping(s). Each node in the cluster exchanges messages with other nodes in the cluster to update its state based on the state seen by the other cluster nodes. One of the nodes of the cluster may be designated as the leader or primary node to which changes to the volume mapping are proposed. The nodes of the cluster can implement a consensus protocol, such as the Paxos protocol, to propose and agree on changes to the volume mapping data for a given volume. A cluster may track volume mapping data for one or more volumes. As shown, cluster 1066 tracks volume mapping data 1098 for volume A, while other clusters 1067 track volume mapping data for other volumes.

In some embodiments, the nodes of the cluster are instances run by hosts in the provider network. Such instances persist individual views of volume mapping data to a host's non-volatile storage (e.g., via a block storage client to a volume hosted by a block storage server instance). In other embodiments, the nodes of the cluster are pieces of block storage server software executed by block storage server instances. As shown in example node software environment 1090, a node can run as a container 1082 hosted within a container engine process included in block storage server software 1080. Such nodes can persist views of volume mapping data directly to volumes provided by block storage server software. Preferably, the nodes of the cluster are hosted by separate instances or within separate fault domains.

Like other software executed by the PSE's host, the nodes are subject to hardware failures. In such case, the remaining nodes (or the provider network's block storage service) will detect the loss of the node, create a new node, and replace the lost node based on a consensus view of the other nodes' volume mapping data. update the volume mapping data for the new node. As a result, not only the volume mapping data can be changed, but also the identity of the instance hosting the nodes of the cluster. Cluster mapping data can be used to track clusters. Exemplary cluster mapping data 1099 includes several items, such as a node identifier (eg, VOL_A_NODE1), a node address (eg, IP address), and a node type (eg, leader node). In this example, the cluster is formed with five nodes.

Cluster mapping data may be determined and maintained by cluster discovery service 1062. As shown in circle A, cluster discovery service 1062 can monitor the location of cluster nodes for various volumes hosted by block storage server instances within a PSE. Cluster discovery service 1062 can monitor node locations in a variety of ways. For example, in an embodiment where the nodes run in environment 1090, cluster discovery service 1062 may periodically poll all of the block storage server instances 1050 hosted by PSE 1088 to obtain the identity of any resident node. . As another example, the PSE 1088's host network manager may be configured to route special broadcast messages to any of the hosted cluster nodes (e.g., hosted directly or indirectly by a block storage server instance, etc.). ). Cluster discovery service 1062 can periodically broadcast queries to obtain the identity of any of the hosted cluster nodes.

In some embodiments, cluster discovery service 1062 is an instance hosted by one of the hosts of PSE 1088. Such an instance may have a VNA with a reserved IP address in the BSS's IVN 1052 so that it can be reached even if it needs to change hosts due to hardware failure. In other embodiments, cluster discovery service 1062 can be integrated with the PSE's DNS service. For example, a volume cluster can be associated with a domain, and a DNS service can resolve name resolution requests for that name to the IP addresses of one or more nodes in the cluster.

As shown in circle B, the instance management service 1040 can send a message to a particular host's block storage client 1060 to attach a volume to a hosted instance (not shown). For example, instance management service 1040 can send a message to the host that includes the instance identifier and volume identifier for volume A. As shown in circle C, block storage client 1060 can query cluster discovery service 1062 to obtain the identity of one or more nodes in volume A's cluster 1066. In some embodiments, block storage client 1060 can cache cluster mapping data in cluster mapping data cache 1066. In some embodiments, the cluster discovery service 1062 may be omitted, and the block storage client 1060 queries the block storage server instances of the PSE 1088 (e.g., by the broadcast mechanism described above) and determines the cluster 1066 of the volume A. Note that the configuration is configured to identify nodes.

Block storage client 1060 obtains a current view of volume mapping data for volume A from cluster 1066, as shown in circle D, and based on the volume mapping data, hosts volume A, as shown in circle E. A block storage server 1050 can be connected to the block storage server 1050. Although not shown, in some embodiments, upon receiving a connection request from a client, the block storage server sends a message indicating whether the block storage server is still hosting the volume (or at least a portion of the volume). can be sent to the volume cluster. Despite receiving a connection request from a block storage client, the block storage server is unable to host the volume for various reasons. For example, recent changes to the set of servers hosting the volume may not yet have been propagated to or through the volume cluster, or the block storage client sending the connection request may have an old cache May depend on volume mapping data. Regardless of whether a block storage server hosts a volume, a block storage server that receives a connection request from a client can obtain the identity of one or more nodes of the volume from cluster discovery service 1062. If a block storage server no longer hosts a volume, the block storage server can suggest an update to the data store maintained by the cluster to remove the block storage server from the volume mapping data. Further, the block storage server may send a response to the block storage client that initiated the request indicating that the connection attempt failed and optionally indicating that the volume is no longer hosted by the server. If the block storage server is still hosting the volume, the block storage server can send an acknowledgment to the cluster to indicate to the cluster that at least that portion of the mapping data is still valid.

FIG. 11 is a block diagram illustrating an example system for tracking volume mapping, according to at least some embodiments. As explained above, connectivity between the PSE and the provider network cannot be guaranteed. To meet the desired level of data availability and data durability, the PSE includes facilities that allow the mapping between a given volume and the block storage server instance hosting that volume to be changed. As discussed with reference to FIG. 10, volumes for a given volume can be tracked using a cluster implementing a distributed data store. FIG. 11 illustrates a step-by-step approach to volume placement, or the process of selecting a block storage server instance to host a replica of a volume or a portion of a replica. Specifically, the BSS Volume Placement Service 1107 of the BSS 1106 makes initial placement decisions and associated volume mappings during volume creation, and the PSE Volume Placement Service 1108 manages subsequent changes to the volume mappings over the lifetime of the volume. do. PSE volume placement service 1108 can be implemented in a variety of ways, such as by an instance hosted by PSE 1188 integrated as a component of a PSE framework (eg, PSE framework 202).

As shown in circle A, PSE volume placement service 1108 monitors the status of block storage server instances hosted by PSE 1188. For example, the PSE volume placement service 1108 can periodically poll a hosted block storage server instance to check for a response and/or (e.g., as described with reference to FIG. 9). ) can collect metrics related to the resource usage of hosted block storage server instances, their failure domains, and/or hosts. As shown, PSE volume placement service 1108 can send collected server status to BSS volume placement service 1107, as shown in circle B. Note that in other embodiments, the BSS volume placement service 1107 may obtain metrics related to resource utilization from an instance management service (such as described with reference to FIG. 9).

Upon receiving a request to create a new block storage volume for an instance hosted by PSE 1188, BSS 1106 can request volume placement recommendations from BSS placement service 1107. Depending on the new volume's profile (eg, number of replicas, whether replicas as stripes, etc.), the BSS placement service 1107 can provide the identity of a recommended block storage server instance. In this example, BSS placement service 1107 recommends that block storage server instances 1150A and 1150B host new volume A with two replicas. Adopting the recommended block storage server instance, BSS 1106 creates a new cluster 1166 and tracks volume mapping data for volume A as shown in circle C. The mapping data initially identifies block storage server instance 1150A as hosting the primary replica block and storage server instance 1150B as hosting the secondary replica. Additionally, the BSS 1106 sends one or more messages to the identified block storage servers to identify the storage volumes (e.g., the storage volumes hosted by the block storage server instance 1150A and the block storage servers), as shown in circle D. Create a storage volume (hosted by instance 1150B). Storage volumes can be backed up by the capacity of the underlying host storage device provisioned to each block storage server instance. A storage volume created in block storage server instance 1150A can host a primary replica 1152 of volume A, and a storage volume created in block storage server instance 1150B can host a secondary replica 1154A of volume A. In some embodiments, a newly created storage volume can be loaded from a volume snapshot or machine image, such as that described with reference to FIG. In this example, an instance (not shown) attached to volume A performs block storage operations for some time (e.g., propagates writes from primary replica 1152 to secondary replica 1154B, as shown in circle E). block storage server instance 1150A communicates with block storage server instance 1150B.

At some point, block storage server instance 1150B may experience a problem, as shown in circle F. For example, block storage server instance 1150B may become delayed or unresponsive (eg, due to memory leaks, hardware failures, etc.). Detection of problems can occur in various ways. In some embodiments, block storage server instance 1150A detects problems due to propagated write confirmation failures and the like. In such a case, block storage server instance 1150A may include a policy that includes one or more actions to take in response to the detected problem. For example, block storage server instance 1150A may wait for a number of consecutive propagated writes to go unacknowledged, or may wait for a period of time. At that point, block storage server instance 1150A can request a replacement block storage server instance for secondary replica 1154 from PSE volume placement service 1108. In other embodiments, the PSE volume placement service 1108 detects problems during monitoring (e.g., based on collected metrics or responsiveness) as described above with reference to Circle A. Again, the PSE volume placement service 1108 may include a policy that includes one or more actions to take in response to a detected problem, including initiating a replacement of the block storage server instance of the secondary replica 1154. . Regardless of the detector, PSE volume placement service 1108 provides block storage server instance 1150A with the identity of a replacement block storage server instance.

In this example, PSE volume placement service 1108 identifies block storage server instance 1150C to block storage server instance 1150A, as shown in circle G. A message is sent to block storage server instance 1150C to create a storage volume to back up the replica data being relocated. For example, upon identifying block storage server instance 1150C, PSE volume placement service 1108 can send a message to block storage server instance 1150C to create a storage volume. As another example, block storage server instance 1150A can send a message to create a storage volume upon receiving identification information from PSE volume placement service 1108. Once created, block storage server instance 1150A can begin mirroring replica 1152 to block storage server instance 1150C as replica 1154B, as shown in circle H. Note that if block storage server instance 1150B is responsive (but exhibits poor performance, for example), a mirroring operation can be performed by copying replica 1154A to replica 1154B. Although remirroring from either replica 1154 or replica 1152 is possible in this scenario because those replicas are not distributed across storage servers, in other embodiments the replicas are distributed across various storage servers. It may be necessary to recreate or otherwise generate lost data by accessing and using redundancies etc. (e.g. parity bits, error correction codes, etc.) encoded in the stored data. . For example, if replica data is encoded and distributed across 10 block storage server instances and one of the 10 is lost, reading the data associated with the replica from the remaining 9 block storage server instances Lost data can be recreated. As another example, if another 10 block storage servers host another replica of a volume using the same distribution pattern, the lost data will be lost from the block storage server instance hosting the corresponding portion of the other replica. Can be copied.

As shown in circle I, block storage server instance 1150A submits a request to cluster 1166 to update the volume mapping for volume A, causing block storage server instance 1150B to host block storage server instance 1150C as the host of the secondary replica. can be exchanged for When block storage server instance 1150A initiates or completes a remirror to block storage server instance 1150C, it may submit a request to cluster 1166. In other embodiments, another entity (such as PSE volume placement service 1108 or block storage server instance 1150C) may submit a request to cluster 1166 to update the volume mapping for volume A.

FIG. 12 is a flow diagram illustrating the operation of a method for starting a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes or variations described herein, and/or combinations thereof) may be performed under the control of one or more computer systems comprised of executable instructions. , hardware, or a combination thereof, as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) of the operations are performed by a computer program or application executed by one or more components of the provider network extension. A provider network extension includes one or more physical computing devices or systems that are located remotely from the provider network's data center (eg, outside of the data center network), such as on the premises of a provider network customer. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data centers. One or more components of the extension communicate with the provider network, such as by receiving administrative operations from services executed by computer systems of the provider network. In some embodiments, one or more (or all) of the operations are performed by components of other illustrated hosts (eg, host 420A).

At block 1205, the operations include receiving, by a computer system of a provider network extension, a first request to launch a first virtual machine to host a block storage server application; communicates with the provider network via at least a third party network. When providing block storage devices to instances hosted by a PSE, the virtualization provided by the host of the PSE can be used to host the block storage server. For example, a provider network's block storage service can initiate the startup of a block storage server virtual machine by an instance management service, which then initiates the startup over a secure communication channel between the provider network and the PSE. A request can be issued to the host manager of the selected host in the PSE.

The operations further include, at block 1210, provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as a provisioned storage device. As part of starting a virtual machine, a host manager of the host system can allocate or provision a portion of the host system's computing resources to the virtual machine. Such resources may include, for example, storage capacity of a storage device (eg, SSD), memory capacity (eg, RAM), processor or processor core, etc. of the host system.

At block 1215, the operations further include running a block storage server application on the first virtual machine. As part of executing the block storage server application, at block 1220, the operations further include creating a logical volume on the provisioned storage device in response to a second request from a block storage service of a provider network, and creating a logical Including creating volumes. For example, a provider network's block storage service can send one or more messages to create a volume (using provisioned storage capacity) that can be attached to other instances so that the instance The volume can be accessed through.

As part of executing the block storage server application, at block 1225, the operation further includes receiving a third request to perform an I/O operation on the logical volume; This includes performing input/output operations. For example, referring to FIG. 4, instance A 430 may issue a command to read a block of data from a block address of a virtual block device attached to the instance and backed by block storage volume A 434. Block storage server instance 450 can receive the command (eg, via BSS's IVN 452) and process it against block storage volume A 434. For example, referring to FIG. 5, instance B 436 may issue a command to write a block of data to a block address of a virtual block device attached to the instance and backed by block storage volume B 440. Block storage server instance 450 can receive the command (eg, via BSS's IVN 452) and process it against block storage volume A 440.

FIG. 13 is a flow diagram illustrating the operation of a method for using a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes or variations described herein, and/or combinations thereof) may be performed under the control of one or more computer systems comprised of executable instructions. , hardware, or a combination thereof, as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) of the operations are performed by a computer program or application executed by one or more components of the provider network extension. A provider network extension includes one or more physical computing devices or systems that are located remotely from the provider network's data center (eg, outside of the data center network), such as on the premises of a provider network customer. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data centers. One or more components of the extension communicate with the provider network, such as by receiving administrative operations from services executed by computer systems of the provider network. In some embodiments, one or more (or all) of the operations are performed by a host in another figure (eg, host 420A).

At block 1305, operations include executing, by the computer system, a first block storage server virtual machine to host a first volume using one or more storage devices of the computer system. As shown in FIG. 4, for example, host 420A is hosting a virtual machine (ie, block storage server instance 450). At block 1310, the operations further include executing, by the computer system, a second virtual machine that has access to the virtual block storage device. Host 420A is also hosting another virtual machine (ie, instance 430). At block 1315, the operations further include executing a block storage client by the computer system. Host 420A includes a block storage client 460A, which can facilitate the attachment of block storage devices to hosted virtual machines. As part of executing the block storage client, at block 1320, the operation further includes receiving, from a second virtual machine, a first block storage operation to perform on the virtual block storage device; transmitting to cause the first block storage server virtual machine to perform a first block storage operation on the first volume. A virtual machine can issue block storage operations (eg, block reads, writes, burst reads, writes, etc.) that block attached storage devices via a block storage client. Block storage clients may, for example, relay their block storage operations across a network to block storage servers that host volumes containing block addresses.

FIG. 14 is a flow diagram illustrating the operation of a method for managing virtualized block storage servers in a provider board extension, according to at least some embodiments. Some or all of the operations (or other processes or variations described herein, and/or combinations thereof) may be performed under the control of one or more computer systems comprised of executable instructions. , hardware, or a combination thereof, as code (eg, executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program containing instructions executable by one or more processors. Computer-readable storage media are non-transitory. In some embodiments, one or more (or all) of the operations are performed by a computer program or application executed by one or more components of the provider network extension. A provider network extension includes one or more physical computing devices or systems that are located remotely from the provider network's data center (eg, outside of the data center network), such as on the premises of a provider network customer. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data centers. One or more components of the extension communicate with the provider network, such as by receiving administrative operations from services executed by computer systems of the provider network. In some embodiments, one or more (or all) of the operations are performed by components of PSEs in other figures (eg, PSE 1088, PSE 1188).

At block 1405, operations include receiving a first request by a first block storage server instance to create a first storage volume and storing a first portion of a first logical volume; At 1410, a second request is received by a second block storage server instance to create a second storage volume and store a second portion of the first logical volume. As shown in FIG. 11, for example, initial placement of volumes (eg, replicas, striped replicas, etc.) can originate from the provider network's block storage service 1106 and be received by components of the PSE 1188. In the example shown in FIG. 11, volume A is initially stored using block storage server instances 1150A and 1150C hosted on PSE 1188.

At block 1415, the operation further includes sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume. include. At some point, you may need to change the block storage server instance that hosts some of your volumes. As described herein, the PSE volume placement service 1108 or one of the other block storage server instances hosting the volume sends a message to another block storage server instance to replace the instance that is being modified. can.

At block 1420, the operations further include storing the second portion of the first logical volume in a third storage volume by a third block storage server instance. At block 1425, the operation further updates the data store containing the identification information of each block storage server instance that hosts the portion of the first logical volume and removes the identification information of the second block storage server instance. , including adding an identification of a third block storage server instance. To track migration of volumes across instances, PSE 1188 can host datastores that map volumes (including replicas, striping across servers, etc., if any). Such a data store may be a cluster as described with reference to FIG.

FIG. 15 illustrates an example provider network (or “service provider system”) environment, according to at least some embodiments. Provider network 1500 may provide resource virtualization to a customer via one or more virtualization services 1510, which may provide resource virtualization to a customer via one or more provider networks of one or more data centers or Enables purchasing, renting, or acquiring instances 1512 of virtualized resources (including, but not limited to, compute and storage resources) implemented on devices in multiple provider networks. A local Internet Protocol (IP) address 1516 may be associated with resource instance 1512. The local IP address is the internal network address of the resource instance 1512 of the provider network 1500. In some embodiments, provider network 1500 also provides a public IP address 1514 and/or range of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) that a customer may obtain from provider 1500. ) address).

Traditionally, the provider network 1500 is configured to provide, via a virtualization service 1510, a service provider's customers (e.g., customers operating one or more client networks 1550A-1550C that include one or more customer device(s) 1552). , may enable at least some public IP addresses 1514 assigned or assigned to a customer to be dynamically associated with a particular resource instance 1512 assigned to the customer. Provider network 1500 also allows a customer to transfer a public IP address 1514 previously mapped to one virtualized computing resource instance 1512 assigned to the customer to another virtualized computing resource instance 1512 also assigned to the customer. It may be possible to remap to instance 1512. Using virtualized computing resource instances 1512 and public IP addresses 1514 provided by the service provider, a customer of the service provider, such as an operator of customer network(s) 1550A-1550C, may implement customer-specific applications, for example. and may present customer applications on an intermediate network 1540, such as the Internet. Other network entities 1520 of intermediate network 1540 may then generate traffic to destination public IP addresses 1514 exposed by customer network(s) 1550A-1550C. Traffic is routed to the service provider data center, where it is routed via the network board to the local IP address 1516 of the virtualized computing resource instance 1512 that is currently mapped to the destination public IP address 1514. Similarly, response traffic from virtualized computing resource instance 1512 may be routed through the network substrate to intermediate network 1540 and back to source entity 1520.

As used herein, local IP address refers to the internal or "private" network address of a resource instance in a provider network, for example. The local IP address may be within the address block reserved by Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918, and/or may be of the address format specified by IETF RFC 4193, and may be of the address format specified by IETF RFC 4193, and May be changeable within the network. Network traffic originating from outside the provider network is not routed directly to local IP addresses. Instead, traffic uses a public IP address that maps to the resource instance's local IP address. A provider network may include network devices or appliances that provide network address translation (NAT) or similar functionality to map public IP addresses to local IP addresses (or vice versa).

A public IP address is an Internet variable network address assigned to a resource instance by either a service provider or a customer. Traffic routed to the public IP address is translated by, for example, 1:1 NAT and forwarded to the local IP address of each of the resource instances.

Several public IP addresses may be assigned to a particular resource instance by the provider network infrastructure. These public IP addresses may be referred to as standard public IP addresses or just standard IP addresses. In some embodiments, mapping a standard IP address to a resource instance's local IP address is the default startup configuration for all resource instance types.

At least some public IP addresses may be assigned to or obtained by customers of provider network 1500. The customer may then assign the assigned public IP address to a particular resource instance assigned to the customer. These public IP addresses may be referred to as customer public IP addresses or simply customer IP addresses. Instead of being assigned to a resource instance by the provider network 1500 as is the case with standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and can be remapped by each customer to other resource instances as needed or desired. A customer IP address is associated with the customer's account, rather than a particular resource instance, and controls the IP address until the customer chooses to release it. Unlike traditional static IP addresses, customer IP addresses allow a customer to mask the failure of a resource instance or Availability Zone by remapping the customer's public IP address to any resource instance associated with the customer's account. make it possible to For example, the customer IP address may be enabled to address problems with the customer's resource instance or software by remapping the customer IP address to a replacement resource instance.

FIG. 16 is a block diagram of an example provider network that provides storage services and hardware virtualization services to customers, according to at least some embodiments. Hardware virtualization service 1620 provides multiple computational resources 1624 (eg, VMs) to a customer. Computing resources 1624 may be rented or leased, for example, to a customer of provider network 1600 (eg, a customer implementing customer network 1650). Each of the computational resources 1624 may be provided with one or more local IP addresses. Provider network 1600 may be configured to route packets from a local IP address of computational resource 1624 to a public Internet destination and from a public Internet source to a local IP address of computational resource 1624 .

Provider network 1600 may, for example, connect virtual computing system 1692 to customer network 1650 which is coupled to intermediate network 1640 via local network 1656 and to virtual computing system 1692 via hardware virtualization service 1620 which is coupled to intermediate network 1640 and provider network 1600. can provide the ability to implement In some embodiments, the hardware virtualization service 1620 may provide one or more APIs 1602, e.g., a web services interface, through which the customer network 1650 can, e.g., connect the console 1694 (e.g., web-based applications, standalone applications, mobile applications, etc.) may access the functionality provided by hardware virtualization services 1620. In some embodiments, in provider network 1600, each virtual computing system 1692 in customer network 1650 may correspond to a computing resource 1624 that is leased, rented, or otherwise provided to customer network 1650.

From an instance of virtual computing system 1692 and/or another customer device 1690 (e.g., via console 1694), the customer can access the functionality of storage service 1610, e.g., via one or more APIs 1602, and Access data from and store data in storage resources 1618A-1618N of virtual data stores 1616 (e.g., folders or “buckets”, virtualized volumes, databases, etc.) provided by provider network 1600. obtain. In some embodiments, a virtualized data store gateway (not shown) may be provided to customer network 1650, where customer network 1650 stores at least some data, such as frequently accessed or critical data, locally. may communicate with storage service 1610 via one or more of a number of communication channels to upload new or modified data from the local cache so that the data's primary store (virtualized data store 1616) is maintained. In some embodiments, a user may mount and mount volumes of a virtual datastore 1616 via a virtual computing system 1692 and/or on another customer device 1690 via a storage service 1610 that functions as a storage virtualization service. These volumes may appear to the user as local (virtualized) storage 1698.

Although not shown in FIG. 16, virtualized service(s) may also be accessed from resource instances within provider network 1600 via API(s) 1602. For example, a customer, appliance service provider, or other entity may access virtualization services from within each virtual network in provider network 1600 via API 1602 to access one or more resources within the virtual network or within another virtual network. May request instance allocation.

FIG. 17 is a block diagram illustrating an example computing device that may be used in at least some embodiments. In at least some embodiments, such computer systems include control plane components and/or data plane components used to support the provider boards and/or PSEs described herein, various virtualization components (virtual machine, container, etc.) and/or a server implementing one or more SEDs. Such a computer system may include a general purpose or special purpose computer system that includes or is configured to access one or more computer-accessible media. In at least some embodiments, such computer systems may also be used to implement components external to the provider board and/or provider board extensions (e.g., customer gateway/router 186, other customer resources 187, etc.). . In the illustrated embodiment of the computer system, computer system 1700 includes one or more processors 1710 coupled to system memory 1720 through an input/output (I/O) interface 1730. Computing device 1700 further includes a network interface 1740 coupled to I/O interface 1730. FIG. 17 depicts computer system 1700 as a single computing device, and in various embodiments, computer system 1700 is configured to operate together as a single computing device or May include any number of computing devices.

In various embodiments, computer system 1700 is a uniprocessor system that includes one processor 1710 or a multiprocessor system that includes several processors 1710 (e.g., two, four, eight, or another suitable number). It can be a system. Processor 1710 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2010 is a general purpose processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. Or it can be an embedded processor. In a multiprocessor system, each of the processors 1710 may typically, but not necessarily, implement the same ISA.

System memory 1720 may store instructions and data that can be accessed by processor(s) 1710. In various embodiments, system memory 1720 is any of random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/flash type memory, or any other type of memory, etc. may be implemented using any suitable memory technology. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above, are stored in system memory 1720 as code 1725 and data 1726. It is shown that there is.

In one embodiment, I/O interface 1730 is configured to coordinate I/O traffic between processor 1710, system memory 1720, and any peripheral devices of the device, including network interface 1740 or other peripheral interfaces. obtain. In some embodiments, I/O interface 1730 performs any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., system memory 1720) to another component. (eg, processor 1710). In some embodiments, I/O interface 1730 provides support for devices attached by various types of peripheral buses, such as, for example, variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. may include. In some embodiments, the functionality of I/O interface 1730 may be divided into two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of I/O interface 1730, such as an interface to system memory 1720, may be integrated directly into processor 1710.

Network interface 1740 exchanges data between computer system 1700 and other devices 1760 (such as other computer systems or devices as shown in FIG. 1) that are attached to network or networks 1750. may be configured to allow. In various embodiments, network interface 1740 may support communication via any suitable wired or wireless common data network, such as, for example, a type of Ethernet network. Additionally, the network interface 1740 can be configured via a telecommunications/telephony network, such as an analog voice network or a digital fiber communications network, via a storage area network (SAN), such as a Fiber Channel SAN, or any other suitable type. Communication may be supported via networks and/or protocols.

In some embodiments, the computer system 1700 has an I/O interface 1730 (e.g., based on the Peripheral Component Interconnect Express (PCI-E) standard, or another such as the QuickPath Interconnect (QPI) or the UltraPath Interconnect (UPI)). one or more offload cards 1770 (including one or more processors 1775, and optionally including one or more network interfaces 1740) connected using a bus that implements a version of an interconnect. For example, in some embodiments, computer system 1700 may function as a host electronic device (e.g., operating as part of a hardware virtualization service) that hosts compute instances and one or more offload cards 1770 runs a virtualization manager that can manage Compute instances running on host electronic devices. By way of example, in some embodiments, offload card(s) 1770 can suspend and/or unpause a Compute instance, start and/or terminate a Compute instance, perform memory transfers/copies, You can perform instance management operations, etc. These management operations, in some embodiments, are performed by the offload card in coordination with the hypervisor (e.g., in response to a request from the hypervisor) performed by other processors 1710A-1710N of the hypervisor's computer system 1700. 1770. However, in some embodiments, the virtualization manager implemented by offload card(s) 1770 can service requests from other entities (e.g., from the Compute instance itself), and can serve requests from any other entity. Unable to coordinate with (or provide services to) the hypervisor. Referring to FIG. 2, in at least some embodiments, at least some of the functionality of the PSE framework 202 and host manager 222 is performed on one or more processors 1775 of the offload card 1770, while the instances (e.g. , 232, 234, 236) are executed on one or more processors 1710.

In some embodiments, computer system 1700 includes one or more storage devices (SD) 1780. Exemplary storage devices 1780 include solid state drives (eg, with various types of flash or other memory) and magnetic drives. Processor(s) 1710 may access SD 1780 via interface(s) 1730 or, optionally, via offload card(s) 1770. For example, offload card(s) 1770 may include a system-on-chip (SoC) that includes multiple interconnect interfaces and bridges interface 1730 and an interface to SD 1780 (e.g., performs a PCIe-to-PCIe bridge). ).

In some embodiments, system memory 1720 may be one embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and/or data may be received, transmitted, or stored on different types of computer-accessible media. Generally speaking, computer-accessible media refers to non-transitory storage media such as magnetic or optical media (e.g., disks or DVDs/CDs) that are coupled to computing device 1700 via I/O interface 1730 or may include a memory medium. Non-transitory computer-accessible storage media may also be included in some embodiments of computing device 1700 as system memory 1720 or another type of memory, such as RAM (e.g., SDRAM, double data rate (DDR), SDRAM, SRAM, etc.), read-only memory (ROM), and any other volatile or non-volatile media. Additionally, computer-accessible media can include transmission media or electrical, electromagnetic, or digital signals conveyed through a communication medium, such as a network and/or wireless link, such as may be implemented through network interface 1740. may include signals.

Various embodiments as described or suggested herein may be implemented in a variety of operating environments, including, in some cases, one or more user computers that can be used to run any number of applications; May include a computing device or a processing device. User devices or client devices include desktop or laptop computers running a standard operating system, as well as mobile devices running mobile phone software and capable of supporting several networking and messaging protocols. It may include any of a number of general purpose personal computers, such as wireless devices and handheld devices. Such systems may also include several workstations running any of a variety of commercially available operating systems as well as other known applications for purposes such as development and database management. These devices may also include other electronic devices such as dummy terminals, thin clients, gaming consoles, and other devices capable of communicating over a network.

Most embodiments support Transmission Control Protocol/Internet Protocol (TCP/IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extended Messaging and Presence Protocol (XMPP), AppleTalk, etc., to support communications using any of a variety of commercially available protocols, such as AppleTalk. The network(s) can be, for example, a local area network (LAN), wide area network (WAN), virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network. , and any combination thereof.

In embodiments that utilize a web server, the web server may be a variety of servers, including an HTTP server, a file transfer protocol (FTP) server, a common gateway interface (CGI) server, a data server, a Java server, a business application server, etc. can start either a server or a middle-tier application. The server(s) may also be written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, or TCL, and combinations thereof. It may be possible to execute a program or script in response to a request from a user device by executing one or more web applications, etc., which may be implemented as one or more scripts or programs written in . The server(s) may also include database servers including, but not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM®, and the like. A database server may be relational or non-relational (eg, "NoSQL"), distributed or non-distributed, etc.

The environment may include various data stores and other memory and storage media as described above. These may reside in various locations, such as on a storage medium that is local to (and/or resides on) one or more computers or remote from any or all of the computers across a network. In a particular set of embodiments, the information may reside within a storage area network (“SAN”) that is familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and/or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically coupled via a bus, the hardware elements including, for example, at least one central processing unit (“CPU ”), at least one input device (e.g., mouse, keyboard, controller, touch screen, or keypad), and/or at least one output device (e.g., display device, printer, or speakers). Such systems may also include one or more disk drives, optical storage devices, and solid state storage devices such as random access memory (RAM) or read only memory (ROM), as well as removable storage devices such as memory cards, flash cards, etc. May include storage devices.

Such devices may also include computer-readable storage media readers, communication devices (eg, modems, network cards (wireless or wired), infrared communication devices, etc.), and working memory as described above. A computer-readable storage medium reader is capable of connecting to or configured to receive a computer-readable storage medium and is capable of storing computer-readable information on a remote device, a local device, a fixed device, and/or a removable storage device, as well as temporarily storing computer-readable information. Refers to a storage medium for permanently and/or permanently containing, storing, transmitting, and retrieving information. The systems and various devices also generally include a number of software applications, modules, services, or other elements located within at least one working memory device and an operating system such as a client application or web browser. and application programs. It should be understood that alternative embodiments may have many variations from those described above. For example, customized hardware may also be used and/or certain elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices such as network input/output devices may be employed.

Storage media and computer-readable media for containing code or portions of code are defined in the art, including, but not limited to, storage media and communication media such as volatile and non-volatile media, removable and non-removable media. It may include any suitable medium known or in use for storing and/or transmitting information such as computer readable instructions, data structures, program modules, or other data. A method or technology implemented in RAM, ROM, electrically erasable programmable read-only memory ("EEPROM"), flash memory, or other memory technology, compact disk read-only memory ("CD-ROM"), digital Use disc (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device, or any other device that can be used to store desired information and that can be accessed by a system device. Including other media. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other means and/or ways to implement the various embodiments.

In the foregoing description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments may be practiced without these specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiments.

Herein, we describe the option of adding additional characteristics to some embodiments using bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dashed lines, and dotted lines). Explain the operation. However, such notation should not be construed to mean that these are the only options or optional operations and/or that blocks with solid borders are not optional in certain embodiments.

Reference numbers with a last letter (e.g., 101A, 102A, etc.) may be used to indicate that there may be one or more instances of the referenced entity in various embodiments, and multiple instances may be used. When present, each need not be identical, but instead may share some common characteristics or play a role in a common way. Furthermore, the particular suffix used is not meant to imply the presence of a particular amount of the entity, unless specifically indicated to the contrary. Thus, two entities that use the same or different suffixes may or may not have the same number of instances in various embodiments.

References to "one embodiment," "an embodiment," "exemplary embodiment," etc. refer to all embodiments, although the described embodiment may include the particular feature, structure, or feature. Indicates that it does not necessarily include a particular property, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Additionally, when a particular feature, structure, or feature is described in connection with an embodiment, such feature, structure, or feature may be discussed in connection with other embodiments, whether or not explicitly described. It is considered to be within the knowledge of those skilled in the art.

Furthermore, in the various embodiments described above, unless otherwise specified, disjunctive language such as the phrase "at least one of A, B, or C" refers to either A, B, or C, or It is intended to be understood to mean any combination thereof (eg, A, B, and/or C). Thus, such disjunctive language is not intended to imply that a given embodiment requires the presence of at least one of A, at least one of B, or at least one of C, respectively. Nor should we understand.

At least some of the embodiments of the disclosed technology can be explained in light of the following examples.
1. A computer-implemented method, comprising:
A block storage server virtual machine is executed by a first one or more processors of a first computer system of a provider network to perform first storage using one or more storage devices of said first computer system. hosting the volume; and
executing a customer virtual machine that has access to a virtual block storage device by the first one or more processors;
executing a block storage client by a second one or more processors of the first computer system;
The block storage client is
receiving from the customer virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the block storage server virtual machine to cause the block storage server virtual machine to perform the first block storage operation on a first storage volume;
The computer-implemented method.
2. 2. The computer-implemented method of clause 1, wherein the message is sent over the virtual network securely by encrypting and decrypting traffic sent over the virtual network using a key.
3. 3. The computer-implemented method of clause 2, wherein a first virtual network address of the virtual network is associated with the block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client.
4. A computer-implemented method, comprising:
executing a first block storage server virtual machine by a computer system to host a first volume using one or more storage devices of the computer system;
executing, by the computer system, a second virtual machine that has access to a virtual block storage device;
executing a block storage client by the computer system;
Running the block storage client includes:
receiving from the second virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the block storage operation on the first volume;
The computer-implemented method.
5. 5. The computer-implemented method of clause 4, wherein the message is sent over the virtual network securely by encrypting and decrypting traffic sent over the virtual network using a key.
6. The computer-implemented method of clause 5, wherein a first virtual network address of the virtual network is associated with the first block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client. Method.
7. The first block storage server virtual machine and the second virtual machine are executed by a first one or more processors of the computer system, and the block storage client is executed by a second one or more processors of the computer system. 7. A computer-implemented method according to any one of clauses 4 to 6, executed by a processor.
8. Clauses 4 to 4, wherein the first block storage server virtual machine is a first virtual machine hosted by the computer system, and the second virtual machine is a second virtual machine hosted by the computer system. 7. The computer-implemented method of any one of 7.
9. further comprising: executing a second block storage server virtual machine by the computer system to host a second volume using the one or more storage devices of the computer system; 5. The computer-implemented method of clause 4, wherein is a replica associated with the first volume.
10. The first block storage server virtual machine is executed using a first physical component of the computer system, and the second block storage server virtual machine is executed using a first physical component of the computer system that is different from the first physical component. 10. The computer-implemented method of clause 9, wherein the computer-implemented method is performed using two physical components.
11. The first block storage operation is writing a block of data, and the block storage client further encrypts the block of data using an encryption key associated with the virtual block storage device and writes a block of data. the message that generates an encrypted block and is sent to the first block storage server virtual machine includes the encrypted block of data; 11. The computer-implemented method of any one of clauses 4 to 10, wherein the first volume is written to.
12. the computer system is included in an extension of a provider network, the extension of the provider network communicating with the provider network via at least a third party network;
A computer according to any one of clauses 4 to 11, wherein the instance management service of the provider network initiates the execution of the first block storage server virtual machine and the second virtual machine by the computer system. Method of implementation.
13. one or more storage devices of the host computer system;
a first one or more processors of the host computer system executing a first block storage server application and a second application that have access to a virtual block storage device, the first block storage server application executing the first one or more processors, the first one or more processors comprising instructions for causing the first block storage server application to host a first volume using the one or more storage devices;
A first one or more processors of the host computer system running a block storage client application, the block storage client application including instructions that, when executed, cause the block storage client application to:
receiving from the second application a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server application to cause the first block storage server application to perform the first block storage operation on the first volume; one or more processors of 1;
system, including.
14. 14. The system of clause 13, wherein the message is sent over the virtual network securely by encrypting and decrypting traffic sent over the virtual network using a key.
15. 15. The system of clause 14, wherein a first virtual network address of the virtual network is associated with the first block storage server application and a second virtual network address of the virtual network is associated with the block storage application.
16. The first block storage server application executes within a first virtual machine hosted by the host computer system, and the second application executes within a second virtual machine hosted by the host computer system. The system according to any one of Articles 13 to 15,
17. The first one or more processors of the host computer system further execute a second block storage server application, the second block storage server application, when executed, controlling the second block storage server application. Clause 13, wherein the application includes instructions for using the one or more storage devices of the host computer system to host a second volume, the second volume being a replica associated with the first volume. system described in.
18. The first block storage server application is executed using a first physical component of the computer system, and the second block storage server application is executed using a second physical component of the computer system that is different from the first physical component. 18. The system of clause 17, implemented using physical components.
19. The first block storage operation is writing a block of data, and the block storage client application further causes the block storage client application, at runtime, to write a block of data using an encryption key associated with the virtual block storage device. , the message sent to the first block storage server application includes instructions for encrypting the block of data to generate an encrypted block of data, and the message sent to the first block storage server application includes the encrypted block of data, 19. The system of any one of clauses 13-18, causing a block storage server application to write an encrypted block of the data to the first volume.
20. the host computer system is included in an extension of a provider network, the extension of the provider network communicating with the provider network via at least a third party network;
20. The system of any one of clauses 13-19, wherein the provider network instance management service initiates the execution of the first block storage server application and the second application by the host computer system.
21. A computer-implemented method, comprising:
At a host computer system of an extension of a provider network, receiving a first request to launch a first virtual machine to host a block storage server application, the extension of the provider network performing at least one of the following steps: said hosting in communication with said provider network via a third party network;
provisioning at least a portion of the storage capacity of one or more physical storage devices of the host computer system to the first virtual machine as a provisioned storage device;
executing the block storage server application on the first virtual machine;
Running the block storage server application includes:
creating the logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume;
receiving a third request from a block storage client application via a virtual network to perform an I/O operation on the logical volume;
performing the requested input/output operation on the logical volume;
The computer-implemented method.
22. Before running the block storage server application on the first virtual machine, the method further includes:
executing another application on the first virtual machine to load a boot volume from a machine image obtained from a data store of the provider network to a boot volume, the boot volume being a part of the provisioned storage; loading said device to be another logical volume of the device;
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
The computer-implemented method of clause 21, comprising:
23. Before running the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device. 22. The computer-implemented method of clause 21, wherein the boot volume is another logical volume of the provisioned storage device and the first block storage device is a virtual block storage device.
24. A computer-implemented method, comprising:
receiving, by a computer system of an extension of a provider network, a first request to launch a first virtual machine to host a block storage server application; said hosting in communication with said provider network via a party network;
provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as a provisioned storage device;
executing the block storage server application on the first virtual machine;
Running the block storage server application includes:
creating the logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume;
receiving a third request to perform an input/output operation on the logical volume;
performing the requested input/output operation on the logical volume;
The computer-implemented method.
25. Before running the block storage server application on the first virtual machine, the method further includes:
executing another application on the first virtual machine to load a boot volume from a machine image obtained from a data store of the provider network, the boot volume being separate from the provisioned storage device; loading the logical volume of
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
25. The computer-implemented method of clause 24, comprising:
26. 26. The computer-implemented method of clause 25, further comprising clearing memory of the first virtual machine before running the block storage server application.
27. Before running the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device. 25. The computer-implemented method of clause 24, wherein the boot volume is another logical volume of the provisioned storage device and the first block storage device is a virtual block storage device.
28. 28. The computer-implemented method of clause 27, wherein the loading is performed by at least one of a basic input/output system of the first virtual machine and an integrated extensible firmware interface of the first virtual machine.
29. the third request is received from a block storage client application over a virtual network;
29. The computer-implemented method of any one of clauses 24-28, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.
30. the first virtual machine is one of a plurality of virtual machines, each of the plurality of virtual machines running a block storage server application, the method further comprising:
determining that resource utilization associated with one or more of the plurality of virtual machines exceeds a threshold;
provisioning at least a portion of the storage capacity of one or more storage devices of another host computer system to a second virtual machine;
running another block storage server application on the second virtual machine;
The computer-implemented method of any one of clauses 24-29, comprising:
31. provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system as another provisioned storage device to a second virtual machine;
executing another block storage server application on the second virtual machine, the first virtual machine being executed using a first physical component of the computer system; is performed using a second physical component of the computer system that is different from the first physical component;
31. The computer-implemented method of any one of clauses 24-30, further comprising:
32. The input/output operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested input/output on the logical volume means writing the block of data to a block address of the logical volume. writing to the block address of the provider network, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network writes to the block address of the provider network. 32. The computer-implemented method of clause 31, comprising one or more physical computing devices located outside a data center of a provider network and on the premises of a customer of the provider network.
33. one or more storage devices of a host computer system of an extension of a provider network, wherein the extension of the provider network communicates with the provider network via at least a third party network; ,
one or more processors of the host computer system for executing the host manager application, the host manager application including instructions that, when executed, cause the host manager application to:
receiving a first request to launch a first virtual machine to host a block storage server application;
provisioning at least a portion of the storage capacity of the one or more storage devices to the first virtual machine as a provisioned storage device;
executing the block storage server application on the first virtual machine, the block storage server application including instructions that, when executed, cause the block storage server application to:
creating the logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume;
receiving a third request to perform an input/output operation on the logical volume;
performing the requested input/output operation on the logical volume;
system, including.
34. The host manager application includes further instructions, the further instructions causing the host manager application, during execution, to:
executing another application on the first virtual machine to load a boot volume from a machine image obtained from a data store of the provider network, the boot volume being separate from the provisioned storage device; loading the logical volume of
modifying the first virtual machine to boot from the boot volume;
restarting the first virtual machine;
The system described in Article 33, which causes the system to perform.
35. 35. The host manager application, upon execution, includes further instructions causing the host manager application to clear memory of the first virtual machine before executing the block storage server application. system.
36. A component of the first virtual machine, at runtime, causes the component to have a boot volume of the first virtual machine stored in the first virtual machine before running the block storage server application on the first virtual machine. instructions for loading from a machine image stored by a block storage device, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device; The system described in 33.
37. 37. The system of clause 36, wherein the component is at least one of a basic input/output system of the first virtual machine and an integrated extensible firmware interface of the first virtual machine.
38. the third request is received from a block storage client application over a virtual network;
38. The system of any one of clauses 33-37, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.
39. The host manager application includes further instructions, which upon execution cause the host manager application to:
provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system as another provisioned storage device to a second virtual machine;
executing another block storage server application on the second virtual machine, the first virtual machine executing using a first physical component of the computer system; the performing is performed using a second physical component of the computer system that is different from the first physical component;
The system according to any one of Articles 33 to 38, which causes the system to perform.
40. The input/output operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested input/output on the logical volume means writing the block of data to a block address of the logical volume. writing to the block address of the provider network, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network writes to the block address of the provider network. 40. The system of clause 39, comprising one or more physical computing devices located outside a data center of a provider network and on the premises of a customer of said provider network.
41. A computer-implemented method, comprising:
receiving, by a first block storage server instance, a first request from a block storage service of a provider network, the first request to create a first storage volume and to create a first logical volume; said receiving, storing a first part of said first part;
receiving, by a second block storage server instance, a second request from the block storage service, the second request to create a second storage volume and to create a second storage volume of the first logical volume; said receiving and storing the second part;
determining that the second block storage server instance is unresponsive;
sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume;
storing the second portion of the first logical volume in the third storage volume by the third block storage server instance;
updating a data store containing the identity of each block storage server instance that hosts a portion of the first logical volume to remove the identity of the second block storage server instance; adding identification information for the server instance;
the first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems of an extension of the provider network; The computer-implemented method, wherein the extension communicates with the provider network via at least a third party network.
42. receiving, by a block storage client, the identification information of each block storage server instance hosting a portion of the first logical volume from the data store;
establishing a connection by the block storage client to at least one block storage server instance included in the identification information;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; and,
sending a message by the block storage client to the at least one block storage server instance via the connection to send the at least one block storage server instance the block of data to the block of the first storage volume; to write to the address, and
42. The computer-implemented method of clause 41, further comprising:
43. 43. The computer-implemented method of any one of clauses 41-42, further comprising polling the first block storage server instance and the second block storage server instance for responses.
44. A computer-implemented method, comprising:
receiving, by a first block storage server instance, a first request to create a first storage volume and store a first portion of the first logical volume;
receiving a second request to create a second storage volume and store a second portion of the first logical volume by a second block storage server instance;
sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume;
storing the second portion of the first logical volume in the third storage volume by the third block storage server instance;
updating a data store containing the identity of each block storage server instance that hosts a portion of the first logical volume to remove the identity of the second block storage server instance; Adding identification information for the server instance;
The computer-implemented method.
45. receiving, by a block storage client, the identification information of each block storage server instance hosting a portion of the first logical volume from the data store;
establishing a connection by the block storage client to at least one block storage server instance included in the identification information;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; and,
sending a message by the block storage client to the at least one block storage server instance via the connection to send the at least one block storage server instance the block of data to the block of the first storage volume; to write to the address, and
45. The computer-implemented method of clause 44, further comprising:
46. 46. The computer-implemented method of clause 45, wherein the data store is a distributed data store that includes multiple nodes that independently execute a consensus protocol to update the identification information.
47. 47. The computer-implemented method of clause 46, further comprising obtaining, by the block storage client, an identification of each node of the plurality of nodes from a service.
48. 47. The computer-implemented method of clause 46, wherein one node of the plurality of nodes executes within a container hosted by the first block storage server instance.
49. 49. The computer-implemented method of clause 48, further comprising broadcasting, by the block storage client, a request for identification information of any node of the plurality of nodes to a plurality of block storage server instances.
50. further comprising polling the first block storage server instance and the second block storage server instance for responses, the third request being responsive to a determination that the second block storage server instance is not responsive. 49. The computer-implemented method according to any one of clauses 44 to 49, wherein the computer-implemented method is transmitted as:
51. 51. The computer-implemented method of any one of clauses 44-50, wherein the first storage volume is striped across a plurality of block storage server instances, each of the plurality of block storage server instances being included in the identification information.
52. the first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems of an extension of a provider network; and said second request is sent from a block storage service of said provider network, and said extension of said provider network communicates with said provider network via at least a third party network. The computer implementation method described in .
53. one or more computing devices of an extension of a provider network, the extension of the provider network communicating with the provider network via at least a third party network; The device includes instructions that, when executed on one or more processors, cause the one or more computing devices to:
receiving, by a first block storage server instance, a first request to create a first storage volume and store a first portion of the first logical volume;
receiving a second request to create a second storage volume and store a second portion of the first logical volume by a second block storage server instance;
sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume;
storing the second portion of the first logical volume in the third storage volume by the third block storage server instance;
updating a data store containing the identity of each block storage server instance that hosts a portion of the first logical volume to remove the identity of the second block storage server instance; Adding identification information for the server instance;
system, including.
54. The one or more computing devices include further instructions, and the further instructions, when executed on the one or more processors, cause the one or more computing devices to:
receiving, by a block storage client, the identification information of each block storage server instance hosting a portion of the first logical volume from the data store;
establishing a connection by the block storage client to at least one block storage server instance included in the identification information;
receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; and,
sending a message by the block storage client to the at least one block storage server instance via the connection to send the at least one block storage server instance the block of data to the block of the first storage volume; to write to the address, and
The system according to clause 53, which causes
55. 55. The system of clause 54, wherein the data store is a distributed data store that includes multiple nodes that independently execute a consensus protocol to update the identification information.
56. The one or more computing devices, when executed on the one or more processors, cause the one or more computing devices to obtain identification information for each node of the plurality of nodes from a service by the block storage client. 56. The system according to clause 55, comprising further instructions to cause the system to:
57. 56. The system of clause 55, wherein one node of the plurality of nodes runs within a container hosted by the first block storage server instance.
58. The one or more computing devices, when executed on the one or more processors, request the one or more computing devices, by the block storage client, for identification information of any node of the plurality of nodes. 58. The system of clause 57, comprising further instructions for causing the block storage server instances to broadcast the block storage server instance to the plurality of block storage server instances.
59. The one or more computing devices transmit responses of the first block storage server instance and the second block storage server instance to the one or more computing devices when executed on the one or more processors. 59. The system of any one of clauses 53-58, including further instructions to poll, wherein the third request is sent in response to a determination that the second block storage server instance is not responsive.
60. 60. The system of any one of clauses 53-59, wherein the first storage volume is striped across a plurality of block storage server instances, each of the plurality of block storage server instances being included in the identification information.

Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be evident, however, that various modifications and changes may be made herein without departing from the broader spirit and scope of the disclosure as set forth in the claims.

Claims (14)

A computer-implemented method, comprising:
executing a first block storage server virtual machine by a computer system to host a first volume using one or more storage devices of the computer system;
executing, by the computer system, a second virtual machine that has access to a virtual block storage device;
executing a block storage client by the computer system;
Running the block storage client includes:
receiving from the second virtual machine a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the first block storage operation on the first volume;
the computer system is included in an extension of a provider network, the extension of the provider network communicating with the provider network via at least a third party network;
the provider network instance management service initiates the execution of the first block storage server virtual machine and the second virtual machine by the computer system;
Computer-implemented method. 2. The computer-implemented method of claim 1, wherein the message is sent over the virtual network securely by encrypting and decrypting traffic sent over the virtual network using a key. 3. The computer of claim 2, wherein a first virtual network address of the virtual network is associated with the first block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client. Method of implementation. The first block storage server virtual machine and the second virtual machine are executed by a first one or more processors of the computer system, and the block storage client is executed by a second one or more processors of the computer system. A computer-implemented method according to any one of claims 1 to 3, executed by a processor. 2. The first block storage server virtual machine is a first virtual machine hosted by the computer system, and the second virtual machine is a second virtual machine hosted by the computer system. 4. The computer-implemented method according to any one of items 4 to 4. further comprising: executing a second block storage server virtual machine by the computer system to host a second volume using the one or more storage devices of the computer system; The computer-implemented method of claim 1, wherein is a replica associated with the first volume. The first block storage server virtual machine is executed using a first physical component of the computer system, and the second block storage server virtual machine is executed using a first physical component of the computer system that is different from the first physical component. 7. The computer-implemented method of claim 6, wherein the computer-implemented method is performed using two physical components. The first block storage operation is writing a block of data, and the block storage client further encrypts the block of data using an encryption key associated with the virtual block storage device and writes a block of data. the message that generates an encrypted block and is sent to the first block storage server virtual machine includes the encrypted block of data; A computer-implemented method according to any one of claims 1 to 7, wherein the first volume is written to. one or more storage devices of the host computer system;
a first one or more processors of the host computer system executing a first block storage server application and a second application that have access to a virtual block storage device, the first block storage server application executing the first one or more processors, the first one or more processors comprising instructions for causing the first block storage server application to host a first volume using the one or more storage devices;
a first one or more processors of the host computer system running a block storage client application;
The block storage client application includes instructions;
The instructions, when executed, cause the block storage client application to:
receiving from the second application a first block storage operation to perform on the virtual block storage device;
sending a message to the first block storage server application to cause the first block storage server application to perform the first block storage operation on the first volume;
the host computer system is included in an extension of a provider network, the extension of the provider network communicating with the provider network via at least a third party network;
The system wherein the instance management service of the provider network initiates the execution of the first block storage server application and the second application by the host computer system. 10. The system of claim 9, wherein the message is sent over the virtual network securely using a key to encrypt and decrypt traffic sent over the virtual network. The system of claim 10, wherein a first virtual network address of the virtual network is associated with the first block storage server application and a second virtual network address of the virtual network is associated with the block storage client application. . The first block storage server application executes within a first virtual machine hosted by the host computer system, and the second application executes within a second virtual machine hosted by the host computer system. The system according to any one of claims 9 to 11. The first one or more processors of the host computer system further execute a second block storage server application, the second block storage server application, when executed, controlling the second block storage server application. 10. An application comprising instructions for causing a second volume to be hosted using the one or more storage devices of the host computer system, the second volume being a replica associated with the first volume. 9. The system described in 9. The first block storage server application is executed using a first physical component of the host computer system, and the second block storage server application is executed using a first physical component of the host computer system that is different from the first physical component. 14. The system of claim 13, implemented using two physical components.