TWI805380B - Method for performing simple storage service seamless migration using index objects, host device, and storage server - Google Patents

Abstract

A method for performing Simple Storage Service (S3) seamless migration using index objects and associated apparatus are provided. The method includes: in response to a request of migrating user data of a user of the storage server from a remote S3-compatible server into the storage server, during an index stage, utilizing an index-object-based S3 migration management module among multiple program modules running on a host device within the storage server to create and store multiple index objects into a storage device layer of the storage server, and migrate respective Access Control Lists (ACLs) of the multiple normal objects to the storage server; and during a data stage, utilizing the index-object-based S3 migration management module to trigger one or more migration agents to migrate object data.

Description

Method, main device and storage server for seamless migration of simple storage service by using index object

The present invention relates to memory control, and more particularly to a method and an apparatus for seamlessly migrating a simple storage service (S3) using an index object, wherein an example of the apparatus may include: an entire storage server , a main device in the storage server, a processing circuit in the main device, and at least one processor/processor core (for example: a central processing unit (central processing unit)) running in the processing circuit corresponding to one or more program modules of the method processing unit, CPU)/central processing unit core), but the present invention is not limited thereto.

The server can be used for cloud computing (cloud computing), for example, the storage server can be used to implement remote storage (eg: the cloud that can store data for users). When the user who uses the old public cloud (public cloud) finds that the public cloud is almost full or the access speed is very slow, the user may need to find a new cloud. A data migration method has been proposed in the prior art to try Copying data from an old public cloud to another public cloud, however, some problems may arise, for example, this method of data migration usually first copies data, including all data in the old public cloud before the start time point, and then performs port Point switching, etc., so that data that should be deleted (for example, data that has been deleted by the user after the start time point) may be mistakenly copied to another public cloud. In addition, according to the suggestion of this data migration method, it may take a long time (such as several months) to copy the data first. In particular, this data migration method is not suitable for private cloud (private cloud). cloud, it seems that there is no suitable solution for migrating data from the old cloud to the private cloud in related technologies. Therefore, a novel architecture is needed to perform seamless migration of simple storage services for Easy storage service seamless migration from any public/private cloud to a private cloud) to allow users to access data normally and smoothly during the data migration process.

Therefore, one of the objectives of the present invention is to disclose a method for seamlessly migrating simple storage services using index objects, and to disclose related devices (such as storage servers, master devices in storage servers, etc.) to solve the above problems .

Another object of the present invention is to disclose a method for seamlessly migrating simple storage services by using index objects, and to disclose related devices (such as storage servers, master devices in storage servers, etc.), so as not to introduce side effects Optimal performance under certain circumstances or in a way that is less likely to introduce side effects.

At least one embodiment of the present invention discloses a method for seamless migration of simple storage services using index objects, wherein the method is applied to a storage server, and the method may include: A request by a (S3-compatible) server to migrate user data of a user of the storage server to the storage server, at an index stage of one of stages of a migration of the user data In, using an index-object-based simple storage service migration management module among a plurality of program modules running on a main device in the storage server to create and store a plurality of index objects to a storage layer of the storage server as respective representations of normal objects of the user data in the storage server, and migrating respective accesses of the normal objects access control lists (ACLs) to the storage server as respective ACLs for the plurality of indexed objects to minimize any change in any one of the plurality of normal objects affecting the user Probability of correctness of the migration of data, wherein the storage server includes the main device and the storage device layer, the storage device layer includes at least one storage device coupled to the main device, the main device is used for controlling the operation of the storage server, and the at least one storage device is used to store information for the storage server; and in a data stage following the indexing stage of the plurality of stages, using the The Simple Storage Service migration management module for indexing objects to trigger one or more migration agents (migration agents) to migrate each of a group of normal objects among the plurality of normal objects from the remote Simple Storage Service compatible server The object data of the user data is transferred to the storage device layer of the storage server as respective replacements of a group of index objects in the plurality of index objects to complete the migration of the user data.

In addition to the above-mentioned method, the present invention also discloses a host device, which may include a processing circuit for controlling the host device to perform simple storage service migration management based on index objects in a storage server, wherein the storage The server includes the main device and a storage device layer, the storage device layer includes at least one storage device coupled to the main device, the main device is used to control the operation of the storage server, and the at least one storage The device is used to store information for the storage server, for example, in response to a request to migrate a user profile of a user of the storage server from a remote Simple Storage Service compatible server to the storage server, in During an indexing phase of one of the phases of migration of the user data, an index object-based simple storage service migration management module among the program modules running on the processing circuit creates and stores multiple indexes objects to the storage device layer as respective representations of the plurality of normal objects of the user data in the storage server, and migrate the respective access control lists of the plurality of normal objects to the storage server , as the respective access control lists of the plurality of index objects, to minimize the chance that any change in any normal object of the plurality of normal objects affects the correctness of the migration of the user data; and in the In a data stage following the indexing stage of the plurality of stages, the index object-based SIMS migration management module triggers one or more migration agents to migrate the Respective object data of a group of normal objects among the plurality of normal objects to the storage device layer of the storage server as a respective replacement of a group of index objects among the plurality of index objects to complete the user The transfer of data.

In addition to the above method, the present invention also discloses a storage server, the storage server may include a master device and a storage device layer, wherein the master device is used to control the operation of the storage server, for example, the master device A processing circuit may be included for controlling the host device to perform index object-based simple storage service migration management in the storage server. In addition, the storage device layer may include at least one storage device for storing information for the storage server, for example, a user in response to migrating a user of the storage server from a remote Simple Storage Service compatible server A request for data to the storage server, during an indexing phase of one of the phases of migration of the user data, an indexed object-based simple storage in program modules running on the processing circuit The service migration management module creates and stores a plurality of index objects to the storage device layer as respective representations of the plurality of normal objects of the user data in the storage server, and the respective representations of the plurality of normal objects The access control list of the plurality of normal objects is migrated to the storage server as the respective access control lists of the plurality of index objects, so as to minimize any change of any normal object in the plurality of normal objects affecting the user data a probability of correctness of the migration; and in a data phase after the index phase of the plurality of phases, the index object-based storage services migration management module triggers one or more migration agents from the Remote Simple Storage Service compliant server migrating respective object data of a group of normal objects among the plurality of normal objects to the storage layer of the storage server as a set of indexes among the plurality of index objects The respective replacement of the objects to complete the migration of the user data.

The method and related devices of the present invention can use index objects to perform seamless migration of simple storage services. Seamless migration can be performed with less overhead in a data phase/epoch, where the overhead of the data phase/epoch is usually smaller than that of the index phase/epoch.

The embodiment of the present invention discloses a method and device for seamless migration of simple storage services by using index objects. For example, the method and related devices can use a novel seamless migration architecture of simple storage services for access control, and In particular, seamless migration of simple storage services for private clouds, such as from any public/private cloud to a private cloud (such as a destination cloud for data migration), can be performed to allow the use of Ordinary and smooth access to data during the data migration process. For better understanding, a first cloud, such as Simple Storage Service compliant storage (for example, it may be almost full and access speed may be slow), and a second cloud, such as Bigtera storage implemented according to the method , can be used as an example of any public/private cloud and the private cloud (such as the target cloud), respectively, but the present invention is not limited thereto. In addition, when the migration starts at the first time point T1, the method and related devices may be preceded by an index phase starting from the first time point T1, such as the INDEX phase (for example, an index period, such as the INDEX period). Create an index object (such as a virtual index object (dummy index object)) in the second cloud, and can start a data phase from the second time point T2 (such as the time point when the index object is created), such as the DATA phase ( For example a data period, such as a DATA period), object data is migrated (eg transferred) from the first cloud to the second cloud. In some embodiments, the simple storage service seamless migration architecture can utilize one or more migration agent programs (agent) to perform object migration, and in addition, the device can include at least a part (for example, a part or all) of the storage server, For example, the apparatus may comprise a portion of a storage server, such as a host device within the storage server or associated control circuitry within the host device (eg, processing circuitry running on one or more program modules corresponding to the method and associated circuitry), in another example, the device may contain the entire storage server, according to some embodiments for storage servers, the INDEX period may represent a short period of time (such as hours or days), and the DATA An epoch may represent a long period of time (eg, several weeks).

1 is a schematic diagram of a storage server 10 according to an embodiment of the present invention, the storage server 10 includes a main device 50, and includes at least one storage device, such as one or more storage devices (such as a plurality of storage devices 90 ), the plurality of storage devices 90 are coupled to the main device 50, according to the present embodiment, the main device 50 can be used to control the operation of the storage server 10, and the plurality of storage devices 90 can be used to store information for the storage server 10. As shown in FIG. 1, the main device 50 may include a processing circuit 52 (for example, at least one processor/processor core and related circuits such as random access memory (random access memory, RAM) for controlling the operation of the main device 50. , bus bars, etc.), used to couple a plurality of storage devices 90 and couple storage or memory devices located on the main device 50 (such as one or more traditional hard disk drives (hard disk drive, HDD) and/or At least one storage interface circuit 54 of one or more solid state drives (SSD), and a network interface circuit 58 for coupling the host device 50 to at least one network. The storage or memory device may include at least one storage device (eg, one or more storage devices), which may be collectively referred to as storage device 56, for example, storage device 56 may include a set of storage devices, wherein one of the set of storage devices stores The device can be used as a system disk of the main device 50, and other storage devices of the set of storage devices can be used to store user data for the main device 50, but the present invention is not limited thereto. In another example, the storage device 56 may include a storage device, and this storage device may be used as a system disk of the main device 50 .

According to this embodiment, the processing circuit 52 of the running program module 52P (especially the index object-based simple storage service migration management module 53 corresponding to the method) can control the operation of the main device 50 according to the method, for example, control The main device 50 can perform simple storage service migration management based on index objects in the storage server 10, and the storage interface circuit 54 can conform to one or more standards (such as one or more Serial Advanced Technology Attachment, Serial ATA or SATA) standard, Peripheral Component Interconnect (PCI) standard, Peripheral Component Interconnect Express (PCIe) standard, Non-Volatile Memory Storage Device (Non-Volatile Memory Express, NVMe ) standard, non-volatile memory storage device external access (NVMe-over-Fabrics, NVMeoF) standard, small computer system interface (Small Computer System Interface, SCSI) standard, universal flash memory storage (Universal Flash Storage, UFS ) standard, etc.), and can communicate according to one or more standards to allow the processing circuit 52 running the program module 52P to access the storage device 56 and the plurality of storage devices 90 through the storage interface circuit 54. In addition, the network The road interface circuit 58 can be used to provide a wired or wireless network connection, and one or more client devices corresponding to one or more users can be connected to the storage server 10 (such as the storage device therein) through the wired or wireless network. device 56 and a plurality of storage devices 90) to access (for example, read or write) user data. For a better understanding, the main device 50 shown in the left half of FIG. 1 and the related circuits/modules/devices in this architecture (such as the processing circuit 52 and the storage interface circuit 54 running the program module 52P) can be respectively As an example of the above-mentioned main device and related circuits/modules/devices therein (for example, a processing circuit and related circuits corresponding to the method and running one or more program modules).

In the architecture shown in FIG. 1, the storage server 10 can be shown as including a main device 50 and a plurality of storage devices 90 coupled to the main device 50, but the present invention is not limited thereto. For example, the main device 50 It may additionally include a shell/casing (such as a computer casing, which may be composed of metal and/or one or more other materials) and at least a part (such as a part or all) of a plurality of storage devices 90, and for another example, the storage server 10 may further include a plurality of storage devices coupled to the main device 50 and At least one switching circuit (switch circuit) (such as one or more switching circuits) between at least a part (such as a part or all) of the device 90 is used to switch between the main device 50 and at least a part of the above-mentioned plurality of storage devices 90 Between the signal switching (signal switching).

According to some embodiments, the processing circuit 52 or the storage interface circuit 54 of the running program module 52P can configure at least a part (for example, a part or all) of the plurality of storage devices 90 to form a storage pool (storage pool) architecture, But the present invention is not limited thereto. According to some embodiments, the processing circuit 52 or the storage interface circuit 54 of the running program module 52P can set at least a part (for example, a part or all) of a plurality of storage devices 90 to form a storage server 10 fault-tolerant disk array (Redundant Array of Independent Disks, RAID) (such as All Flash Array (All Flash Array, AFA)).

FIG. 2 is a schematic diagram of some implementation details of the storage device in the storage server 10 shown in FIG. 1 according to an embodiment of the present invention, wherein the storage device 100 can be used as a storage device of any one of the plurality of storage devices 90 For example, in particular, each storage device of the plurality of storage devices 90 may be implemented according to the architecture of the storage device 100, but the present invention is not limited thereto. The storage device 100 can be used to provide storage space to the main device 50. Under the control of the main device 50, one or more client devices can access (for example, read or write) user data in the storage space. The main device 50 Examples of the storage device 100 may include but not limited to: personal computers (such as desktop computers or notebook computers), examples of the storage device 100 may include but not limited to: solid state drives and various types of embedded memory devices (such as flash memory storage or embedded multi media card (embedded multi media card, EMMC) standard, etc.). According to this embodiment, the storage device 100 may include a controller (such as a memory controller 110), and further includes a non-volatile memory 120, wherein the controller is used to control the operation of the storage device 100 and access non-volatile memory. The volatile memory 120 and the non-volatile memory 120 are used to store information. The non-volatile memory 120 may include at least one non-volatile memory element (such as one or more non-volatile memory elements), Such as a plurality of non-volatile memory elements 120-1, 120-2, ..., and 122-N, wherein "N" can represent a positive integer greater than one, for example, the non-volatile memory 120 can be a flash memory Body, and the plurality of non-volatile memory elements 120-1, 120-2, ..., and 122-N can be a plurality of flash memory chips (chip) or a plurality of flash memory bare crystals (die), But the present invention is not limited thereto.

According to this embodiment, the memory controller 110 can be used to control the access of the non-volatile memory 120, so that the host device 50 can access the non-volatile memory 120 through the memory controller 110, so as to store the non-volatile memory 120 in the storage server 10 access management, as shown in Figure 2, the memory controller 110 may include a processing circuit (such as a microprocessor 112), a storage unit (such as a read-only memory (read-only memory, ROM) 112M) , a control logic circuit 114 , a random access memory 116 and a transmission interface circuit 118 , wherein the above components can be coupled to each other through a bus bar. The random access memory 116 is implemented by a static random access memory (static RAM, SRAM), but the present invention is not limited thereto, the random access memory 116 can be used to provide internal storage space to the memory controller 110. For example, the random access memory 116 can be used as a buffer memory to buffer data. In addition, the read-only memory 112M of this embodiment is used to store the program code 112C, and the microprocessor 112 is used to execute The program code 112C controls the operation of the memory controller 110 to control the access of the non-volatile memory 120 to allow the host device 50 to access the non-volatile memory 120 through the memory controller 110 . It should be noted that, in some examples, program code 112C may be stored in random access memory 116 or any type of memory. In addition, control logic 114 may be used to control non-volatile memory 120 and may include Data protection circuitry (not shown) for protecting data and/or performing error correction, but the invention is not limited thereto, and transmission interface circuitry 118 may conform to a particular communication standard (e.g., Serial Advanced Technology Attachment standard, External Component Interconnect standard , Rapid External Component Interconnect Standard, Non-Volatile Memory Storage Device Standard, Non-Volatile Memory Storage Device External Access Standard, Small Computer System Interface Standard, Universal Flash Memory Storage Standard, etc.), and can be based on the A specific communication standard is used to communicate, for example, the storage device 100 communicates with the host device 50 , wherein the storage interface circuit 54 can meet the specific communication standard to communicate with the host device 50 and the storage device 100 .

According to some embodiments, the host device 50 can transmit a plurality of host commands and corresponding logical addresses to the memory controller 110 to access the storage device 100, and the memory controller 110 receives the plurality of host commands and The plurality of logical addresses, and convert the plurality of host commands into a plurality of memory operation commands (for the sake of brevity, can be called a plurality of operation commands), and further use the plurality of operation commands to control non- The volatile memory 120 can read, write/program, etc. on storage units (such as data pages) with a plurality of physical addresses in the non-volatile memory 120, wherein the plurality of physical addresses can be associated with the The plurality of logical addresses are associated, when the memory controller 110 is in any non-volatile memory element 122-n ( "n" can represent any integer in the range [1, N]), when performing an erase operation on at least one block of multiple blocks of the non-volatile memory element 122-n can be erased , wherein each of the blocks may contain multiple pages (eg, data pages), and an access operation (eg, read or write) may be performed on one or more pages.

According to some embodiments, the storage server 10 (such as the main device 50) can operate according to the Ceph solution, so that the storage server 10 can be defined by software as part of a distributed storage system, preferably a plurality of storage systems ( For example, a plurality of storage servers {10}, such as storage server 10, can be used to form a resource pool (resource pool) of a distributed storage system to support such as having certain features (features) (such as fault tolerance, automatic failover control etc.), such as block device type (e.g. used to access an emulated block device in a distributed storage system), file system type (e.g. in a distributed storage system used to access a file system) and object type (for example, used to access an object in an object namespace (object namespace) in a distributed storage system), but the present invention is not limited thereto. For example, a file can be stored as an object in the object namespace in the distributed storage system, and for example, a data block corresponding to a certain address can be stored as an object namespace in the distributed storage system One of the objects, in order to implement the characteristics of the distributed storage system, in addition to the data (such as user data), the storage server 10 can be used to store additional information (such as metadata (metadata) and log (journal)).

FIG. 3 is an index object-based migration control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention, wherein the method can be applied to the storage server 10 . The storage server 10 can act as the second cloud, and can be regarded as the destination (destination) of data migration, while the remote simple storage service compatible server can act as the role of the first cloud, and can be regarded as the source of data migration (source), for example, before the first point in time T1, a user can access (for example, read or write) data in a remote simple storage service compatible server, and finds that he/she is in remote simple storage The personal storage capacity in the service compatible server is almost exhausted and may suffer from poor performance of the remote simple storage service compatible server.

In response to a request to migrate the user data of the user of the storage server 10 from the remote simple storage service compatible server to the storage server 10, in the indexing phase (such as the INDEX phase) of the multiple phases of user data migration ) period (such as migrating user data from a remote Simple Storage Service compatible server to the storage server 10 according to the user's request), the index-based object-based simple The storage service migration management module 53 can create and store multiple index objects (such as virtual objects without object data) to the storage device layer (storage device layer) of the storage server 10 as user data in the storage server 10 Respective representatives of a plurality of normal objects (normal objects), and migrate the respective access control lists (Access Control List, ACL) of the plurality of normal objects to the storage server 10, as the respective storage of the plurality of index objects taking the control list as if the indexed objects were normal objects to minimize the chance that any change in any one of the normal objects would affect the correctness of the migration of the user data, wherein the storage device The layer includes at least one storage device (eg, one or more storage devices) coupled to the main device 50 , such as a plurality of storage devices 90 . By migrating the respective access control lists of multiple normal objects to the storage server 10 as the respective access control lists of multiple index objects, the simple storage service migration management module 53 based on index objects can utilize multiple index object to simulate (emulate) multiple normal objects, for example, the destination (such as storage server 10) can start migration at the first time point T1, after the first time point T1, when the user sends any other request When sending to a destination (such as storage server 10) to attempt any change to any normal object, based on whether one or more predetermined conditions are met (such as whether any other requirement meets one or more predetermined rules), the storage server The device 10 can optionally proxy (for example, forward (forward)) any other request to the source end (for example, a remote Simple Storage Service compatible server) and respond to a response (response) corresponding to any other request ) is proxied from the source to the user.

During the data phase (such as the DATA phase) following the index phase in the multiple phases, the index object-based simple storage service migration management module 53 can trigger one or more other program modules (such as one or more migration agent programs ), to migrate the respective object data of a group of normal objects among the plurality of normal objects from the remote simple storage service compatible server to the storage device layer of the storage server 10 as a group of index objects among the plurality of index objects To complete the migration of user data, for example, the destination (such as storage server 10) can start the data phase at the second time point T2 and complete the migration at time point T3, assuming that the data phase Meanwhile, the user sends any other request to the destination (e.g. storage server 10) to attempt any changes to any normal object, similarly based on whether one or more predetermined conditions (e.g. any other request conform to one or more predetermined rules), the storage server 10 can optionally proxy (for example, forward) any other request to the source end (for example, a remote Simple Storage Service compatible server) and will correspond to any The response to other requests is proxied (e.g., forwarded) from the source to the user, for example, when any other request is a Get-object request (e.g., "GetObject"), the storage server 10 may proxy (e.g. forward) the request between the user and the source, and especially for some of their respective object data that has not yet been migrated to the destination (e.g. during migration, when the storage server 10 When an object acquisition request is received, the index agent (for example, forwards) the request for which the respective object data of the indexed objects has not been copied to the destination and needs to be copied to the destination. For better understanding, when any other request does not exist (for example, the user does not send any other request during time points T1 and T3), the set of normal objects may contain all of the plurality of normal objects, and the set of An index object can contain all of the multiple index objects.

As long as the implementation of the present invention is not hindered, one or more migration agent programs can be implemented by one or more program modules in the following program modules: (1) one or more program modules in the program module 52P running on the processing circuit 52;

(2) In the case of a multi-node architecture with a plurality of processing circuits {52} in the main device 50, among the plurality of processing circuits {52} running on one or more other processing circuits one or more program modules, wherein the plurality of processing circuits {52} includes processing circuit 52; and

(3) In the case of a multi-host architecture with a plurality of host devices {50} in the storage server 10, one or more other host devices among the plurality of host devices {50} run on One or more program modules on one or more corresponding processing circuits, wherein the plurality of main devices {50} include the main device 50;

But the present invention is not limited thereto.

According to some embodiments, the method and related devices can be used to perform certain basic operations on the Ceph architecture using Ceph software libraries (such as the "librados" software library) that can be used to provide client applications Provides direct access to object-based reliable automated distributed object store (RADOS) storage systems, and supports certain Ceph features (such as reliable automated distributed object store block device (RADOS block device, RBD), reliable automated distributed object storage gateway (RADOS gateway, RGW), Ceph file system, etc.) provide a basis, and reliable automated distributed object storage can be part of the Ceph distributed storage system, and can be regarded as A utility for interacting with a Ceph object storage cluster, but the invention is not limited thereto.

Figure 4 is a schematic diagram of some related operations of the migration control scheme based on index objects shown in Figure 3 according to an embodiment of the present invention, the first cloud and the second cloud (for better understanding, respectively marked as "simple "Storage Service Compatible Storage" and "Bigtera Storage") can be used as examples of remote simple storage service compatible server and storage server 10 respectively, and users can use a client device (such as a simple storage service client, for simplicity For the sake of sake, marked as "S3 client") to access the remote Simple Storage Service compatible server and the storage server 10 respectively, for example, each of the remote Simple Storage Service compatible server and the storage server 10 can Provides simple storage services and can be compatible with simple storage services, and especially can provide object storage to handle large amounts of unstructured data, and can be based on the application program interface (representational state transfer, REST) of the presentation layer ( application program interface, API) (such as a network service (web service) application program interface that conforms to the constraints of the presentation layer state transition architecture, such as a presentation layer state transition application program interface (RESTful API)) and by using the hypertext transfer protocol ( hypertext transfer protocol, HTTP)/hypertext transfer protocol secure (hypertext transfer protocol secure, HTTPS) to be accessed. For certain Simple Storage Service features, each remote Simple Storage Service compatible server and

The storage server 10 can be used to store one or more objects in a bucket (such as a container of objects), and any one of the one or more objects can include a file and optionally Contains any metadata describing the file. The object prefix of any object can contain one or more symbols "/" to support the folder concept (folder concept) to simulate a folder. In addition , each Remote Simple Storage Service compliant server and storage server 10 may have at least one Ceph Object Gateway (labeled "Gateway" for brevity), such as built on top of librados to provide Application of a presentation layer state transition gateway to at least one object storage interface of a Ceph storage cluster, wherein the at least one Ceph object gateway supports a remote simple storage service compatible server and the storage server 10 is simple storage Service compatible and Swift compatible. As shown in the lower part of Figure 4, during user data migration (for brevity, marked as "data" in the hollow arrow), the user can send any other request (such as a GET request) (by borrowing By using the simple storage service client) to the storage server 10 to request part of the user data, the storage server 10 can proxy (for example, forward) any other request (such as a GET request) to the remote simple storage service server, and proxies (eg forwards) part of the data corresponding to that one of the other requests (eg, from a remote Simple Storage Service compatible server to the user (eg, the user's Simple Storage Service client), for brevity See this response labeled "Information" on some of the dotted arrows). For the sake of brevity, similar content in this embodiment will not be repeatedly described in detail here.

According to some embodiments, in the INDEX stage (for example, the INDEX period), under default conditions, each index object of the plurality of index objects can be created to have a source (source, src) object (for example, in the first certain attributes of a corresponding object in the cloud, such as a normal object, wherein the index object acts as a representative of the normal object), and these attributes may include the last modification time (last -modified-time, mtime) (e.g. source modification time (src-mtime), such as bigtera-scr-mtime), size of source object (e.g. source size (src-size), such as bigtera-src-size), and source object The entity tag (entity tag, Etag) (for example, a file hash (file hash) for integrity checking, such as MD5 sum), such as the source entity tag (src-etagt), such as bigtera-src-etag).

FIG. 5 is a schematic diagram of some objects and some related features of the index object-based migration control scheme shown in FIG. 3 according to an embodiment of the present invention. During user data migration, the source (such as remote simple storage service compatible server) may contain one or more storage bodies (e.g., one or more containers of objects), such as a remote storage body (remote bucket), and the destination (e.g., storage server 10) may contain a or multiple storage bodies (such as one or more containers of objects), such as the storage body shown above the remote storage body, wherein the request and the response represent the storage server 10 from the user (such as the user's The request received by the simple storage service client) and the response sent back to the user (such as the user's simple storage service client) by the storage server 10, for example, the remote storage body may include multiple normal Objects, for better understanding, are shown as small circles in the distal reservoir. In addition, the repository of the destination may contain at least a portion (eg, a portion or all) of indexed objects, and in particular one or more normal objects, depending on the progress of the migration (eg, the percentage of the migration completed, such as 1%, 2%, etc.) and/or the existence of any other requirement.

Before migrating the object data of any one index object of at least a part of the plurality of index objects, the storage server 10 can set the storage body of the destination to have ) with the same repository name, and has the same user credential as the remote user credential (eg, the remote repository's user credential), and especially has fake usage information, such as fake usage (such as size, number of objects, etc.) as temporary user information for migration phases (such as index phase and data phase). At the beginning, before migrating the object data of any index object, the storage server 10 can first create a plurality of index objects with temporary attributes such as virtual (dummy) index objects (such as dummy index objects) in the destination storage body (for example, no object data virtual object), when a normal object among multiple normal objects has been migrated to the destination or an operation on the normal object has been performed on the destination (for example, the object data of the normal object has been migrated, or the user has overridden the normal object), the normal object may exist in the container (as shown in Figure 5), and, for each index object of the plurality of index objects, the temporary properties may include: (1) The last modification time of the source object (for example, the last modification time of the source object), such as bigtera-src-mtime;

(2) The size of the source object (for example, the size of the source object), such as bigtera-src-size;

(3) Source object entity tags (such as source object entity tags), such as bigtera-src-etag;

Any one of these temporary attributes (for example, each attribute) can be implemented as an extended attribute (x-attribute/XATTR/xattr), but the present invention is not limited thereto. Once the object data of the source object is migrated to the destination or The user overwrites the object data at the destination, and the destination (such as the storage server 10) can remove the temporary attributes listed above. For the sake of brevity, the similar content of this embodiment will not be repeated here A detailed description.

According to some embodiments, taking the normal object shown in the upper right of FIG. 5 as an example of the source object, when the object data of the source object is migrated to the destination or the user overwrites the object data at the destination, the storage server The implementer 10 may invalidate indexed objects that represent normal objects (e.g., by removing their temporary attributes) and/or replace indexed objects with normal objects in the repository at the destination, but the invention is not limited to this.

Generally speaking, changing an object's access control list will not change the modification time of the object. The related art data migration method does not seem to take this into account when it is proposed, so it cannot properly handle deleted objects, read-only objects, etc. . The method and associated apparatus of the present invention may suitably operate under any of a variety of conditions, further implementation details of which are described in the following examples.

Fig. 6 is a schematic diagram of a migration agent control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention, in response to migrating user data from remote simple storage service compatible servers to storage According to the requirements of the server 10, the storage server 10 can use the simple storage service migration management module 53 based on the index object during a set-up phase (such as the setup period) before the index phase among the multiple phases. To carry out the pre-processing of the user data migration, for example, the pre-processing may include the operations of steps S01-S05, but the present invention is not limited thereto.

In step S01, the index object-based simple storage service migration management module 53, for example, can provide a group of simple storage service users (such as a single user, such as the above-mentioned users, or include Multiple users of the above-mentioned users) start simple storage service migration.

In step S02 , the index object-based simple storage service migration management module 53 can migrate respective user credentials and respective settings (such as quotas) of the group of simple storage service users.

In step S03, the index object-based simple storage service migration management module 53 can create one or more storage bodies for each user of the group of users in the destination, and set related migration information, for example, the migration The information may include proxy parameters (eg, remote user credentials to proxy (eg, forward) to) and other proxy parameters to direct the destination (eg, storage server 10) to proxy (eg, forward) any other request to an origin (eg, a remote Simple Storage Service compatible server), and proxies (eg, forwards) a response corresponding to the any other request from the origin to the user.

In step S04, for each of the one or more storages, the index object-based simple storage service migration management module 53 can migrate the storage access control list of one of the source terminals corresponding to the remote storage. (bucket ACL), storage metadata (bucket metadata), and storage quota (bucket quota), as the storage access control list, storage metadata and storage body quota.

In step S05, the index object-based simple storage service migration management module 53 can switch at least one endpoint to access user data from at least one cluster of remote simple storage service compatible servers to the storage server 10 At least one cluster of .

In step S06, after the setup phase, the index object-based simple storage service migration management module 53 may create an index storage job (such as an INDEX job) for each of the one or more storages, In order to migrate the corresponding remote storage of the source to the storage of the one or more storages, for example, the simple storage service migration management module 53 based on the index object can set the storage to the index stage (eg INDEX stage).

For better understanding, any one of one or more migration agents (e.g., each agent) can be viewed as a program module that actually performs object migration, e.g., the Simple Storage Service migration management module based on indexed objects Group 53 can create multiple job types (such as multiple index jobs corresponding to the index stage (such as multiple INDEX jobs) and multiple data jobs corresponding to the data stage (such as multiple DATA jobs)), and especially Refers to sending and/or assigning multiple job types to one or more migration agents for data phase and index phase operations respectively. Assuming that the one or more agent programs comprise a plurality of agent programs, the plurality of agent programs may be respectively implemented to run on a plurality of cluster nodes, wherein an agent program runs on each cluster node of the plurality of cluster nodes, Therefore, the migration management module 53 of the simple storage service based on index objects can transfer user data (such as the respective object data of the object) in parallel by means of one or more migration agent programs (such as multiple migration agent programs) Migrate from source to destination.

In step S07, the index object-based simple storage service migration management module 53 can use one or more migration agent programs to execute the INDEX work for each object (such as multiple normal Every normal object of the object) does the following: (1) Create a virtual object with temporal properties (such as source object last modification time, such as bigtera-src-mtime; source object size, such as bigtera-src-size; and source object entity tag, such as bigtera-src-etag) (e.g. an object with no object data) as one of multiple index objects, where the temporal attributes can be false attributes of each object at the source (e.g. false last-modified time, false entity tags such as false file hash and false size), rather than the real attributes of each object at the source, so they can be regarded as special attributes;

(2) migrating the object access control list of the object from the source to the destination;

All the index objects among the plurality of index objects are created after all the operations in step S07 are completed.

When an index job (such as an INDEX job) is completed, the index object-based simple storage service migration management module 53 can create a data job (such as a DATA job) for each of the one or more storage bodies to perform Migration of the corresponding distal reservoir of the origin to the one reservoir of the one or more reservoirs.

In step S08, for each of the one or more storage bodies, the index object-based simple storage service migration management module 53 can execute DATA by an agent program of one or more migration agent programs. work for each object at the source (e.g. each normal object of multiple normal objects if no other requirement exists, or each normal object of the set of normal objects if any other requirement exists object, where any other requirement may cause an object to be overwritten in the destination as a normal object in the repository) do the following: (1) migrating the object data of the object from the source to the destination;

Wherein the migration agent program may migrate the latest version of the storage quota (for example, in the absence of any other requirements, the original storage quota of the corresponding remote storage; or in the presence of any other requirements , the storage quota updated in response to any changes made by the user) as the storage quota of the storage at the end of the job, for example, the index object based simple storage service migration management module 53 can use the storage Set to data stage (eg DATA stage).

Of all the repositories corresponding to the group of simple storage service users, while one of the repositories is being processed in an indexing phase (eg, an INDEX phase), another of those repositories is available at Processing is in progress in the data phase (eg DATA phase). Assuming the one or more buckets comprise multiple buckets, while one bucket of the multiple buckets is being processed in an indexing phase (eg, an INDEX phase), another bucket of the multiple buckets is available in the data stage (such as the DATA stage), and for the sake of brevity, similar content in this embodiment will not be described in detail here.

According to some embodiments, the index phase (such as the INDEX phase) may be much shorter than the data phase (such as the DATA phase). In this index phase, the simple storage service migration management module 53 based on the index object can pass the The migration agent program performs INDEX work to list remote objects (such as multiple normal objects at the source) for creating virtual objects and migrating object access control lists. Since any other requirement may exist in the indexing phase (such as the INDEX phase), different situations need to be considered, such as a first situation in which the index object has not been created, and a second situation in which the storage server 10 is creating the index object, etc. etc., for example, processing for PUT/POST requests should be done at a local site (e.g. destination, such as storage server 10), and storage server 10 can do this processing for PUT/POST requests and At the same time, the PUT/POST request is proxied to a remote site (for example, the source, such as a remote Simple Storage Service compatible server), so that the storage usage and the object list (List-Objects) request ( For example ListObjects request) is correct. A user may overwrite one or more objects during an index phase (such as the INDEX phase) if any other requirement exists in that phase. If an index object is overwritten by a user, the object is no longer an Index object, so the migration agent will not migrate object data for this object in the data phase (such as DATA phase), in addition, before the data phase (such as DATA phase), all indexed objects may have been created (including their object access control list). In the data stage (such as the DATA stage), the index object-based simple storage service migration management module 53 can perform the DATA work through the migration agent program in step S08 to list local objects (such as objects in local locations, such as index object), and migrate object data for each local object. Generally speaking, most operations can be performed normally, except GET/HEAD object request, in order to solve competition hazard (racing hazard), storage server 10 can use atomic (atomic) operation to create index object and by this index object To migrate data (such as object data of multiple normal objects on the source side), for example, when the migration agent is creating index objects or migrating data about index objects, if a target object of the PUT operation already exists or does not have the above temporary attribute, the PUT operation is canceled, which means that the user has overwritten the object. For proxying (e.g., forwarding) indexed objects in a reliable automated distributed object storage gateway, in the presence of any other requirement, when a client device (e.g., a simple storage service client) operates on an indexed object or While the repository for the destination is in the INDEX phase, it may be desirable that the destination (e.g. storage server 10) can proxy any other request to a remote location (e.g. source, such as a remote Simple Storage Service container server), and return the proxy response (for example, the response corresponding to any other request) to the client device. For the sake of brevity, the similar content of these embodiments will not be described in detail here again.

FIG. 7 is a schematic diagram of an example of a Ceph object storage architecture with an index object-based simple storage service migration management module 53 shown in FIG. 1, and a program module 52P running on the processing circuit 52 shown in FIG. 1 may contain a Ceph Object Gateway module (labeled "Ceph Object Gateway" in Figure 7 for brevity), where the Ceph Object Gateway module contains submodules such as a reliable Automated decentralized object storage gateway module (labeled "RADOS Gateway" for brevity and better understanding) and librados software library (labeled "librados" for brevity) and index based The Simple Storage Service Migration Management Module 53 for Objects is implemented in the Reliable Automated Distributed Object Storage Gateway Module. According to this embodiment, a client device (labeled as "Client") can communicate via Hypertext Transfer Protocol (such as Simple Storage Service or Swift API) (for brevity, marked as "HTTP (S3 or Swift API)") to access the storage server 10, the storage server 10 can use multi-node architecture, multi-host architecture, etc., but the present invention is not limited thereto.

As shown in FIG. 7, a reliable automated distributed object storage of the storage server 10 (for example, an object-based storage system for reliable automated distributed object storage) may include passing through at least one public network (such as one or a plurality of public networks) to be coupled to a plurality of object storage device (object storage device, OSD) nodes (for brevity, marked as "OSD nodes") of the main device 50 shown in FIG. 1, and also include Other parts of the Ceph object storage architecture, such as at least one cluster network (cluster network) (for example, one or more cluster networks) coupled to the plurality of object storage device nodes, an odd number of monitors that constitute a quorum (Quorum) monitor and so on. The at least one public network can be used to achieve a first predetermined speed such as gigabits per second (labeled "1G" for brevity), and the at least one clustered network can be used to achieve a second predetermined speed such as 10 billion bits per second (marked as "10G" for the sake of brevity), wherein the second predetermined speed has as many bits per second as possible, and the plurality of object storage device nodes may include journal (journal ) (e.g. various types of journaling, such as object storage device journaling, etc.) (just a bunch of disks/drives, JBOD)), for the sake of brevity, the similar content of these embodiments will not be described in detail again here.

Examples of object-related operations for simple storage service actions may include, but are not limited to: (1) Object list request (such as ListObjects request);

(2) Head-Object requirements (such as HeadObject request used to obtain object properties);

(3) Object acquisition request (such as GetObject request);

(4) Object access control list (Get-Object-ACL) request (eg GetObjectaACL request);

(5) Object placement (Put-Object) request (such as PutObject request);

(6) Create a multipart upload (Create-Multipart-Upload) request (eg CreateMultipartUpload request);

(7) Complete multi-component upload (Complete-Multipart-Upload) requirements (such as CompleteMultipartUpload request);

(8) Object placement access control list (Put-Object-ACL) request (such as PutObjectACL request);

(9) Object deletion (Delete-Object) request (such as DeleteObject request);

Some of the subsequent embodiments may focus on operations related to the objects listed above.

Fig. 8 is a schematic diagram of an object listing control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. As shown in Fig. 8, it is used for processing during user data migration A workflow for listing one or more objects may include steps S10-S14, wherein S14 may include some sub-steps such as steps S14A-S14C.

In step S10 , the storage server 10 (such as the index object-based simple storage service migration management module 53 ) may receive an object listing request among one or more object listing requests from the user.

In step S11, in response to the object listing request, the index object-based simple storage service migration management module 53 may check the storage corresponding to the object listing request in at least one storage in the storage device layer of the storage server 10 Whether a bucket (eg, destination's bucket) is being processed in an indexing phase (eg, INDEX phase), where the at least one bucket may represent at least one container (eg, one or more containers) of objects, if so , go to step S12; if not, go to step S14 (for example, step S14A), for example, the object listing request requires listing the object of the storage body.

In step S12, because the storage body is being processed in the index phase (such as the INDEX phase), the simple storage service migration management module 53 based on the index object can control the storage server 10 to list the object to the request agent (such as Forwarding) to the source (such as a remote Simple Storage Service compatible server).

In step S13, the index object-based simple storage service migration management module 53 can control the storage server 10 to list the response agent corresponding to the object from the source (such as a remote simple storage service compatible server) (e.g. forward) to the user.

In step S14, since the storage body is not being processed in the index phase (for example, the INDEX phase), the index object-based simple storage service migration management module 53 can control the storage server 10 to perform At least one normal action, no need to list the object and request proxy (such as forwarding) to the source (such as a remote simple storage service compatible server), for example, the simple storage service migration management module 53 based on the index object can control the storage The server 10 optionally passes a local object listing result (eg, a local object listing result for the local location, such as the result of executing the object listing request at the local location) according to an associated ACL check. returned to the user.

In step S14A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S14B; if not, then enter step S14C, for example, when the access control list When indicating that the user has the authority to apply the object list request to the storage body (for example, the authority to execute the object list request to obtain the result), the index object-based simple storage service migration management module 53 can determine the access control list Check passed, and for example, when the access control list indicates that the user does not have the permission to apply the object list request to the storage (such as the permission to execute the object list request to get the result), the simple storage service based on the index object The migration management module 53 may determine that the ACL check fails.

In step S14B, the index object-based simple storage service migration management module 53 can control the storage server 10 to return the local object list result to the user.

In step S14C, the index object-based simple storage service migration management module 53 can control

The storage server 10 prevents the local object listing result from being returned to the user.

For better understanding, the method can be described using the workflow shown in FIG. 8, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 8 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Figure 9 is a schematic diagram of the object header control scheme of the method for seamless migration of simple storage services using index objects according to an embodiment of the present invention. As shown in Figure 9, it is used to process a user data during migration A workflow required by or multiple object headers may include steps S20~S28, S29A, and S29B, wherein step S25 may include some sub-steps (such as S25A~S25C), and step S27 may include some sub-steps (such as S27A~S27C) .

In step S20, the storage server (for example, the index object-based simple storage service migration management module 53) may receive an object header request of one or more object header requests from the user, for example, the object header request requests an acquisition purpose The object attribute of an object in the storage body of the land, and the object can be regarded as a target object required by the object header. The object attribute can include: (1) The last modification time of the object, such as the last modification time of the object;

(2) Object entity label, such as the entity label of the object (eg file hash);

(3) The size of the object, such as the size of the object;

Wherein these attributes can be retrieved by an HTTP HEAD request.

In step S21, in response to the object header request, the index object-based simple storage service migration management module 53 can check the at least one storage in the storage device layer of the storage server 10 that corresponds to the object header request. Whether a storage body (such as the storage body of the destination) is being processed in the index phase (such as the INDEX phase), if yes, go to step S22; if not; then go to step S26.

In step S22, because the storage body is being processed in the index phase (such as the INDEX phase), the index object-based simple storage service migration management module 53 can check whether the object (such as the target object required by the object header) is Exist, if yes, go to step S23; if not, go to step S28.

In step S23, due to the existence of the object, the index object-based simple storage service migration management module 53 can check whether the object (such as the target object required by the object header) is still an index object, and if so, enter the step S24; if not, go to step S27 (eg step S27A).

In step S24, because the object is still an index object, the index object-based simple storage service migration management module 53 can check whether the object access control list of the object has been migrated; if so

, go to step S25 (such as step S25A); if no, go to step S29A, for example, there is usually a short time delay between creating a virtual object (such as the above-mentioned virtual object) and migrating the object access control list Therefore, the index object-based simple storage service migration management module 53 can perform the checking operation of step S24 in the workflow to comply with the access control policy.

In step S25, the index object-based simple storage service migration management module 53 can control the storage server 10 to perform at least one special action according to the object header requirements, for example, the index object-based simple storage service migration management module 53 The storage server 10 may be controlled to selectively check whether a specific attribute (such as a source modification time, such as attributes in bigtera-src-mtime, source entity tags such as bigtera-src-etag, and source size such as bigtera-src-size) are returned to the consumer.

In step S25A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S25B; if not, then enter step S25C, for example, when the access control list When the list indicates that the user has the authority to apply the object header request to the object (for example, the authority to obtain the result of executing the object header request), the index object-based simple storage service migration management module 53 can determine the access control list check By, for example, when the access control list indicates that the user does not have the right to apply the object header request to the object (such as the right to obtain the result of executing the object header request), the simple storage service migration management based on the index object The module 53 may determine that the ACL is not passed.

In step S25B, the index object-based simple storage service migration management module 53 can control the storage server 10 to return the attributes recorded in the special attributes of the index object (such as the index object mentioned in step S23) to sent to the user.

In step S25C, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent the attributes that will be recorded in the special attributes of the index object (such as the index object mentioned in step S23) returned to the user.

In step S26, because the storage body is not being processed in the index phase (such as the INDEX phase), the index object-based simple storage service migration management module 53 can check the object (such as the target object required by the object header) Is it still an index object? If yes, go to step S25 (such as step S25A); stage), all object access control lists have been migrated, so there is no need to obtain remote object access control lists again.

In step S27, because the object (for example, the target object required by the object header) is no longer an index object (for example, the object has become a normal object), the index object-based simple storage service migration management module 53 can control The storage server 10 can perform at least one normal operation according to the request of the object header, for example, the simple storage service migration management module 53 based on the index object can control the storage server 10 to select according to a relevant access control list check Optionally returns the object's local metadata (eg, the object's metadata at the local location) to the user.

In step S27A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S27B; if not, then enter step S27C, for example, the operation of step S27A can be similar Operation in step S25A.

In step S27B, the index object-based simple storage service migration management module 53 can control the storage server 10 to return the local metadata of the object to the user.

In step S27C, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent the local metadata of the object from being returned to the user.

In step S28, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the request (for example, object header request) to the source/remote location (for example, remote simple storage service compatible server) to have the remote location check the object access control list.

In step S29 , the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example, forward) the response corresponding to the object header request to the user.

In step S29E, the index object-based simple storage service migration management module 53 can control the storage server 10 to return an error notification (such as '503 Service Unavailable'), for example, because the migration agent program will Migrate the ACL, which tells the client to try again later.

For better understanding, the method can be described using the workflow shown in FIG. 9, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 9 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Figure 10 is a schematic diagram of the object acquisition control scheme of the method for seamless migration of simple storage services using index objects according to an embodiment of the present invention. As shown in Figure 10, it is used to process a user data during migration A workflow for acquiring one or more objects may include steps S30-S36, wherein step S36 may include some sub-steps (such as S36A-S36C).

In step S30, the storage server 10 (such as the index object-based simple storage service migration management module 53) may receive one or more object acquisition requests from the user, for example, the object acquisition request Obtain object data with attributes, such as the object data of an object in the destination storage and the object attributes of the object, and this object can be regarded as a target object of the object retrieval request.

In step S31, in response to the object acquisition request, the index object-based simple storage service migration management module 53 may check the at least one storage body in the storage device layer of the storage server corresponding to the object acquisition request Whether (such as the storage body of the destination) is being processed in the indexing stage (such as the INDEX stage), if yes, go to step S32; if not, go to step S33.

In step S32, because the storage body is being processed in the index phase (such as the INDEX phase), the index object-based simple storage service migration management module 53 can check whether the object (such as the target object of the object acquisition request) is Yes, if yes, go to step S33; if not, go to step S34. For some workflows between steps S32 and S34 , examples of the reason why the object does not exist may include: the index object has not been created; and the index object has been deleted by the user.

In step S33, the index object-based simple storage service migration management module 53 can check whether the object (such as the target object of the object acquisition request) is still an index object, if yes, go to step S34; if not, then Go to step S36 (for example, step S36A).

In step S34, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the request (for example, the object acquisition request) to the source/remote location (for example, the remote Simple Storage Service compatible server).

In step S35 , the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example, forward) the response corresponding to the object acquisition request to the user.

In step S36, in response to the object (for example, the target object requested by the object) is no longer an index object (for example, the object becomes a normal object), the index object-based simple storage service migration management module 53 can control the storage The server 10 performs at least one normal action according to the object acquisition request. For example, the simple storage service migration management module 53 based on the index object can control the storage server 10 to check whether it is passed according to a related access control list. Local data of the object (eg, the data of the object at the local location) is returned to the user.

In step S36A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S36B; if not, then enter step S36C, for example, when the access control list When the list indicates that the user has the authority to apply the object acquisition request to the object (for example, the authority to obtain the result of executing the object acquisition request), the index object-based simple storage service migration management module 53 can determine the access control list Check passed, and for example, when the access control list indicates that the user does not have the permission to apply the object get request to the object (such as the permission to obtain the result of executing the object get request), the simple storage service based on the index object The migration management module 53 may determine that the ACL is not passed.

In step S36B, the index object-based simple storage service migration management module 53 can control the storage server 10 to return the local data of the object to the user.

In step S36C, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent the local data of the object from being returned to the user.

For better understanding, the method can be described using the workflow shown in FIG. 10, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 10 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Figure 11 is a schematic diagram of the object acquisition access control list control scheme of the method for seamless migration of simple storage services using index objects according to an embodiment of the present invention, as shown in Figure 11, during the period of user data migration A workflow for processing one or more objects to obtain an ACL request may include steps S40-S48, wherein step S45 may include some sub-steps (eg S45A-S45C).

In step S40, the storage server 10 (such as the indexed object-based simple storage service migration management module 53) may receive an object acquisition ACL in one or more object acquisition ACL requests from the user Requires, e.g., the object get ACL request requires a get object ACL (eg, the ACL of a bin in the destination's bin), and this object can be considered as the object get ACL request A target object of .

In step S41, in response to the request of the object to obtain the access control list, the simple storage service migration management module 53 based on the index object can check the above-mentioned at least one storage body in the storage device layer of the storage server 10 corresponding to the object obtained Whether a storage body (such as the storage body of the destination) required by the access control list is being processed in the index phase (such as the INDEX phase), if yes, enter step S42; if not, then enter step S45 (such as step S45A) .

In step S42, because the storage body is being processed in the index phase (such as the INDEX phase), the simple storage service migration management module 53 based on the index object can check the object (such as the object to obtain the access control list requirements) target object) exists, if yes, go to step S43; if not, go to step S46.

In step S43, the index object-based simple storage service migration management module 53 can check whether the object (such as the target object required by the object to obtain the access control list) is still an index object, and if so, proceed to step S44; If not, go to step S45 (eg step S45A).

In step S44, because the object is still an index object, the index object-based simple storage service migration management module 53 can check whether the object access control list of the object has been migrated, and if so, enter step S45 (such as step S45A ); if not, go to step S48.

In step S45, the index object-based simple storage service migration management module 53 can control the storage server 10 to perform at least one normal operation according to the object’s access control list request, for example, the index object-based simple storage service migration The management module 53 may control the storage server 10 to selectively return the local ACL of the object (eg, the ACL of the object at the local location) to the user.

In step S45A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S45B; if not, then enter step S45C, for example, when the access control list When indicating that the user has the permission to apply the object's access control list request to the object (for example, to obtain the permission to execute the object's access control list request result), the index object-based simple storage service migration management module 53 It can be judged that the access control list check is passed. For example, when the access control list indicates that the user does not have the permission to apply the object's access control list request to the object (such as obtaining and executing the object's access control list request In the case of the authority of the result), the index object-based simple storage service migration management module 53 may determine that the access control list check fails.

In step S45B, the index object-based simple storage service migration management module 53 can control the storage server 10 to return the local access control list of the object to the user.

In step S45C, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent the local ACL of the object from being sent back to the user.

In step S46, the indexed object-based simple storage service migration management module 53 can control the storage server 10 to proxy (e.g. forward) the request (e.g. the object obtain access control list request) to the source end/remote location (eg Remote Simple Storage Service compatible server).

In step S47 , the indexed object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example, forward) the response corresponding to the object's access control list request to the user.

In step S48, the index object-based simple storage service migration management module 53 can control the storage server 10 to return an error notification (such as '503 Service Unavailable').

For better understanding, the method can be described using the workflow shown in FIG. 11, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 11 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Fig. 12 is a schematic diagram of an object placement control scheme for a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. As shown in Fig. 12, it is used to process a user data during migration A workflow for writing one or more object data related requirements (such as one or more object placement requirements, one or more creation multi-component upload requirements, and one or more completion multi-component upload requirements) may include steps S50-S58 , wherein step S56 may include some sub-steps (eg S56A~S56C).

In step S50, the storage server 10 (such as the index object-based simple storage service migration management module 53) may receive from the user one or more object data writing related requests (such as one or more placement requests, one or multiple CREATE MULTI-COMPONENT UPLOAD requests, and one or more COMPLETE MULTI-COMPONENT UPLOAD requests) in an object data write related request (e.g. an object put/create multipart upload/complete multipart upload request such as an object put request, a create multipart upload request or a complete multipart upload request), for example, the object data write related request requires writing an object to the destination's bucket, and this object can be considered as the object data A target object for writing related requests.

In step S51, in response to the request for writing the object data, the index object-based simple storage service migration management module 53 may check the above-mentioned at least one storage body in the storage device layer of the storage server 10 corresponding to the object data writing Whether a storage body (such as the storage body of the destination) of the relevant request is being processed in the index phase (such as the INDEX phase), if yes, enter step S52; if not, then enter step S56 (such as step S56A).

In step S52, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the request (for example, the request for writing the object data) to the source end (for example, the remote simple Storage service compatible server) to have the remote Simple Storage service compatible server check the object access control list.

In step S53, the index object-based simple storage service migration management module 53 can control whether the request (for example, the request for writing the object data) is successful without any error, and if it is (for example, a response received from the source, such as Corresponding to the response to the object data write-related request, indicating that no error occurs), enter step S54; if not (for example, the response received from the source, such as the response corresponding to the object data write-related request, indicates one or multiple errors (one error response), then go to step S57.

In step S54, the index object-based simple storage service migration management module 53 can check whether the object (for example, the target object required to write the object data) exists, if yes, go to step S55; if not, go to Step S56 (for example, step S56A).

In step S55, the index object-based simple storage service migration management module 53 can check whether the object access control list of the object has been migrated, if yes, go to step S56 (such as step S56A); if not, go to step S58 .

In step S56, the index object-based simple storage service migration management module 53 can control the storage server 10 to perform at least one normal operation according to the object data writing related requirements, for example, the index object-based simple storage service migration management The module 53 can selectively write data (such as the object data carried in the object data writing related request) as the object data of the object according to a related access control list check.

In step S56A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S56B; if not, then enter step S56C, for example, when the access control list When indicating that the user has the permission to write the object data to the relevant request and apply it to the object (the permission to write the object data to the object data carried by the relevant request), the migration management module of the simple storage service based on the index object Group 53 can determine that the access control list check is passed, and for example, when the access control list indicates that the user does not have the authority to apply the object data to the relevant request (write the object data to the relevant request carried The object data write permission), the index object-based simple storage service migration management module 53 can determine that the access control list check fails.

In step S56B, the index object-based simple storage service migration management module 53 can control the storage server 10 to write the data (for example, the object data carried in the request for writing the object data) into the object’s Object data.

In step S56C, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent the data (for example, the object data carried in the object data writing request) from being written into the object. The object data.

In step S57, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the error response corresponding to the object data writing request to the user.

In step S58, the index object-based simple storage service migration management module 53 can control the storage server 10 to migrate the object ACL of the object from the source to the destination.

For better understanding, the method can be described using the workflow shown in FIG. 12, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 12 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Figure 13 is a schematic diagram of the object placement access control list control scheme of the method for seamless migration of simple storage services using index objects according to an embodiment of the present invention, as shown in Figure 13, during the period of user data migration A workflow for processing one or more object placement ACL requests may include steps S60-S65, wherein step S64 may include some sub-steps (eg, S64A-S64C).

In step S60, the storage server 10 (for example, the index object-based simple storage service migration management module 53) may receive an object placement ACL request of one or more object placement ACL requests from the user, For example, the object placement ACL request requires writing or updating the object ACL (such as the ACL of an object in the destination's storage), and the object can be considered the object placement ACL A target object requested.

In step S61, in response to the object placement ACL request, the simple storage service migration management module 53 based on the index object can check that the above-mentioned at least one storage body in the storage device layer of the storage server 10 corresponds to the object Whether a storage body (such as the storage body of the destination) that places the ACL request is being processed in the index phase (such as the INDEX phase), if yes, enter step S62, if not, then enter step S64 (such as step S64A ).

In step S62, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the request (for example, object placement ACL request) to the source end (for example, remote simple storage compatible server).

In step S63, the index object-based simple storage service migration management module 53 can check whether the request (such as the object placement ACL request) is successful without any error, and if so (such as a response received from the source, such as A response corresponding to the object placement ACL request indicating that no error occurs), proceed to step S64 (such as step S64A); if not (such as a response received from the source, such as corresponding to the object placement ACL request The response is an error response (error response) indicating one or more errors), then enter step S65.

In step S64, the index object-based simple storage service migration management module 53 can control the storage server 10 to perform at least one normal operation according to the object placement ACL request, for example, the index object-based simple storage service migration The management module 53 can control the storage server 10 to selectively update the object storage with the new/up-to-date ACL carried by the object placement ACL request according to whether an associated ACL check is passed or not. access control list (such as the access control list for the target object).

In step S64A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S64B; if not, then enter step S64C, for example, when the access control list When indicating that the user has the permission to place the object in the access control list (such as the permission to write or update the object's access control list), the index object-based simple storage service migration management module 53 can determine The ACL check passes, and for example, when the ACL indicates that the user does not have the permission to put the object on the ACL request to apply to the object (such as permission to write or update the ACL on the object ), the index object-based simple storage service migration management module 53 may determine that the ACL check fails.

In step S64B, the index object-based simple storage service migration management module 53 can control the storage server 10 to update the object ACL with the new ACL carried in the object placement ACL request list (such as the access control list for the target object).

In step S64C, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the error response corresponding to the object placement ACL request to the user.

For better understanding, the method can be described using the workflow shown in FIG. 13, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 13 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps.

Figure 14 is a schematic diagram of the object deletion control scheme of the method for seamless migration of simple storage services using index objects according to an embodiment of the present invention. As shown in Figure 14, it is used to process a user data migration period A workflow of one or more object deletion requests may include steps S70-S75, wherein step S74 may include some sub-steps (eg S74A-S74D).

In step S70, the storage server 10 (for example, the index object-based simple storage service migration management module 53) may receive an object deletion request in one or more object deletion requests from the user, for example, the object deletion request request An object in the destination bucket is deleted, and this object can be regarded as a target object of the object deletion request.

In step S71, in response to the object deletion request, the index object-based simple storage service migration management module 53 may check the at least one storage body in the storage device layer of the storage server 10 corresponding to the object deletion request. Whether the storage body (such as the storage body of the destination) is being processed in the index phase (such as the INDEX phase), if yes, go to step S72, if not, then go to step S74 (such as step S74A).

In step S72, the index object-based simple storage service migration management module 53 can control the storage server 10 to proxy (for example forward) the request (for example, an object deletion request) to the source end (for example, a remote simple storage service compatible server).

In step S73, the index object-based simple storage service migration management module 53 can check whether the request (e.g. object deletion request) is successful without any error, if yes (e.g. the response received from the source, such as corresponding to the object delete request response, indicating that no error occurs), enter step S74 (such as step S74A); if not (for example, the response received from the source, such as the response corresponding to the object deletion request, is an indication of one or more errors - error response), then go to step S75.

In step S74, the index object-based simple storage service migration management module 53 can control the storage server 10 to perform at least one normal operation according to the object deletion request, for example, the index object-based simple storage service migration management module 53 The storage server 10 can be controlled to selectively delete the object (such as the target object) according to at least one related access control list check, and for example, the simple storage service migration management module 53 based on the index object can control the storage The server 10 selectively deletes the object (such as the target object) at least according to whether the object exists. Preferably, the simple storage service migration management module 53 based on the index object can control the storage server 10 according to a first The object (eg, the target object) is selectively deleted if a condition and a second condition are met, wherein the first condition represents that the ACL check is passed, and the second condition represents that the object exists.

In step S74A, the index object-based simple storage service migration management module 53 can check whether the access control list check is passed, if yes, enter step S74B; if not, then enter step S74D, for example, when the access control list When indicating that the user has the permission to apply the object deletion request to the object (for example, the permission to delete the object), the index object-based simple storage service migration management module 53 can determine that the ACL check is passed, and for example, When the access control list indicates that the user does not have the authority to apply the object deletion request to the object (for example, the authority to delete the object), the index object-based simple storage service migration management module 53 can determine the access control list Check failed.

In step S74B, the index object-based simple storage service migration management module 53 can check whether the object (for example, the target object requested by the object deletion) exists, if yes, go to step S74C; if not, go to step S74D.

In step S74C, the index object-based simple storage service migration management module 53 can control the storage server 10 to delete the object (such as the target object).

In step S74D, the index object-based simple storage service migration management module 53 can control the storage server 10 to prevent deletion of the object (such as the target object).

In step S75 , the index object-based simple storage service migration management module 53 can control the storage server 10 to delegate (for example, forward) the error response corresponding to the object deletion request to the user.

For better understanding, the method can be described using the workflow shown in FIG. 14, but the present invention is not limited thereto. According to some embodiments, one or more steps in the workflow shown in FIG. 14 can be be added, deleted, or changed, e.g., at least a part (eg, part or all) of the workflow may be performed multiple times, and in particular any step including an inspection operation may be performed multiple times to generate multiple Check the results to individually trigger the execution of some subsequent steps. In addition, the combination of steps S74A and S74B can be used to check whether the first condition and the second condition are satisfied, the conditions respectively checked by steps S74A and S74B can be exchanged, for example, based on the simple storage service migration management model of index objects Group 53 can check whether the second condition in step S74A is satisfied, and check whether the first condition in step S74B is satisfied, if both the first condition and the second condition are satisfied (such as the access If the control list check is passed and the object exists), go to step S74C; otherwise, go to step S74D. Table I index phase data stage GET object If the object does not exist or has an "origin modification time" then the delegate If the object has an "Origin Modified Time" then the delegate GET object ACL; HEAD object Proxy if object does not exist allow PUT Object ACL Proxy, and write to the local allow DELETE object Proxy, and delete local objects (if present) allow list objects acting allow upload object Proxy, and write to the local allow

Table 1 depicts some examples of simple behaviors of the reliable automated distributed object storage gateway actions of the storage server 10 in the index phase (e.g. INDEX phase) and data phase (e.g. DATA phase), where the requirements, such as GET object, GET object ACL, HEAD object, PUT object ACL, DELETE object, list object, and upload object (listed in the left half of Table 1), each representing one or more objects get request, get ACL request for one or more objects, request for one or more object headers, put ACL request for one or more objects, delete request for one or more objects, list one or more objects request, and one or more object data write-related requests (eg, one or more object placement requests, one or more create multi-component upload requests, and one or more complete multi-component upload requests). Under the control of the simple storage service migration management module 53 based on index objects, the storage server 10 can operate correctly and effectively. For better understanding, some operations described in the above embodiments can be briefly summarized in the table In the " Index Phase " and " Data Phase " fields of one.

For example, during the indexing phase, when any other request represents an object fetch request, the storage server 10 may proxy the request to the source if the target object does not exist or has a last modification time (e.g., source modification time) end; when any other request represents any one of the object acquisition access control list request and the object header request, if the target object does not exist, the storage server 10 can proxy this request to the source end; when any other request When placing an ACL request on behalf of an object, the storage server 10 may proxy the request to the source, so that it writes the ACL of the object carried by the request to a local location (for example, a destination, such as a storage server 10); when any other request represents an object deletion request, the storage server 10 may proxy the request to the source and, if the target object exists, delete the local object (such as the target object); when any other request represents The storage server 10 may proxy the request to the origin when an object listing is requested; and when any other request represents an object data write related request (such as an object placement request, create a multipart upload request, or complete a multipart upload request ), the storage server 10 can proxy the request to the source, and write the object data carried in the request to a local location (eg, the destination, such as the storage server 10 ). In the data phase, when any other request represents an object acquisition request, the storage server 10 may proxy the request if the target object has a last modification time (e.g., source modification time); For any of the remaining of these requests, the storage server 10 may allow the associated processing of the request without proxying the request to the origin.

Because the indexing phase takes only a short time, and because the index object-based simple storage service migration management module 53 can properly control the storage server 10 during user migration by means of various control schemes of the method, storage The server 10 can simply perform seamless migration of simple storage services for private clouds (for example, seamless migration of simple storage services from any public/private cloud to a private cloud), so as to allow users to normally And access data smoothly, under the control of the simple storage service migration management module 53 based on index objects, the storage server 10 can correctly migrate user data, and at the same time properly handle any other requests (if any), so Does not generate any erroneous data (for example due to incorrect migration management of related technologies, usually when the user changes (such as writes, overwrites, deletes, etc.) something during the migration Discovery), so as to make the migration of users seamless. For the sake of brevity, similar content in this embodiment will not be described in detail here. The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

50: Main device 52: Processing circuit 52P: Program module 53: Simple storage service migration management module based on index objects 54: storage interface circuit 56: storage device 58: Network interface circuit 90: Multiple storage devices 100: storage device 110: Memory controller 112: Microprocessor 112C: code 112M: read-only memory 114: Control logic circuit 116: random access memory 118: Transmission interface circuit 120: Non-volatile memory 122-1~122-N: Non-volatile memory components T1~T3: time point S01~S08,S10~S14,S20~S29,S30~S36,S40~S48,S50~58,S60~S65,S70~S75,S14A,S14B,S14C,S25A,S25B,S25C,S27A,S S27B,S27C ,S29E,S36A,S36B,S36C,S45A,S45B,S45C,S56A,S56B,S56C,S64A,S64B,S64C,S74A,S74B,S74C: step

FIG. 1 is a schematic diagram of a storage server according to an embodiment of the present invention. FIG. 2 is a schematic diagram of some implementation details of the storage device in the storage server shown in FIG. 1 according to an embodiment of the present invention. FIG. 3 is an index object-based migration control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 4 is a schematic diagram of some related operations of the migration control scheme based on index objects shown in FIG. 3 according to an embodiment of the present invention. FIG. 5 is a schematic diagram of some objects and some related features of the index object-based migration control scheme shown in FIG. 3 according to an embodiment of the present invention. FIG. 6 is a schematic diagram of a migration agent control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 7 is a schematic diagram of an example of a Ceph object storage architecture with an index object-based simple storage service migration management module shown in FIG. 1 . FIG. 8 is a schematic diagram of an object listing control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 9 is a schematic diagram of an object header control scheme of a method for seamless migration of simple storage services using index objects according to an embodiment of the present invention. FIG. 10 is a schematic diagram of an object acquisition control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 11 is a schematic diagram of an object acquisition ACL control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 12 is a schematic diagram of an object placement control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 13 is a schematic diagram of an object placement ACL control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention. FIG. 14 is a schematic diagram of an object deletion control scheme of a method for seamlessly migrating simple storage services using index objects according to an embodiment of the present invention.

T1~T3: time point

Claims (7)

A method for seamlessly migrating simple storage services using index objects, the method is applied to a storage server, the method includes: responding to a user migrating the storage server from a remote simple storage service compatible server A request of a user data of a user to the storage server utilizes multiple An index object-based simple storage service migration management module in the program module to create and store multiple index objects to a storage device layer of the storage server as the user data in the storage server Respective representations of a plurality of normal objects, and migrating respective ACLs of the plurality of normal objects to the storage server as respective ACLs of the plurality of indexed objects, to minimize the plurality of normal objects Any change in any normal object in the object affects the probability of the correctness of the migration of the user data, wherein the storage server includes the main device and the storage device layer, the storage device layer includes a device coupled to the At least one storage device of the main device, the main device is used to control the operation of the storage server, and the at least one storage device is used to store information for the storage server. The method described in item 1 of the scope of the patent application further includes: in a data stage after the index stage among the multiple stages, using the simple storage service migration management module based on index objects to migrate from the remote The client Simple Storage Service compatible server migrates the respective object data of a group of normal objects among the plurality of normal objects to the storage device layer of the storage server. The method described in item 1 of the scope of the patent application also includes: in response to migrating the user data from the remote simple storage service compatible server to the storage server According to the requirement, in a setting stage before the indexing stage among the multiple stages, the pre-processing of the migration of the user data is performed by using the index object-based simple storage service migration management module. A host device (host device), which is set in a storage server, includes: a processing circuit, used to control the host device to perform simple storage service migration management based on index objects in a storage server, wherein the storage The server includes the main device and a storage device layer, the storage device layer includes at least one storage device coupled to the main device, the main device is used to control the operation of the storage server, and the at least one storage The device is used to store information for the storage server, wherein: in response to a request to migrate a user data of a user of the storage server from a remote Simple Storage Service compatible server to the storage server, at During an indexing phase of one of the phases of migration of the user data, an index object-based simple storage service migration management module among the program modules running on the processing circuit creates and stores multiple indexes objects to the storage device layer as respective representations of the plurality of normal objects of the user data in the storage server, and migrate the respective access control lists of the plurality of normal objects to the storage server , as the respective access control lists of the plurality of index objects, so as to minimize the probability that any change of any normal object in the plurality of normal objects affects the correctness of the migration of the user data. The main device as described in item 4 of the scope of the patent application, wherein in a data stage after the index stage among the multiple stages, the index object-based simple storage service migration management module is transferred from the remote simple storage service A compatible server migrates a set of normal objects in the plurality of normal objects the respective object data of the object to the storage device layer of the storage server. A storage server includes: a main device for controlling the operation of the storage server, the main device includes: a processing circuit for controlling the main device to perform index-based object in the storage server Simple storage service migration management; and a storage device layer, including: at least one storage device, coupled to the main device, used to store information for the storage server; wherein: compatible with a remote simple storage service A request by the server to migrate a user data of a user of the storage server to the storage server is run on the processing circuit during an indexing phase of a plurality of phases of migration of the user data An index object-based simple storage service migration management module among a plurality of program modules creates and stores index objects to the storage device layer as regularities of the user data in the storage server objects, and migrate the respective ACLs of the plurality of normal objects to the storage server as the respective ACLs of the plurality of indexed objects to minimize the plurality of normal objects The probability that any change in any of the normal objects affects the correctness of the migration of the user data. The storage server as described in item 6 of the scope of the patent application, wherein in a data stage after the index stage among the multiple stages, the index object-based simple storage service migration management module is transferred from the remote simple storage The service compatible server migrates the respective object data of a group of normal objects among the plurality of normal objects to the storage device layer of the storage server.

Images