WO2018045758A1 - Procédé de stockage de données et centre de données - Google Patents

Procédé de stockage de données et centre de données Download PDF

Info

Publication number
WO2018045758A1
WO2018045758A1 PCT/CN2017/081340 CN2017081340W WO2018045758A1 WO 2018045758 A1 WO2018045758 A1 WO 2018045758A1 CN 2017081340 W CN2017081340 W CN 2017081340W WO 2018045758 A1 WO2018045758 A1 WO 2018045758A1
Authority
WO
WIPO (PCT)
Prior art keywords
version
data
history queue
version number
data center
Prior art date
Application number
PCT/CN2017/081340
Other languages
English (en)
Chinese (zh)
Inventor
白平昌
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2018045758A1 publication Critical patent/WO2018045758A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a data storage method and a data center.
  • cloud storage is a cloud computing system with data storage and management as its core. Users can connect to the cloud to access data through networkable terminals at any time and any place.
  • object storage is the core of cloud storage, also known as object-based storage.
  • the biggest feature of the object storage system is that the object name is a domain name address. Once the domain name is set to be public, everyone can access the object.
  • a multi-data center solution can be adopted.
  • each data center can process the client's business request, that is, the object can be uploaded to any data center, or can be downloaded to the same object from any data center. After the object is uploaded to any data center, it is necessary to ensure that the data of the object finally stored in each data center is the same data, so that the same data can be downloaded when the object is downloaded from any data center. Therefore, when there is data of the uploaded object in the object storage system, a round of resolution process is started. If accepted by most data centers, each data center saves the data of the object uploaded this time, thereby making any one of the data When the data center downloads the object, it can download the same data. However, reaching a round of resolutions requires multiple network interactions in various data centers, resulting in large network overhead.
  • the embodiment of the invention provides a data storage method and a data center, which can reduce network overhead.
  • a first aspect of the embodiments of the present invention provides a data storage method, which is applied to an object storage system including a plurality of data centers, and includes: when the data center stores the data of the object, the first version of the version number of the object in the local end is read. The second version of the history queue; the data center determines whether the version number of the data of the object stored in the first version history queue is greater than the version number in the second version history queue; if the version number in the second version history queue is greater than the version number, the data is The center determines that no optimistic lock conflict has occurred, and the data storage of the object stored this time is successful.
  • the version number includes a receiving time and a universal unique identifier UUID of the storage request for receiving data for storing the object sent by the client; the version number after the receiving time is greater than the version number of the receiving time, and the receiving time is the same.
  • the version number of the UUID is larger than the version number of the UUID.
  • the UUID is a Universally Unique Identifier (UUID), which is unique in a distributed system, that is, in each data center, and is identified by the current date, time, clock sequence, and globally unique IEEE machine. Number composition.
  • the first part of the UUID is related to time. For example, after a UUID is generated by the same data center, a UUID is generated after a few seconds, and the first part is different, and the others are the same. Even if different data centers receive the same two storage requests for the data of the object, the universal unique identifier is different. Therefore, the receiving time in the version number may be the local time of the data center that receives the storage request, and need not be Full The precise timing of the atomic clock determined by the precise synchronization of the ball time. This embodiment reduces the difficulty of implementing the data storage method and saves the cost of data storage because the data center does not strictly require the atomic clocks of the data centers to be synchronized with the global time to ensure the same time.
  • each storage request corresponds to a version number
  • the version number can be used to distinguish the old and new relationship of the data of the object corresponding to different storage requests
  • the version number of the data of the object stored in each data center in the object storage system is greater than
  • the existing version number of the object can be stored successfully, so that the data of the object stored in each data center is the latest data of the object received by the object storage system, and the data of the object in each data center is consistent.
  • the data center may receive the copy request sent by the other data center; The request extracts the data of the object and the first version history queue of the object; in response to the copy request, the data of the object is stored.
  • This embodiment can also synchronize the data of the objects received by other data centers into the data center.
  • the data center receives the storage request of the object before reading the second version history queue formed by the version number of the object existing in the local end;
  • the data center generates a version number of the data of the object to be stored for the storage request;
  • the data center adds the version number to the tail of the version history queue of the object, constitutes the first version history queue of the object;
  • the data center responds to the storage request, and stores The object's data, and sends a copy request to the other data center, the copy request carries the first version of the object's historical queue.
  • This embodiment can not only store the data of the object in the storage request received by the data center, but also send a copy request to other data centers, so that other data can also synchronously store the data of the object according to the copy request.
  • the data center is determined in the first version history queue
  • the version number of the data of the stored object is not greater than the version number in the history queue of the second version, and it can be determined that an optimistic lock conflict occurs, and the data storage of the object stored this time fails; then, the data center can compare the first version of the historical queue.
  • the third version history queue is merged with the version number in the second version history queue; the third version history queue is stored to update the version history queue of the local object.
  • the data center may be in an optimistic lock conflict, and the data of the object to be stored is determined to be expired data.
  • the first version of the historical queue and the local end may still be used.
  • the second version history queue formed by the existing version number of the object is merged to obtain a third version history queue to update the version history queue of the object in the local end.
  • the storage request triggers the data center storage data or the replication request triggers the data center to store the data
  • the data center determines that no optimistic lock conflict occurs
  • the first version history queue is the current latest version history queue, and the data center can
  • the first version history queue is stored as a version history queue for the object to be used when the data of the object is updated next time.
  • the data center compares the version number of the first version of the historical queue with the second version of the historical queue, and aligns the third version of the historical queue, which may be specifically: the order of the data center from the tail to the head, and determining the first version of the historical queue and the first Whether the first version number is the same in the second version history queue; if there is the first version number, the data center will match the first version history queue with the first version number in the second version history queue. All version numbers of the tail are sorted, and the third version history queue is merged; if the first version number is not present, the data center sorts the first version history queue and all version numbers in the second version history queue. The third version history queue is merged.
  • a second aspect of the embodiments of the present invention provides a data center, where the data center has a behavior function for implementing the method provided by the foregoing first aspect, and the function may be implemented by using hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the functions described above.
  • FIG. 1 is a schematic diagram of an object storage system according to an embodiment of the present invention.
  • FIG. 2 is a schematic flowchart of a data storage method according to an embodiment of the present invention.
  • FIG. 3 is a schematic structural diagram of a data center disclosed in an embodiment of the present invention.
  • FIG. 4 is another schematic structural diagram of a data center according to an embodiment of the present invention.
  • FIG. 1 is a schematic diagram of an object storage system according to an embodiment of the present invention.
  • the object storage system may include multiple data centers.
  • two data centers DC1 and DC2 are taken as an example, and each data center may include a load balancing module (Load Balancing, LB), an object storage subsystem, and a KV (Key-Value) subsystem, wherein the load balancing subsystem is configured to allocate storage requests of client Client A and Client B to respective object storage subsystems to avoid storage of an object.
  • load balancing subsystem is configured to allocate storage requests of client Client A and Client B to respective object storage subsystems to avoid storage of an object.
  • each data center can receive a storage request uploaded by a client, and the data storage method provided by the embodiment of the present invention can be used in each data center of the object storage system, regardless of which data center is uploaded from the data storage system.
  • FIG. 2 is a data storage method according to an embodiment of the present invention. As shown in FIG. 2, the data storage method includes the following steps:
  • the data center stores the data of the object, and reads a second version history queue formed by the version number of the object existing in the local end;
  • the data center determines whether the version number of the data of the object stored in the first version history queue is greater than The version number in the second version of the historical queue; if it is greater than the version number in the second version history queue, step S103 is performed; if it is not greater than the version number in the second version history queue, the process ends.
  • the data center determines that an optimistic lock conflict is not generated, and the data storage of the object stored this time is successful.
  • the execution body of the steps S101 to S103 may also be an object storage subsystem of the data center or a server of the data center, etc., which is not limited by the embodiment of the present invention.
  • the version number includes a receiving time and a universal unique identification code UUID of the storage request for receiving the data of the storage object sent by the client; the version number after the receiving time is greater than the version number of the receiving time first.
  • the version number with a large UUID is larger than the version number with a smaller UUID.
  • the different storage requests of the object correspond to different data of the object, and the different data of the object corresponds to different version numbers. Therefore, the data of the object corresponding to each storage request can be distinguished by the version number, and different storage can be determined according to the version number. Request new and old relationships for different data of the corresponding object.
  • the UUID is a Universally Unique Identifier (UUID), which is unique in a distributed system, that is, in each data center, and is identified by the current date, time, clock sequence, and globally unique IEEE machine. Number composition.
  • the first part of the UUID is related to time. For example, after a UUID is generated by the same data center, a UUID is generated after a few seconds, and the first part is different, and the others are the same. Even if different data centers receive the same two storage requests for the data of the object, the universal unique identifier is different. Therefore, the receiving time in the version number may be the local time of the data center that receives the storage request, and need not be Precise timing with precise time synchronization of atomic clocks around the world. This embodiment reduces the difficulty of implementing the data storage method and saves the cost of data storage because the data center does not strictly require the atomic clocks of the data centers to be synchronized with the global time to ensure the same time.
  • the method may further include the following steps:
  • the data center receives the copy request sent by other data centers;
  • the data center extracts the data of the object from the copy request and the first version history queue of the object
  • the data center responds to the copy request and stores the data of the object.
  • the stored data comes from data synchronized by other data centers, thereby ensuring data consistency in each data center.
  • the data of the data center storage object may also be the data of the object corresponding to the storage request sent by the client to the data center, and may include the following steps:
  • the data center receives the storage request of the object
  • the data center generates a version number of the object for the storage request
  • the data center adds the version number to the end of the object's version history queue, forming the first version of the object. This historical queue;
  • the data center responds to the storage request, stores the data of the object, and sends a copy request to the other data center.
  • the copy request is used to synchronously write data of an object corresponding to the storage request to another data center, where the copy request carries a first version history queue of the object.
  • the version number includes a receiving time and a UUID of the storage request corresponding to the version number; the receiving time is a receiving time of the storage request corresponding to the data of the object to be stored in the data center.
  • the version history queue of the object when the data center stores the data of the object, if the object does not exist, the version history queue of the object is empty, NULL; if the object already exists, the version history queue of the object is not empty.
  • the version number generated by the data center for the storage request is V2. If the version history queue of the data center before the storage request is NULL, the first version history queue is NULL->V2; if the data center is in the storage The version history queue before the request is NULL->V1, then the first version history queue is NULL->V1->V2.
  • the version number in the version history queue located near the head is smaller than the version number located near the end of the queue, and the newly generated version number can be directly added to the tail of the version history pair column to form the first version history queue.
  • the data center when the data center receives the storage request, the data center generates a version number for the object to be stored, and adds the version number to the data center that has been stored or already exists.
  • the end of the version history queue of the object is formed; the second version history queue is a version history queue of the object stored or existing in the data center.
  • the data center DC1 may generate a version number of the data of the object to be stored for the storage request, and add the version number to the tail of the version history queue of the object in DC1 to form the first version.
  • the history queue when the data center DC2 receives the copy request sent by the DC1, the first version history queue can be extracted, and the second version history queue of the version number of the object already existing in the DC2 is read, and the DC2 is based on the first
  • the version history queue and the second version history queue store data of the object corresponding to the storage request.
  • the data storage method may further include the following steps:
  • the data center determines to generate an optimistic lock conflict, and the data storage of the object stored in the current storage fails;
  • the data center compares the version numbers of the first version history queue with the second version history queue, and merges and lists the third version history queue;
  • the data center stores the third version history queue to update the version history queue of the object in the local end.
  • the data center stores the third version history queue, so that when the data of the object is updated next time, the third version history queue is used as the version history queue of the object that already exists in the data center.
  • scenario 1 is: the data center DC1 receives the storage request of the object, and the generated version number is V1; if the version history queue of the object that already exists in DC1 is NULL, the first version history queue is NULL->V1.
  • the second version history queue read by DC1 is NULL, and DC1 determines the history of the first version in the column.
  • the version number V1 is greater than the NULL in the second version history queue. Therefore, DC1 determines that no optimistic lock conflict is generated, so DC1 can successfully store the data of the object corresponding to the version number V1.
  • the scenario 2 is: the data center DC1 receives the copy request sent by the data center DC2, and the copy request is used to synchronously write the data of the object corresponding to the storage request received by the DC2 to the DC1, where the copy request carries the object.
  • the first version history queue is NULL->V1
  • the version history queue of the object already existing by DC1 is NULL, wherein the version number V1 is generated by DC2 for the data of the object corresponding to the storage request it receives.
  • the second version history queue read by DC1 is NULL.
  • DC1 determines that the version number V1 in the first version history pair column is greater than the NULL value in the second version history queue. Therefore, DC1 determines that no optimistic lock conflict occurs. The data of the object corresponding to the version number V1 is successfully stored.
  • the scenario 3 is: the data center DC1 receives two storage requests of the same object, and the version numbers generated for the two storage requests are respectively V1 and V2, wherein the storage request receiving time corresponding to the version number V1 is received. Prior to the receiving time of the storage request corresponding to the version number V2, the version number V1 is smaller than the version number V2, and the version history queue of the object already existing by DC1 is NULL.
  • DC1 first stores the data of the object corresponding to the version number V1, and determines that the first version history queue is NULL->V1, and the read second version history queue is NULL, and DC1 determines the first version history pair column.
  • the medium version number V1 is greater than the NULL in the second version history queue. Therefore, the DC1 determines that the optimistic lock conflict is not generated, and the DC1 can successfully store the data of the object corresponding to the version number V1.
  • the DC1 stores the object corresponding to the version number V2.
  • the data determines that the first version history queue is NULL->V1->V2, and the read second version history queue is NULL->V1, and DC1 determines that the version number V2 in the first version history is greater than the second version history.
  • the version number V1 in the queue therefore, DC1 determines that no optimistic lock conflict is generated, and DC1 can successfully store the data of the object corresponding to the version number V2.
  • the data center DC1 receives the storage request of the object, and the generated version number is V3.
  • the version history queue of the object that already exists in DC1 is NULL->V1->V2, and the first version history queue is NULL->V1->V2->V3.
  • the second version history queue read by DC1 is NULL->V1->V2, and DC1 determines that the version number V3 in the first version history pair column is greater than each version number in the second version history queue, therefore, DC1 determines that no optimistic lock conflict is generated, and DC1 can successfully store the data of the object corresponding to version number V3.
  • the scenario 5 is: the data center DC1 receives the copy request sent by the data center DC2, and the copy request is used to synchronously write the data of the object corresponding to the storage request received by the DC2 to the DC1, where the copy request carries the object.
  • the first version history queue is NULL->V1->V2
  • the version history queue of the object already existing by DC1 is NULL->V1, wherein the version number V2 is generated by DC2 for the data of the object corresponding to the storage request it receives. .
  • the second version history queue read by DC1 is NULL->V1
  • DC1 determines that the version number V2 of the tail of the first version history column is greater than the V1 of the tail of the second version history queue, therefore, DC1 is determined.
  • the optimistic lock conflict is not generated, and DC1 can successfully store the data of the object corresponding to the version number V2.
  • scenario 6 is: the data center DC1 receives the storage request of the object, and the generated version number is V1; meanwhile, the data center DC2 receives the storage request of the same object, and the generated version number is V2, and the DC1 and DC2 are already
  • the existing version history queue of the object is NULL, and the version number V1 is greater than the version number V2 (ie, the data center DC1 and the data center DC2 receive the storage request at the same time, but the UUID is different in the version number generated by the two, V1 UUID is greater than the UUID of V2).
  • the first version history queue is NULL->V1
  • the second version history queue is NULL
  • the first version is determined according to the version number comparison rule.
  • the V1 in the history queue is larger than the NULL in the history queue of the second version, and it is determined that the optimistic lock conflict is not generated.
  • DC1 stores the data of the object corresponding to the version number V1 successfully.
  • the DC1 receives the copy request sent by the DC2, and stores the version number V2.
  • the data of the corresponding object is the NULL->V2 of the first version.
  • the history queue of the second version is NULL->V1 (that is, the data of the object corresponding to the successful version number V1 is stored by DC1).
  • the queue is the first version history queue NULL->V1 when storing the data of the object corresponding to the version number V1. Since V2 in the first version history queue is smaller than V1 in the second version history queue, it is determined that an optimistic lock conflict occurs, DC1 The data of the object corresponding to the version number V2 fails to be stored, and the data corresponding to the version number V2 is expired data. Therefore, the order of the DC1 from the tail to the head determines the first version of the historical queue and the second.
  • the same version number does not exist in the history queue; all the version numbers in the first version history queue and the second version history queue are sorted, that is, the version number V2 and the version number V1 are sorted; the version number V1 is greater than the version number V2, Therefore, the third version history queue is NULL->V2->V1, and DC1 stores the third version history queue to update the version history queue of the object to NULL->V2->V1.
  • scenario 7 is: the data center DC1 receives the storage request of the object, and the generated version number is V1; meanwhile, the data center DC2 receives the storage request of the same object, and the generated version number is V2, and the DC1 and DC2 have already been generated.
  • the existing version history queue of the object is NULL, and the version number V1 is smaller than the version number V2 (that is, although the data center DC1 and the data center DC2 receive the storage request at the same time, the UUID in the version number generated by the two is different, V1 UUID is less than the UUID of V2).
  • the first version history queue is NULL->V1
  • the second version history queue is NULL
  • the first version history queue has V1 greater than the second version.
  • NULL in the history queue determines that no optimistic lock conflict is generated.
  • DC1 stores the data of the object corresponding to version number V1 successfully.
  • DC1 receives the copy request sent by DC2, and stores the data of the object corresponding to version number V2.
  • the version history queue of the version is NULL->V2.
  • the history queue of the second version is NULL->V1 (that is, when the data of the object corresponding to the successful version number V1 is stored by DC1, the version history queue of the object is the storage version number V1.
  • the first version history queue of the corresponding object data is NULL->V1); V2 in the first version history queue is larger than V1 in the second version history queue, and it is determined that no optimistic lock conflict is generated, and DC1 will be the object corresponding to the version number V2.
  • the data was stored successfully.
  • DC1 storage data triggered by the storage request or the DC1 storage data triggered by the replication request if DC1 determines that no optimistic lock conflict is generated, it indicates that the first version history queue is the current latest version history queue, and DC1 can The first version history queue is stored as a version history queue for the object to be used when the data of the object is updated next time.
  • the data storage method shown in FIG. 2 can read the existing version of the object in the local end when storing the data of the object.
  • the second version of the history queue is configured by the data center; the data center determines whether the version number of the data stored in the first version of the historical queue is greater than the version number in the history queue of the second version; The version number, the data center determines that the optimistic lock conflict is not generated, and the stored data of the object is successfully stored. It can be seen that the version number of the data of the object stored in the present embodiment is greater than the version of the object that already exists. When the number is stored, the data of the object stored in this storage is successfully stored, and the network overhead is greatly reduced compared with the method that needs to achieve a round of resolution for storage.
  • each storage request corresponds to a version number
  • the version number can be used to distinguish the old and new relationship of the data of the object corresponding to different storage requests
  • the version number of the data of the object stored in each data center in the object storage system is greater than
  • the existing version number of the object can be stored successfully, so that the data of the object stored in each data center is the latest data of the object received by the object storage system, and the data of the object in each data center is consistent.
  • the embodiment of the present invention takes the object storage system in FIG. 1 as an example, and describes the data storage method in detail, which may include the following steps:
  • the client Client A sends a storage request to the data center DC1, and the storage request is used to store the data 1 of the object Obj;
  • DC1 receives the storage request, and the load balancing module 1 in DC1 allocates the storage request to the object storage subsystem 11 in DC1 for processing;
  • the object storage subsystem 11 generates a version number V1 of the data 1 of the object Obj for the storage request;
  • the object storage subsystem 11 reads the version history queue of the object Obj in DC1 to be NULL;
  • the object storage subsystem 11 adds the version number V1 to the tail of the version history queue of the object Obj, and the first version history queue constituting the object Obj is NULL->V1;
  • the object storage subsystem 11 calls the data 1 of the KV subsystem 1 storage object Obj in DC1, and sends a copy request to the data center DC2, the copy request carrying the first version history queue NULL->V1 and the data 1 of the object Obj;
  • the object storage subsystem 11 calls the data 1 of the KV subsystem 1 storage object Obj in DC1
  • the optimistic lock conflict returned by the KV subsystem 1 is not received, and the data 1 of the object Obj is determined to be successfully stored, and at this time, the object Obj of the DC1 is
  • the version history queue is NULL->V1;
  • the data center DC2 receives the copy request sent by the object storage subsystem 11, and the load balancing module 1 in DC2 allocates the copy request to the object storage subsystem 21 in DC2 for processing;
  • the object storage subsystem 21 extracts the data 1 of the object Obj from the copy request and the first version history queue of the object Obj is NULL->V1;
  • the object storage subsystem 21 calls the KV subsystem 2 in DC2 to respond to the copy request, and stores the data 1 of the object Obj;
  • the second version history queue formed by reading the version number of the object Obj already existing in the DC2 is NULL;
  • the KV subsystem 2 determines that the version number V1 of the data 1 of the object Obj in the first version history queue is greater than the empty second version history queue, and determines that no optimistic lock conflict is generated;
  • the object storage subsystem 21 does not receive the optimistic lock conflict returned by the KV subsystem 2, and determines that the data 1 of the object Obj is successfully stored, and at this time, the version history queue of the object Obj in DC2 is NULL->V1;
  • the client Client B sends a storage request to the data center DC2 for storing the data 2 of the object Obj; meanwhile, the client Client A sends a storage request to the data center DC1 for storing the object Obj.
  • DC2 receives the storage request, and the load balancing module 2 in DC2 allocates the storage request to the object storage subsystem 21 in DC2 for processing;
  • DC1 receives the storage request, and the load balancing module 1 in DC1 allocates the storage request to the object storage in DC1.
  • the object storage subsystem 21 generates a version number V2 of the data 2 of the object Obj for the storage request; the object storage subsystem 11 generates a version number V3 of the data 3 of the object Obj for the storage request; wherein the reception time in the version number V2 is The receiving time in the version number V3 is the same, but the UUID in the version number V2 is smaller than the UUID in the version number V3;
  • the object storage subsystem 21 reads the version history queue of the object Obj in DC2 as NULL->V1; the object storage subsystem 21 adds the version number V2 to the end of the version history queue of the object Obj, constituting the first version history of the object Obj
  • the queue is NULL->V1->V2; the object storage subsystem 11 reads the version history queue of the object Obj in DC1 to be NULL->V1; the object storage subsystem 11 adds the version number V3 to the team of the version history queue of the object Obj.
  • the first version of the historical queue constituting the object Obj is NULL->V1->V3;
  • the object storage subsystem 21 calls the data 2 of the KV subsystem 2 in the DC2 storage object Obj, and sends a copy request to the data center DC1, which carries the data of the first version history queue NULL->V1->V2 and the object Obj. 2; the object storage subsystem 11 calls the data 3 of the KV subsystem 1 storage object Obj in DC1, and sends a copy request to the data center DC2, the copy request carrying the first version history queue NULL->V1->V3 and the object Obj Data 3;
  • the object storage subsystem 21 calls the data 2 of the object Vb subsystem 2 in the DC2 storage object Obj
  • the optimistic lock conflict returned by the KV subsystem 2 is not received, and the data 2 of the object Obj is determined to be successfully stored, and at this time, the object Obj of the DC2 is
  • the version history queue is NULL->V1->V2; when the object storage subsystem 11 calls the data 3 of the KV subsystem 1 storage object Obj in DC1, the optimistic lock conflict returned by the KV subsystem 1 is not received, and the data of the object Obj is determined. 3
  • the version history queue of the object Obj in DC1 is NULL->V1->V3;
  • the data center DC2 receives the copy request sent by the object storage subsystem 11, the load balancing module 1 in DC2 assigns the copy request to the object storage subsystem 21 in DC2; the data center DC1 receives the copy request sent by the object storage subsystem 21. , the load balancing module 1 in DC1 assigns the copy request to the object storage subsystem in DC1 11 processing;
  • the object storage subsystem 21 extracts the data 3 of the object Obj from the copy request and the first version history queue of the object Obj is NULL->V1->V3; the object storage subsystem 11 extracts the data of the object Obj from the copy request 2 And the first version history queue of the object Obj is NULL->V1->V2;
  • the object storage subsystem 21 calls the KV subsystem 2 in DC2 to respond to the copy request, and stores the data 3 of the object Obj;
  • the object storage subsystem 11 calls the KV subsystem 1 in DC2 to respond to the copy request, and stores the data 2 of the object Obj;
  • the second version history queue formed by reading the version number of the object Obj already existing in DC2 is NULL->V1->V2
  • the KV subsystem 1 stores the data 2 of the object Obj Reading the second version history queue formed by the version number of the object Obj existing in DC1 is NULL->V1->V3;
  • the KV subsystem 2 determines that the version number V3 of the data 3 of the object Obj in the first version history queue is greater than the version number of the second version history queue NULL->V1->V2, and determines that no optimistic lock conflict is generated; the KV subsystem 1 determining that the version number V2 of the data 2 of the object Obj in the first version history queue is not greater than V3 of the second version history queue ULL->V1->V3, determining that an optimistic lock conflict is generated;
  • the object storage subsystem 21 does not receive the optimistic lock conflict returned by the KV subsystem 2, and determines that the data 3 of the object Obj is successfully stored, and at this time, the version history queue of the object Obj in DC2 is NULL->V1->V2->V3;
  • the object storage subsystem 11 receives the optimistic lock conflict returned by the KV subsystem 1, and determines that the data 2 storage of the object Obj fails.
  • the object storage subsystem 11 compares the first version history queue NULL->V1->V2 with the second.
  • the version number in the version history queue NULL->V1->V3 merges the third version history queue NULL->V1->V2->V3; the object storage subsystem 11 stores the third version history queue as the object Obj Version history queue in DC1.
  • the data of the object Obj finally stored in DC1 and DC2 is data 3; the version history queue of the object Obj finally stored is NULL->V1->V2->V3.
  • the data storage method does not need to perform network resolution between the data centers DC1 and DC2, and each data center can achieve consistent data of the stored objects, thereby reducing network overhead.
  • the receiving time of the storage request in the version number is the local time of the data center that receives the storage request, and it is not necessary to accurately time the atomic clock of the global time precision synchronization through GPS, thereby reducing the difficulty of implementing the solution. And cost.
  • FIG. 3 is a schematic structural diagram of a data center according to an embodiment of the present invention.
  • the data center may include the following modules:
  • the reading module 310 is configured to: when storing the data of the object, read the second version history queue formed by the version number of the object already existing in the local end;
  • the determining module 320 is configured to determine whether the version number of the data of the object stored in the first version history queue is greater than the version number in the second version history queue;
  • the determining module 330 is configured to determine, when the determining module 320 determines that the version number of the data of the object stored in the first version historical queue is greater than the version number in the historical queue of the second version, determining that no optimistic lock conflict occurs.
  • the stored data of the object is successfully stored;
  • the version number includes a receiving time and a universal unique identification code UUID of the storage request for receiving the data of the storage object sent by the client; the version number after the receiving time is greater than the version number of the receiving time first.
  • the version number with a large UUID is larger than the version number with a smaller UUID.
  • the different storage requests of the object correspond to different data of the object, and different storage requests also correspond to different version numbers. Therefore, the data of the object corresponding to each storage request can be distinguished by the version number, and different storage can be determined according to the version number. Request new and old relationships for different data of the corresponding object.
  • the UUID is a Universally Unique Identifier (UUID), which is unique in a distributed system, that is, in each data center, and is identified by the current date, time, clock sequence, and globally unique IEEE machine. Number composition.
  • the first part of the UUID is related to time. For example, after a UUID is generated by the same data center, a UUID is generated after a few seconds, and the first part is different, and the others are the same. Even if different data centers receive the same two storage requests for the data of the object, the universal unique identifier is different. Therefore, the receiving time in the version number may be the local time of the data center that receives the storage request, and need not be Precise timing with precise time synchronization of atomic clocks around the world. This embodiment reduces the difficulty of implementing the data storage method and saves the cost of data storage because the data center does not strictly require the atomic clocks of the data centers to be synchronized with the global time to ensure the same time.
  • the data center shown in FIG. 3 further includes:
  • the first receiving module 340 is configured to receive a copy request sent by another data center before the reading module 310 reads the second version history queue formed by the version number of the object existing in the local end;
  • the extracting module 350 is configured to extract data of the object and a first version history queue of the object from the copy request;
  • the first storage module 360 is configured to store data of the object in response to the copy request.
  • the data center shown in FIG. 3 further includes:
  • the second receiving module 370 is configured to receive a storage request of the object before the reading module 310 reads the second version history queue formed by the version number of the object existing in the local end;
  • a generating module 380 configured to generate, according to the storage request, a version number of the data of the object to be stored this time;
  • the adding module 390 is configured to add a version number to the tail of the version history queue of the object, and constitute a first version history queue of the object;
  • the second storage module 390a is configured to store the data of the object in response to the storage request, and send a copy request to the other data center, where the copy request carries the first version history queue of the object.
  • the first receiving module 340 and the second receiving module 370 may be the same receiving module and perform different operations.
  • the determining module 330 is further configured to: when the determining module 320 determines that the version number of the data of the object stored in the first version history queue is not greater than the version number in the history queue of the second version, Determining that an optimistic lock conflict is generated, and the data storage of the object stored this time fails;
  • data center shown in Figure 3 further includes:
  • the comparison module 390b is configured to compare the version numbers of the first version history queue and the second version history queue, and merge and arrange the third version history queue;
  • the third storage module 390c is further configured to store the third version history queue to update the version history queue of the object in the local end.
  • the first storage module 360, the second storage module 390a, and the third storage module 390c may be the same storage module, and respectively perform different operations.
  • the comparison module 390b may include the following units:
  • a determining unit configured to determine, from the tail of the team to the order of the head, whether the first version number is the same in the first version history queue and the second version history queue;
  • a merging unit configured to: when the determining unit determines that the first version of the historical queue and the second version of the historical queue have the same version number, the first version of the historical queue is the same as the first version of the second version of the historical queue All the version numbers of the number to the end of the team are sorted, and the third version history queue is merged; the merging unit is further configured to determine, in the determining unit, that the first version of the historical queue does not have the first version number in the second version history queue. The first version history queue is sorted with all version numbers in the second version history queue, and the third version history queue is merged.
  • the first version history queue is the current latest version history queue.
  • the data center may store the first version history queue as the version history queue of the object, so as to be used when the data of the object is updated next time.
  • the reading module can read the second version history queue formed by the version number of the object existing in the local end when storing the data of the object; the determining module can determine the first version history queue. Whether the version number of the data stored in the object is greater than the version number in the second version history queue; the determining module is configured to determine, in the determining module, the version of the data of the object stored in the first version history queue When the number is greater than the version number in the history queue of the second version, it is determined that the optimistic lock conflict is not generated, and the data storage of the object stored in this time is successful; it can be seen that the version number of the data of the object that can be stored in the embodiment of the present invention can be If the version number of the object is greater than the version number of the object, the data stored in the object is successfully stored.
  • each storage request corresponds to a version number
  • the version number can be used to distinguish the old and new relationship of the data of the object corresponding to different storage requests
  • the version number of the data of the object stored in each data center in the object storage system is greater than
  • the existing version number of the object can be stored successfully, so that the data of the object stored in each data center is the latest data of the object received by the object storage system, thereby ensuring the data of the object in each data center.
  • FIG. 5 is another schematic structural diagram of a data center according to an embodiment of the present invention.
  • the data center includes a processor 410, a memory 420, and a transceiver 430.
  • the transceiver 430 is configured to transmit and receive data with and from an external device.
  • the number of processors 410 in the data center can be one or more.
  • processor 410, memory 420, and transceiver 430 may be coupled by a bus system or other means.
  • the data center can be used to perform the method shown in Figure 2. For the meanings and examples of the terms involved in the embodiment, reference may be made to the corresponding method embodiments described above. I will not repeat them here.
  • the program code is stored in the memory 420, and the processor 410 is configured to call the program code stored in the memory 420 for performing the following operations:
  • the version number includes a receiving time and a universal unique identifier UUID of the storage request for receiving data for storing the object sent by the client; the version number after the receiving time is greater than the version number of the receiving time, and the receiving time is the same
  • the version number of the UUID is larger than the version number of the UUID.
  • the processor 410 is configured to call the program code stored in the memory 420, and before the data center reads the second version history queue formed by the version number of the object existing in the local end, the following operations may be performed:
  • the data of the object is stored in response to the copy request.
  • the processor 410 is configured to call the program code stored in the memory 420, and before performing the second version history queue formed by the version number of the object existing in the local end, the following operations may be performed:
  • data of the object is stored and a copy request is sent to other data centers via the transceiver 430, the copy request carrying a first version history queue of the object.
  • the processor 410 is configured to invoke the program code stored in the memory 420, and determine that the version number of the data of the object stored in the first version history queue is not greater than the second version history queue.
  • the version number in , you can also do the following:
  • the third version history queue is stored to update the version history queue of the object at the local end.
  • the processor 410 is configured to invoke the program code stored in the memory 420, compare the version numbers in the first version history queue and the second version history queue, and combine and arrange the third version history queue.
  • the processor 410 is configured to invoke the program code stored in the memory 420, compare the version numbers in the first version history queue and the second version history queue, and combine and arrange the third version history queue.
  • the first version history queue and all the version numbers in the second version history queue are sorted to merge the third version history queue.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne in procédé de stockage de données et un centre de données, le procédé de stockage de données comportant les étapes suivantes: un centre de données lit une deuxième file d'attente d'historique de versions composée de numéros de version stockés par un objet dans le présent terminal (S101); le centre de données détermine si un numéro de version de données de l'objet actuellement stocké dans une première file d'attente d'historique de versions est supérieur au numéro de version figurant dans la deuxième file d'attente d'historique de versions (S102); si ledit numéro de version est supérieur au numéro de version figurant dans la deuxième file d'attente d'historique de versions, le centre de données détermine qu'il n'en résulte pas un conflit de verrouillage optimiste, et que le stockage actuel des données de l'objet a réussi (S103). Ainsi, lorsque le numéro de version de données d'un objet actuellement en cours de stockage est supérieur à un numéro de version déjà stocké par l'objet, le présent procédé peut stocker avec succès les données de l'objet actuellement en cours de stockage, réduisant considérablement les surcharges de réseau en comparaison des procédés imposant aux centres de données d'aboutir à une résolution pour le stockage.
PCT/CN2017/081340 2016-09-08 2017-04-21 Procédé de stockage de données et centre de données WO2018045758A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610810735.XA CN106484321A (zh) 2016-09-08 2016-09-08 一种数据存储方法及数据中心
CN201610810735.X 2016-09-08

Publications (1)

Publication Number Publication Date
WO2018045758A1 true WO2018045758A1 (fr) 2018-03-15

Family

ID=58273617

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/081340 WO2018045758A1 (fr) 2016-09-08 2017-04-21 Procédé de stockage de données et centre de données

Country Status (2)

Country Link
CN (1) CN106484321A (fr)
WO (1) WO2018045758A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111901420A (zh) * 2020-07-28 2020-11-06 深圳市康冠科技股份有限公司 一种数据同步方法、装置及系统
CN114598559A (zh) * 2021-07-22 2022-06-07 湖南亚信软件有限公司 数据处理方法、装置、电子设备及计算机可读存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106484321A (zh) * 2016-09-08 2017-03-08 华为数字技术（成都）有限公司 一种数据存储方法及数据中心
CN107153912A (zh) * 2017-04-11 2017-09-12 广州市食蚁兽网络技术有限公司 一种成长数据智能分析系统
CN108769172A (zh) * 2018-05-21 2018-11-06 杭州有赞科技有限公司 一种数据同步方法及系统
CN109088913B (zh) * 2018-06-29 2021-05-11 华为技术有限公司 请求数据的方法和负载均衡服务器
CN109032527B (zh) * 2018-07-27 2021-07-27 深圳华大北斗科技有限公司 数据处理方法、存储介质及计算机设备
CN109634526B (zh) * 2018-12-11 2022-04-22 浪潮(北京)电子信息产业有限公司 一种基于对象存储的数据操作方法及相关装置
CN110597673B (zh) * 2019-09-25 2021-04-30 腾讯科技（深圳）有限公司 存储系统的容灾方法、装置、设备及计算机可读存储介质
CN111400283B (zh) * 2020-03-19 2024-02-06 中国建设银行股份有限公司 一种数据处理方法、系统、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102279857A (zh) * 2010-06-11 2011-12-14 阿里巴巴集团控股有限公司 一种实现数据复制的方法及系统
CN102622284A (zh) * 2012-02-21 2012-08-01 上海交通大学 面向海量存储系统的数据异步复制方法
CN104753987A (zh) * 2013-12-26 2015-07-01 北京东方通科技股份有限公司 一种分布式会话管理方法及系统
US20160140194A1 (en) * 2014-11-13 2016-05-19 Bigtera Limited Data migration system and data migration method thereof
CN106484321A (zh) * 2016-09-08 2017-03-08 华为数字技术（成都）有限公司 一种数据存储方法及数据中心

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7296125B2 (en) * 2001-11-29 2007-11-13 Emc Corporation Preserving a snapshot of selected data of a mass storage system
EP2851803A4 (fr) * 2012-05-15 2016-01-13 Nec Corp Dispositif de gestion de données distribuées et dispositif d'exploitation de données distribuées
CN103220336B (zh) * 2013-03-21 2016-01-27 中国科学院计算技术研究所 一种文件同步中向量时钟的实现方法及系统
CN103561095A (zh) * 2013-11-04 2014-02-05 金蝶软件（中国）有限公司 一种数据同步方法、节点及存储服务集群

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102279857A (zh) * 2010-06-11 2011-12-14 阿里巴巴集团控股有限公司 一种实现数据复制的方法及系统
CN102622284A (zh) * 2012-02-21 2012-08-01 上海交通大学 面向海量存储系统的数据异步复制方法
CN104753987A (zh) * 2013-12-26 2015-07-01 北京东方通科技股份有限公司 一种分布式会话管理方法及系统
US20160140194A1 (en) * 2014-11-13 2016-05-19 Bigtera Limited Data migration system and data migration method thereof
CN106484321A (zh) * 2016-09-08 2017-03-08 华为数字技术（成都）有限公司 一种数据存储方法及数据中心

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111901420A (zh) * 2020-07-28 2020-11-06 深圳市康冠科技股份有限公司 一种数据同步方法、装置及系统
CN111901420B (zh) * 2020-07-28 2023-06-16 深圳市康冠科技股份有限公司 一种数据同步方法、装置及系统
CN114598559A (zh) * 2021-07-22 2022-06-07 湖南亚信软件有限公司 数据处理方法、装置、电子设备及计算机可读存储介质

Also Published As

Publication number Publication date
CN106484321A (zh) 2017-03-08

Similar Documents

Publication Publication Date Title
WO2018045758A1 (fr) Procédé de stockage de données et centre de données
AU2018348323B2 (en) Function-as-a-service (FaaS) platform in blockchain networks
CN106815218B (zh) 数据库访问方法、装置和数据库系统
KR102340296B1 (ko) 트랜잭셔널 환경에서 리소스 관리자(rm) 인스턴스 인지에 기초하여 공통 트랜잭션 식별자(xid) 최적화 및 트랜잭션 친화성을 지원하기 위한 시스템 및 방법
US11514077B2 (en) Replication event ordering using an external data store
TW201229795A (en) Web service patterns for globally distributed service fabric
CN111258976A (zh) 分布式锁实现方法、系统、设备及存储介质
WO2021107988A1 (fr) Traitement distribué de transactions dans un réseau à l'aide d'estampilles temporelles
WO2023045617A1 (fr) Procédé et appareil de traitement de données de transaction, dispositif et support
CN107580032B (zh) 数据处理方法、装置及设备
CN111475583B (zh) 事务处理方法及装置
AU2018348327B2 (en) Utilizing nonce table to resolve concurrent blockchain transaction failure
US20170193070A1 (en) System and method for a distributed replication lock for active-active geo-redundant systems
US20160063029A1 (en) Clustered storage system synchronization
CN110888858A (zh) 数据库的操作方法和装置、存储介质、电子装置
US20180123791A1 (en) Highly available and reliable secret distribution infrastructure
CN113010549A (zh) 基于异地多活系统的数据处理方法、相关设备及存储介质
CN107025257B (zh) 一种事务处理方法及装置
US10467143B1 (en) Event-driven cache
CN107967265B (zh) 文件的访问方法、数据服务器和文件访问系统
CN111857979A (zh) 一种分布式系统的信息管理方法、系统、存储介质及设备
CN115098528B (zh) 业务处理方法、装置、电子设备及计算机可读存储介质
US11290318B2 (en) Disaster recovery of cloud resources
CN113297168B (zh) 分布式系统中数据迁移方法及装置
US10061607B2 (en) System and method for providing single group multiple branches based on instance awareness

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17847932

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17847932

Country of ref document: EP

Kind code of ref document: A1