WO2018076954A1 - Procédé, appareil, et système de stockage de données - Google Patents

Procédé, appareil, et système de stockage de données Download PDF

Info

Publication number
WO2018076954A1
WO2018076954A1 PCT/CN2017/101847 CN2017101847W WO2018076954A1 WO 2018076954 A1 WO2018076954 A1 WO 2018076954A1 CN 2017101847 W CN2017101847 W CN 2017101847W WO 2018076954 A1 WO2018076954 A1 WO 2018076954A1
Authority
WO
WIPO (PCT)
Prior art keywords
storage
data block
block
data
stored
Prior art date
Application number
PCT/CN2017/101847
Other languages
English (en)
Chinese (zh)
Inventor
陈伟
林起芊
汪渭春
Original Assignee
杭州海康威视数字技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视数字技术股份有限公司 filed Critical 杭州海康威视数字技术股份有限公司
Publication of WO2018076954A1 publication Critical patent/WO2018076954A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD

Definitions

  • the present application relates to the field of computer technologies, and in particular, to a data storage method, apparatus, and system.
  • RAID 5 and RAID 6 in the RAID (Redundant Array of Independent Disks) mode.
  • RAID5 is a mode in which block cross-access and check information are evenly distributed.
  • the RAID 5 read and write operations are based on strips.
  • the strips are divided into multiple storage units, which are used to store data blocks and check blocks.
  • Figure 1a is a schematic structural view of a strip, wherein the strip includes n data blocks (D1, D2, ..., Dn) and m check blocks (P1, P2, ..., Pm), each data block And the check blocks are distributed on different disks.
  • disks can be on different servers.
  • n when n takes 3, m takes 1 , the strip corresponds to 4 servers, and FIG. 1b shows a schematic diagram of the logical relationship between the strip and the server in this example.
  • the horizontal 3 data blocks and 1 check block in Fig. 1b form a strip.
  • Table 1 is The composition of each specific strip.
  • the data blocks and check blocks in each stripe are located on different servers.
  • the data block is updated, the corresponding check block is also recalculated and updated.
  • a data block and a check block conform to the above relationship, there is consistency between the data block and the check block.
  • any data block or check block of not less than n is taken from any of them, and other data blocks or check blocks can be obtained according to the above relationship. That is, when a server fails, the data on that server can be recovered from the data on other servers.
  • the method of storing data is usually a full strip write to a disk in the server, that is, a check block is obtained according to the data block, and then the data block and the check block in the stripe are simultaneously written to the disk.
  • a check block is obtained according to the data block, and then the data block and the check block in the stripe are simultaneously written to the disk.
  • the stripe data may not be completely written successfully. After the system is restarted, the data in the stripe may be incomplete, and the valid data in the data block and the data in the check block may be inconsistent.
  • the purpose of the embodiments of the present application is to provide a data storage method, apparatus, and system to avoid the problem of inconsistency between data blocks and check blocks as much as possible.
  • the embodiment of the present application discloses a data storage method, which is applied to a primary storage server in a data storage system, where the data storage system includes: a management server and a storage server; and the storage server includes: a storage server and a non-primary storage server; the primary storage server is any one of the storage servers; the method includes:
  • the method further includes:
  • Deleting the stored data of the first data block when the first data block is not successfully stored, and performing the first data according to all data blocks that have been successfully stored in the first stripe The step of the first check block in the band.
  • the method further includes:
  • the method further includes:
  • the method further includes:
  • the method further includes:
  • the stored data of the second data block is deleted.
  • the method further includes:
  • the data in the first data block is from a file.
  • the embodiment of the present application discloses a data storage device, which is applied to a primary storage server in a data storage system, where the data storage system includes: a management server and a storage server; and the storage server includes: a storage server and a non-primary storage server; the primary storage server is any one of the storage servers; the device includes:
  • a first receiving module configured to receive a first data block to be stored to the first stripe
  • a first storage module configured to send, to the management server, a storage address obtaining request for the first data block, and obtain a first storage address determined by the management server according to a stripe storage mechanism, and use the first data
  • the block is stored to the first storage address
  • a first determining module configured to determine, when the first data block is successfully stored, whether the first data block is the last data block in the first stripe
  • a first calculating module configured to: when the determination result of the first determining module is YES, calculate a first check in the first strip according to all data blocks that have been successfully stored in the first stripe Piece;
  • a second storage module configured to send a storage address obtaining request for the first check block to the management server, and obtain a second storage address determined by the management server according to a stripe storage mechanism, where the first A parity block is stored to the second storage address.
  • the device further includes:
  • a first deleting module configured to delete the stored data of the first data block when the first data block is not stored successfully after storing the first data block to the first storage address, and Executing the first computing module.
  • the device further includes:
  • a second determining module configured to determine whether the first check block is successfully stored after storing the first check block to the second storage address, and if not, returning to execute the first calculating module .
  • the device further includes:
  • a detecting module configured to detect, when the first data block is not the last data block in the first strip, whether to receive the second data block to be stored in the first stripe in a preset time period ;
  • a second calculating module configured to: when the detection result of the detecting module is negative, calculate a second parity block in the first strip according to all data blocks that have been successfully stored in the first stripe;
  • a third storage module configured to send a storage address obtaining request for the second check block to the management server, and obtain a third storage address determined by the management server according to a stripe storage mechanism, and the second storage The check block is stored to the third storage address.
  • the device further includes:
  • a second receiving module when the storage unit for storing the data block in the first strip is not full, after receiving the second parity block to the third storage address, receiving the first Two data blocks;
  • a fourth storage module configured to send, to the management server, a storage address obtaining request for the second data block, and obtain a fourth storage address determined by the management server according to a stripe storage mechanism, and the second data is The block is stored to the fourth storage address;
  • a third determining module configured to determine, when the second data block is successfully stored, whether the second data block is the last data block in the first stripe
  • a third calculating module configured to: when the determination result of the third determining module is YES, calculate a third check in the first strip according to all data blocks that have been successfully stored in the first stripe Piece;
  • an update module configured to update the second check block to the third check block.
  • the device further includes:
  • a second deleting module configured to: after storing the second data block to the fourth storage address, when the second data block is the first data block received after storing the second parity block And storing the stored data of the second data block when the second data block is not successfully stored.
  • the device further includes:
  • a third deleting module configured to: after storing the second data block to the fourth storage address, when the second data block is not the first data block received after storing the second parity block And if the second data block is not successfully stored, the stored data of the second data block is deleted, and the third computing module is executed.
  • the present application discloses a data storage system including: a management server and a storage server; the storage server includes: a primary storage server and a non-primary storage server, wherein the primary storage server is the storage server Any of them;
  • the primary storage server is configured to receive a first data block to be stored to the first strip, and send a storage address obtaining request for the first data block to the management server;
  • the management server is configured to receive a storage address obtaining request for the first data block sent by the primary storage server, determine a first storage address according to a stripe storage mechanism, and send the first storage address to the Said primary storage server;
  • the primary storage server is configured to receive the first storage address sent by the management server, store the first data block to the first storage address, and when determining that the first data block is successfully stored, Determining whether the first data block is the last data block in the first strip, and if so, calculating the first strip according to all data blocks that have been successfully stored in the first strip a first parity block; sending a storage address obtaining request for the first parity block to the management server;
  • the management server is configured to receive a storage address obtaining request for the first check block sent by the primary storage server, determine a second storage address according to a stripe storage mechanism, and send the second storage address to The primary storage server;
  • the primary storage server is configured to receive the second storage address sent by the management server, and store the first parity block to the second storage address.
  • the management server is further configured to: when it is detected that the first data block is not successfully stored, and when detecting that an abnormality occurs in the primary storage server, deleting the stored data of the first data block, Selecting a new primary storage server from the non-primary storage server, switching the primary storage server to the new primary storage server, and transmitting a calculation instruction for the first parity block to the primary storage server
  • the calculation instruction carries a storage address of all data blocks that have been successfully stored in the first stripe
  • the primary storage server is configured to receive the calculation instruction sent by the management server, and obtain a data block according to the storage address carried in the calculation instruction, and calculate the first strip according to the obtained data block.
  • the first check block in .
  • the embodiment of the present application discloses a primary storage server in a data storage system, where the data storage system includes: a management server and a storage server; the storage server includes: a primary storage server and a non-primary storage server; The primary storage server is any one of the storage servers; the primary storage server includes:
  • the circuit board is disposed inside the space enclosed by the housing, the processor and the memory are disposed on the circuit board; and the power circuit is configured to Each circuit or device of the primary storage server is powered; the memory is for storing executable program code; the processor runs the program corresponding to the executable program code by reading executable program code stored in the memory for performing the following steps:
  • the embodiment of the present application discloses a storage medium for storing executable code, which is used to implement the data storage method provided by the embodiment of the present application.
  • the data storage method includes:
  • the primary storage server as the execution entity requests the first storage address of the first data block from the management server when receiving the first data block to be stored in the first stripe. Saving the first data block to the first storage location after obtaining the first storage address site.
  • the first data block is the last data block in the first strip, and if so, the first data is calculated according to all the data blocks that have been successfully stored in the first strip.
  • the first check block in the strip is obtained.
  • the data block is stored first, and when all the data blocks in the stripe are successfully stored, the check block is calculated according to the data block stored in the stripe, and the check block is stored.
  • the check block is first calculated according to the data block, and in the case where both the data block and the check block in the strip are available, the data block and the check block are simultaneously stored. If a power failure occurs during the storage process, both the data block and the check block may be unsuccessful in storage. In this case, the data block and the check block are inconsistent, and a Write Hole problem occurs.
  • the check block is calculated and stored, and even if a power failure occurs during the storage process, the data block and the data block can be avoided as much as possible.
  • the check block is inconsistent.
  • Figure 1a is a schematic structural view of a strip
  • Figure 1b is a schematic diagram of the logical relationship between a stripe and a server
  • FIG. 2 is a schematic structural diagram of a data storage system according to an embodiment of the present disclosure
  • FIG. 3 is a schematic flowchart of a data storage method according to an embodiment of the present application.
  • FIG. 4 is another schematic flowchart of a data storage method according to an embodiment of the present application.
  • Figure 5a is a schematic structural diagram of a strip after data has been stored
  • FIG. 5b is another schematic structural diagram of a strip after data has been stored
  • FIG. 5c is a schematic structural diagram of another strip after the data has been stored
  • FIG. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present disclosure.
  • FIG. 7 is another schematic structural diagram of a data storage device according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a data storage system according to an embodiment of the present application.
  • the embodiment of the present application provides a data storage method, device, and system, which can avoid the problem of inconsistency between a data block and a check block as much as possible.
  • the application can be applied to a primary storage server in a data storage system.
  • the data storage system can include a management server and a storage server.
  • the storage server may in turn include a primary storage server and a non-primary storage server, and the primary storage server may be any one of the storage servers.
  • FIG. 2 is a schematic structural diagram of a data storage system according to an embodiment of the present application.
  • a data storage system can also be referred to as a distributed storage system.
  • the system can include a management server 21, a storage server 22, and a client 23.
  • the management server 21 (Master Data Services, MDS) is also referred to as a metadata server for organizing and managing storage space information in the storage server 22.
  • the client 23 is configured to send data to be stored to the storage server.
  • FIG. 3 is a schematic flowchart of a data storage method according to an embodiment of the present disclosure, which is applied to a primary storage server in a data storage system.
  • the data storage system includes: a management server and a storage server.
  • Storage servers include: primary storage servers and non-primary storage servers.
  • the primary storage server is any one of the storage servers. Specifically, the method includes the following steps:
  • Step S301 Receive a first data block to be stored to the first strip.
  • the number of the first data blocks is one.
  • the first data block received by the primary storage server may be the first data block of the first stripe or the last data block of the first stripe.
  • the primary storage server As the execution body does not know which strip the first stripe is.
  • the primary storage server may send a stripe obtaining request to the management server, and after receiving the stripe obtaining request sent by the primary storage server, the management server determines the first strip according to the strip allocation mechanism, and sends the first strip to the Primary storage server
  • the storage server receives and records the first strip.
  • the management server determines the first strip, which may include determining the identity of the first strip.
  • the foregoing strip allocation mechanism can be understood as a stripe allocation rule, which can adopt a stripe allocation rule in the related art, which is not described in detail in this embodiment.
  • the stripe obtains a format of the stripe to be acquired in the request, the format may include a first quantity of the storage unit for storing the data block in the stripe, and a second quantity of the storage unit for storing the check block. Quantity.
  • the stripe to be acquired by the main storage server as the execution subject includes four storage units that store data blocks and one storage unit that stores check blocks.
  • the main storage server as the execution body has determined which stripe the first stripe is. Specifically, when the first data block is not the first data block of the first strip, the primary storage server may determine, according to the saved label of the first strip, the first strip corresponding to the first data block. The mark of the first strip may be that when the first data block is the first data block of the first strip, the primary storage server saves after receiving the first strip determined by the management server.
  • the primary storage server may divide the file data according to the size of each storage unit for storing the data block in the strip stored by itself. Get multiple blocks of data one by one.
  • the size of the file A to be stored is 1.6 MB
  • the size of each storage unit for storing data blocks in the strip is 1 MB
  • the primary storage server can divide the file A into 2 data blocks, one is 1 MB.
  • the first data block received in step S301 may be the first data block or the second data block.
  • Step S302 Send a storage address obtaining request for the first data block to the management server, and obtain a first storage address determined by the management server according to the stripe storage mechanism, and store the first data block to the first storage address.
  • the stripe storage mechanism can also be understood as a stripe storage rule, which includes a storage address allocation rule of each data block in the stripe and each check block.
  • the stripe storage mechanism in this embodiment may adopt a storage mechanism in the related art, which is not detailed in this embodiment.
  • the primary storage server as the execution entity may simultaneously send a stripe obtaining request for the first data block to the management server and for the first The storage address of the data block obtains the request, and may also send a stripe obtaining request for the first data block and a storage address obtaining request for the first data block to the management server.
  • the first storage address determined by the management server for storing the first data block may include: a first storage server for storing the first data block and a disk therein.
  • the first storage server can be any of all storage servers in the data storage system, including the primary storage server itself. That is, the first storage address may be located on the primary storage server itself or on other storage servers in the data storage system.
  • storing the first data block to the first storage address may include: the primary storage server directly storing the first data block to the first storage address.
  • storing the first data block to the first storage address may include: the primary storage server sending the first data block to the other storage server, to Causes the other storage server to store the first data block to the first storage address.
  • Step S303 When it is determined that the first data block is successfully stored, it is determined whether the first data block is the last data block in the first strip, and if yes, step S304 is performed.
  • the step may include: determining whether the first data block has been successfully stored, and if yes, determining whether the first data block is the last data block in the first strip.
  • the primary storage server may divide the file to be stored into data blocks according to the manner in which one data block corresponds to the same file. That is, the data in the first data block comes from a file.
  • the file is stored as a block of data, and the data of other files is no longer filled in the block. In this way, the following situation can be avoided: when the data of one data block comes from at least two files, if the data storage of one file is successful, the data storage of the other file fails, and finally it may be impossible to determine whether the data block is successfully stored.
  • Determining whether the first data block has been successfully stored may include the following implementation manners:
  • the primary storage server may directly determine whether the first data block has been successfully stored.
  • the primary storage server may detect whether the first data block storage success confirmation message sent by the other storage server is received within the preset time period, and if received, Then it is determined that the first data block has been successfully stored.
  • Determining whether the first data block is the last data block in the first strip may include: the number of data blocks stored in the first strip according to the first storage server and the first strip in the first strip A total number of storage units for storing data blocks, and determining whether the first data block is the last data block in the first strip.
  • the primary storage server may continue to receive the next first data to be stored to the first strip. Blocking, sending a storage address obtaining request for the next first data block to the management server, and obtaining a next first storage address determined by the management server according to the stripe storage mechanism, storing the next first data block to the next A first storage address. Until the storage unit for storing the data block in the first strip is full of data, then step S304 is performed.
  • each data block in the first strip is stored sequentially, and after the first data block is successfully stored, the second data block is continued to be stored until the last data in the first strip.
  • the block is stored successfully before the check block is calculated and stored.
  • Step S304 Calculate the first check block in the first strip according to all the data blocks that have been successfully stored in the first strip.
  • the first check block in the first strip is calculated according to all the data blocks that have been successfully stored in the first strip.
  • the primary storage server caches all the data blocks that have been successfully stored in the first strip
  • the primary storage server obtains all the data blocks that have been successfully stored in the first strip of the cache, and calculates the first strip according to the obtained data blocks.
  • the first check block in .
  • the primary storage server When the primary storage server does not cache all the data blocks that have been successfully stored in the first strip, the primary storage server sends the first strip data block obtaining request to each of the first storage servers, where the first storage server is the storage first. A server that strips data blocks. The primary storage server obtains the data block in the first strip sent by each first storage server, and calculates the first check block in the first strip according to the obtained data block.
  • the method for calculating the check block according to the data block may adopt a corresponding method in the related art, and the specific process is not detailed herein.
  • Step S305 Send a storage address obtaining request for the first check block to the management server, and obtain a second storage address determined by the management server according to the stripe storage mechanism, and store the first check block to the second storage address.
  • step S305 is substantially similar to the step S302. Therefore, the related content in step S302 can be referred to in step S305, and the specific process is not described in detail.
  • the primary storage server that is the execution entity when the primary storage server that is the execution entity receives the first data block to be stored in the first strip, the first storage address of the first data block is requested from the management server. The first data block is stored to the first storage address after the first storage address. When it is determined that the first data block is successfully stored, it is determined whether the first data block is the last data block in the first strip, and if so, the first data is calculated according to all the data blocks that have been successfully stored in the first strip. The first check block in the strip. Then, the second storage address of the first parity block is requested from the management server, and the first parity block is stored to the second storage address after the second storage address is obtained.
  • the data block is stored first, and when all the data blocks in the stripe are successfully stored, the check block is calculated according to the data block stored in the stripe, and the check block is stored.
  • the check block is first calculated according to the data block, and in the case where both the data block and the check block in the strip are available, the data block and the check block are simultaneously stored. If a power failure occurs during the storage process, both the data block and the check block may be unsuccessful in storage. In this case, the data block and the check block are inconsistent, and a Write Hole problem occurs.
  • the check block is calculated and stored, and even if a power failure occurs during the storage process, the data block and the data block can be avoided as much as possible.
  • the check block is inconsistent.
  • the method may further include:
  • step S304 is performed, that is, the step of calculating the first parity block in the first strip according to all the data blocks that have been successfully stored in the first stripe is performed.
  • the first data block is not stored successfully, and the data of the first data block may only be stored in one part, and the other part is not stored successfully; or all the data of the first data block may be stored, but part of the data is stored. An error occurred.
  • deleting the stored data of the first data block may include the following implementation manners:
  • the primary storage server may directly delete the stored data of the first data block.
  • the primary storage server may send the first storage server to the first storage server.
  • the delete instruction of the first data block is deleted, and the first storage server deletes the stored data of the first data block after receiving the delete instruction sent by the primary storage server.
  • the first storage server is a server that stores the first data block.
  • the step of deleting the stored data of the first data block can also be performed by the management server. That is, the management server may determine whether the confirmation information that the first data block is successfully stored by the first storage server is received within the preset time period, and if not received, determining that the first data block is not successfully stored, to the first storage server. Sending a delete instruction for deleting the first data block, the first storage server deleting the stored data of the first data block after receiving the delete instruction sent by the management server.
  • the first storage server is a server that stores the first data block. In this embodiment, the first storage server may be a primary storage server or a non-primary storage server.
  • the first data block is the first data block in the first strip
  • the stored data of the first data block is deleted
  • the first data block is not the first data block in the first strip, after the stored data of the first data block is deleted, the data block that has been successfully stored in the first stripe may continue to perform the calculation.
  • a step of a check block is performed.
  • step S305 that is, after storing the first parity block to the second storage address
  • the method may further include:
  • step S304 It is judged whether the first check block is successfully stored, and if not, the process returns to step S304; if so, it may not be processed.
  • determining whether the first check block is successfully stored may include the following implementation manners:
  • the primary storage server may directly determine whether the first verification block has been successfully stored.
  • the primary storage server may detect whether the confirmation information that the first check block is successfully stored by the other storage server is received within the preset time period, if received Then, it is determined that the first check block has been successfully stored.
  • the primary storage server is again stored according to the first strip. All the data blocks that are successful, calculate the first check block in the first stripe; resend the storage address obtaining request for the first check block to the management server, and regain the second determined by the management server according to the stripe storage mechanism The address is stored, and the first check block is stored to the second storage address.
  • the first check block is not stored successfully, which may be because the storage server storing the first check block is abnormal, so the storage address of the first check block needs to be re-acquired.
  • the stored data of the first check block may also be deleted to save storage space.
  • the method may further include the steps as shown in FIG. 4:
  • Step S306 When the first data block is not the last data block in the first strip, detecting whether the second data block to be stored to the first stripe is received within the preset time period, if not, executing Step S307.
  • the check block cannot be calculated.
  • the check block can be calculated and stored when the first strip is not full. In this way, the management server can temporarily set the state of the first stripe to be unwritable when it detects that the check block storage is successful.
  • Step S307 Calculate a second check block in the first strip according to all data blocks that have been successfully stored in the first strip.
  • Step S308 Send a storage address obtaining request for the second check block to the management server, and obtain a third storage address determined by the management server according to the stripe storage mechanism, and store the second check block to the third storage address.
  • steps S307 and S308 in the embodiment shown in FIG. 4 are similar to steps S304 and S305 in the embodiment shown in FIG. 3, respectively. Therefore, reference may be made to the related description in the embodiment shown in FIG.
  • the method may further include:
  • Step 1 Receive the second data block.
  • Step 2 Send a storage address obtaining request for the second data block to the management server, and obtain a fourth storage address determined by the management server according to the stripe storage mechanism, and store the second data block to the fourth storage address.
  • Step 3 When it is determined that the second data block is successfully stored, it is determined whether the second data block is the last data block in the first strip, and if yes, step 4 is performed.
  • Step 4 Calculate the third check block in the first strip according to all the data blocks that have been successfully stored in the first strip.
  • Step 5 Update the second parity block to the third parity block.
  • the strip A shown in FIG. 5a four memory cells for storing data blocks and one memory cell for storing parity blocks are included. It is known that the data block D1, the data block D2, and the check block P are stored in the strip A.
  • the storage address obtaining request of the data block D3 is sent to the management server, and the storage address determined by the management server according to the stripe storage mechanism is obtained, and the data block D3 is stored to The storage address.
  • the data block D4 is further sent to the management server for the received data block D4 to be stored to the strip A.
  • the storage address obtains the request, and obtains the storage address determined by the management server according to the stripe storage mechanism, and stores the data block D4 to the storage address.
  • the parity block P' in the strip A is calculated based on the data blocks D1 to D4 that have been successfully stored in the strip A, and the parity block P is updated to P'.
  • the strip A becomes the form shown in Fig. 5b, which includes the data blocks D1 to D4 and the check block P'.
  • the primary storage server receives the data to be stored to the strip again.
  • these data blocks are stored.
  • the check block is recalculated and updated according to the data block in the strip.
  • the method may further include:
  • the stored data of the second data block is deleted.
  • the strips shown in Figures 5a and 5b are still taken as an example for illustration.
  • the data blocks D1 and D2 and the check block P have been stored in the strip A shown in FIG. 5a.
  • the data block D3 is the one to be stored to the strip A after the check block P is stored.
  • the first data block if the data block D3 is abnormal in the storage process, and the storage is not successful, the stored data of the D3 can be directly deleted, so that the consistency of the data block and the check block in the strip A can be ensured. That is, after the data block D3 is directly deleted, the strip A is restored to the form of FIG. 5a, and is retracted to the version when the data block D3 is not written.
  • the method may further include:
  • the stored data of the second data block is deleted, and the execution is performed according to the first All the data blocks that have been successfully stored in the stripe are calculated, and the step of calculating the third check block in the first stripe is returned to step 4.
  • the strips shown in Figures 5a and 5b are still taken as an example for illustration.
  • the data blocks D1 and D2 and the check block P are already stored in the strip A shown in FIG. 5a.
  • the data block D3 When the data block D3 is received, it can be determined that the data block D3 is received after the check block P is stored.
  • the first data block of strip A When the data block D3 is successfully stored, the data block D4 is continuously stored, and the data block D4 is not the first data block to be stored to the strip A received after the check block P is stored.
  • the primary storage server deletes the data block D4 and calculates the check block P ⁇ according to the successfully stored D1, D2, and D3 in the strip A, and updates the check block P to Check block P ⁇ .
  • the strip A becomes the form shown in Fig. 5c, which includes the data blocks D1, D2, D3 and the check block P ⁇ .
  • the successfully stored data block D3 has changed the consistency of the data block and the check block in the strip A. Therefore, the check block should be recalculated and updated to ensure the data block and the check block. consistency.
  • the method may further include:
  • Step 4 Determining whether the third check block is successfully updated, if not, returning to perform the step of calculating the third check block in the first strip according to all the data blocks that have been successfully stored in the first strip, that is, returning to step 4 .
  • the strips shown in Figures 5a and 5b are still taken as an example for illustration.
  • the data blocks D1 and D2 and the check block P have been stored in the strip A shown in FIG. 5a, the data blocks D3 and D4 are added to the strip A shown in FIG. 5b, and the check block P is updated to the school.
  • Check block P' When an abnormality occurs during the update process and P' is not updated successfully, the check block P0 can be calculated again according to the data blocks D1 to D4, and the check block P' is updated to P0 to ensure that the check block update is successful.
  • FIG. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present disclosure, corresponding to the method embodiment shown in FIG. 3, applied to a primary storage server in a data storage system, where the data storage system includes: a management server and a storage server; the storage server includes: a primary storage server and a non-primary storage server; the primary storage server is any one of the storage servers.
  • the device includes:
  • the first receiving module 601 is configured to receive a first data block to be stored to the first stripe
  • a first storage module 602 configured to send, to the management server, a storage address obtaining request for the first data block, and obtain a first storage address determined by the management server according to a stripe storage mechanism, where the first The data block is stored to the first storage address;
  • the first determining module 603 is configured to determine, when the first data block is successfully stored, whether the first data block is the last data block in the first stripe;
  • the first calculating module 604 is configured to: when the determination result of the first determining module 603 is YES, calculate the first one of the first strips according to all the data blocks that have been successfully stored in the first stripe Check block
  • a second storage module 605 configured to send, to the management server, a storage address obtaining request for the first check block, and obtain a second storage address determined by the management server according to a stripe storage mechanism, where the A parity block is stored to the second storage address.
  • the device may further include:
  • a first deleting module (not shown), configured to delete the first data after storing the first data block to the first storage address, when the first data block is not successfully stored
  • the block has stored data and executes the first calculation module 604.
  • the device may further include:
  • a second determining module (not shown) for storing the first check block to the second After storing the address, it is determined whether the first check block is successfully stored, and if not, returning to the first calculating module 604.
  • the device may further include the following modules, which may be as shown in FIG. 7, FIG.
  • the illustrated embodiment corresponds to the method embodiment shown in FIG.
  • the detecting module 606 is configured to detect, when the first data block is not the last data block in the first strip, whether to receive the second data to be stored in the first stripe in the preset time period Piece;
  • the second calculating module 607 is configured to: when the detection result of the detecting module is negative, calculate a second check block in the first strip according to all data blocks that have been successfully stored in the first stripe ;
  • a third storage module 608 configured to send, to the management server, a storage address obtaining request for the second check block, and obtain a third storage address determined by the management server according to a stripe storage mechanism, where the first The second parity block is stored to the third storage address.
  • the device may further include:
  • a second receiving module (not shown) for storing the second parity block to the third storage when the storage unit for storing the data block in the first strip is not full After the address, receiving the second data block;
  • a fourth storage module (not shown), configured to send a storage address obtaining request for the second data block to the management server, and obtain a fourth storage address determined by the management server according to a stripe storage mechanism And storing the second data block to the fourth storage address;
  • a third determining module (not shown), configured to determine, when the second data block is successfully stored, whether the second data block is the last data block in the first stripe;
  • a third calculating module (not shown), configured to calculate the first piece according to all data blocks that have been successfully stored in the first strip when the determining result of the third judging module is YES a third check block in the band;
  • An update module (not shown) for updating the second check block to the third check block.
  • the device may further include:
  • a second deleting module (not shown), configured to: after storing the second data block to the fourth storage address, receive the second data block after storing the second parity block When the first data block is reached, and the second data block is not successfully stored, the stored data of the second data block is deleted.
  • the device may further include:
  • a third deleting module (not shown), configured to receive after the storing the second data block to the fourth storage address, when the third data block is not storing the second parity block When the first data block is reached, and the second data block is not successfully stored, the stored data of the second data block is deleted, and the third calculation module is executed.
  • FIG. 8 is a schematic structural diagram of a data storage system according to an embodiment of the present disclosure.
  • the system includes: a management server 81 and a storage server 82.
  • the storage server 82 includes a primary storage server 821 and a non-primary storage server 822.
  • the primary storage server is any one of the storage servers;
  • the primary storage server 821 is configured to receive a first data block to be stored to the first strip, and send a storage address obtaining request for the first data block to the management server 81;
  • the management server 81 is configured to receive a storage address obtaining request for the first data block sent by the primary storage server 821, determine a first storage address according to a stripe storage mechanism, and send the first storage address To the primary storage server 821;
  • the primary storage server 821 is configured to receive the first storage address sent by the management server 81, store the first data block to the first storage address, and determine that the first data block is successfully stored. Determining whether the first data block is the last data block in the first strip, and if so, calculating the first strip according to all data blocks in the first strip that have been successfully stored. a first parity block in the band; sending a storage address obtaining request for the first parity block to the management server 81;
  • the management server 81 is configured to receive a storage address obtaining request for the first check block sent by the primary storage server 821, determine a second storage address according to a strip storage mechanism, and set the second storage address Send to the primary storage server 821;
  • the primary storage server 821 is configured to receive the second storage address sent by the management server 81, and store the first parity block to the second storage address.
  • the management server 81 is further configured to: when detecting that the first data block is not successfully stored, and when detecting that the primary storage server 821 is generated When abnormal, deleting the stored data of the first data block, selecting a new primary storage server from the non-primary storage server 822, switching the primary storage server to the new primary storage server, and The primary storage server sends a calculation instruction for the first parity block, wherein the calculation instruction carries a storage address of all data blocks that have been successfully stored in the first stripe;
  • the primary storage server is configured to receive the calculation instruction sent by the management server 81, And obtaining a data block according to the storage address carried in the calculation instruction, and calculating a first parity block in the first strip according to the obtained data block.
  • the failure to store the first data block may be caused by various reasons, such as an abnormality occurs in the storage server storing the first data block, or an abnormality may occur in the primary storage server.
  • the management server detects that an abnormality has occurred in the primary storage server, it can reselect the new storage server as the primary storage server from the storage server.
  • the embodiment of the present application discloses a primary storage server in a data storage system, where the data storage system includes: a management server and a storage server; the storage server includes: a primary storage server and a non-primary storage server; The primary storage server is any one of the storage servers; the primary storage server includes:
  • the circuit board is disposed inside a space enclosed by the housing, the processor and the memory are disposed on the circuit board; and the power supply circuit is configured to be the main storage server
  • the memory is for storing executable program code
  • the processor runs the program corresponding to the executable program code by reading executable program code stored in the memory for performing the following steps:
  • the memory mentioned by the primary storage server may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
  • RAM random access memory
  • NVM non-volatile memory
  • the memory may also be at least one storage device located away from the aforementioned processor.
  • the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., or a digital signal processing (DSP), an application specific integrated circuit. (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component.
  • CPU central processing unit
  • NP network processor
  • DSP digital signal processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • the embodiment of the present invention and the method embodiment shown in FIG. 3 are based on the same inventive concept.
  • the specific embodiment of the present embodiment can adopt the content of the method embodiment shown in FIG.
  • the data block can be stored first.
  • the check block is calculated according to the data block successfully stored in the stripe, and then the check block is stored.
  • the check block is first calculated according to the data block, and in the case where both the data block and the check block in the strip are available, the data block and the check block are simultaneously stored. If a power failure occurs during the storage process, both the data block and the check block may be unsuccessful in storage. In this case, the data block and the check block are inconsistent, and a Write Hole problem occurs.
  • the check block is calculated and stored, and even if a power failure occurs during the storage process, the data block and the data block can be avoided as much as possible.
  • the check block is inconsistent.
  • the embodiment of the present application discloses a storage medium for storing executable code, which is used to implement the data storage method provided by the embodiment of the present application.
  • the data storage method includes:
  • the data block can be stored first.
  • the check block is calculated according to the data block successfully stored in the stripe, and then the check block is stored.
  • the check block is first calculated according to the data block, and in the case where both the data block and the check block in the strip are available, the data block and the check block are simultaneously stored. If a power failure occurs during the storage process, both the data block and the check block may be unsuccessful in storage. In this case, the data block and the check block are inconsistent, and a Write Hole problem occurs.
  • the check block is calculated and stored, and even if a power failure occurs during the storage process, the data block and the data block can be avoided as much as possible.
  • the check block is inconsistent.
  • the storage medium referred to herein means a ROM/RAM, a magnetic disk, an optical disk, or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Retry When Errors Occur (AREA)

Abstract

L'invention concerne un procédé, un appareil et un système de stockage de données, se rapportant au domaine de la technologie informatique. Le procédé de stockage de données est appliqué à un serveur de stockage principal dans un système de stockage de données, et le système de stockage de données comprend un serveur de gestion et un serveur de stockage. Le procédé consiste : à recevoir un premier bloc de données devant être stockées sur une première piste (S301) ; à envoyer une demande d'acquisition d'adresse de stockage pour le premier bloc de données à un serveur de gestion, acquérir une première adresse de stockage déterminée par le serveur de gestion selon un mécanisme de stockage de pistes, et stocker le premier bloc de données sur la première adresse de stockage (S302) ; à déterminer si le premier bloc de données est le dernier bloc de données sur la première piste (S303) ; si tel est le cas, à calculer un premier bloc de parité sur la première piste selon tous les blocs de données qui ont été stockés avec succès sur la première piste (S304) ; et à envoyer une demande d'acquisition d'adresse de stockage pour le premier bloc de parité au serveur de gestion, acquérir une seconde adresse de stockage déterminée par le serveur de gestion selon le mécanisme de stockage de pistes, et à stocker le premier bloc de parité dans la seconde adresse de stockage (S305). L'application dudit procédé permet d'éviter dans la plus large mesure des blocs de données et des blocs de parité incohérents.
PCT/CN2017/101847 2016-10-31 2017-09-15 Procédé, appareil, et système de stockage de données WO2018076954A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610930966.4 2016-10-31
CN201610930966.4A CN108008909B (zh) 2016-10-31 2016-10-31 一种数据存储方法、装置及系统

Publications (1)

Publication Number Publication Date
WO2018076954A1 true WO2018076954A1 (fr) 2018-05-03

Family

ID=62024307

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/101847 WO2018076954A1 (fr) 2016-10-31 2017-09-15 Procédé, appareil, et système de stockage de données

Country Status (2)

Country Link
CN (1) CN108008909B (fr)
WO (1) WO2018076954A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110471615B (zh) * 2018-05-10 2022-08-02 杭州宏杉科技股份有限公司 一种数据写入方法及装置
CN113835637A (zh) * 2020-03-19 2021-12-24 北京奥星贝斯科技有限公司 一种数据的写入方法、装置以及设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110154105A1 (en) * 2009-12-22 2011-06-23 David Woodhouse Redundant File System
CN102750195A (zh) * 2012-06-07 2012-10-24 浪潮电子信息产业股份有限公司 一种集群文件系统数据容错的方法
CN105404469A (zh) * 2015-10-22 2016-03-16 浙江宇视科技有限公司 一种视频数据的存储方法和系统
CN106027638A (zh) * 2016-05-18 2016-10-12 华中科技大学 一种基于混合编码的hadoop数据分发方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8527698B2 (en) * 2010-06-22 2013-09-03 Lsi Corporation Parity-based raid system configured to protect against data corruption caused by the occurrence of write holes
CN103729150A (zh) * 2014-01-14 2014-04-16 浪潮电子信息产业股份有限公司 一种在存储阵列中处理Write Hole的方法
CN105930097B (zh) * 2015-05-20 2019-01-29 德州学院 一种消除局部并行中小写操作的分布校验式磁盘阵列

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110154105A1 (en) * 2009-12-22 2011-06-23 David Woodhouse Redundant File System
CN102750195A (zh) * 2012-06-07 2012-10-24 浪潮电子信息产业股份有限公司 一种集群文件系统数据容错的方法
CN105404469A (zh) * 2015-10-22 2016-03-16 浙江宇视科技有限公司 一种视频数据的存储方法和系统
CN106027638A (zh) * 2016-05-18 2016-10-12 华中科技大学 一种基于混合编码的hadoop数据分发方法

Also Published As

Publication number Publication date
CN108008909A (zh) 2018-05-08
CN108008909B (zh) 2019-11-22

Similar Documents

Publication Publication Date Title
AU2017228544B2 (en) Nonvolatile media dirty region tracking
US9268648B1 (en) System and method for consistency verification of replicated data in a recovery system
US10705918B1 (en) Online metadata backup consistency check
US9372743B1 (en) System and method for storage management
US9471448B2 (en) Performing an atomic write operation across multiple storage devices
KR101870521B1 (ko) 스토리지 저널링을 개선하는 방법 및 시스템
US10261705B2 (en) Efficient data consistency verification for flash storage
US9836231B2 (en) Managing updates and copying data in a point-in-time copy relationship expressed as source logical addresses and target logical addresses
US10474527B1 (en) Host-assisted error recovery
US9740634B2 (en) Establishing a point-in-time copy relationship between source logical addresses and target logical addresses
US20150378642A1 (en) File system back-up for multiple storage medium device
US9329799B2 (en) Background checking for lost writes and data corruption
EP2567320B1 (fr) Procédés et système permettant de vérifier l'intégrité d'un dispositif de mémoire
WO2018086409A1 (fr) Procédé et dispositif de lecture/d'écriture de données de disque
US11625303B2 (en) Automatic incremental repair of granular filesystem objects
WO2018076954A1 (fr) Procédé, appareil, et système de stockage de données
WO2017087015A1 (fr) Compte d'opérations de métadonnées
US20160170842A1 (en) Writing to files and file meta-data
US7577804B2 (en) Detecting data integrity
WO2023147738A1 (fr) Procédé et appareil de stockage de données de machine virtuelle, machine virtuelle et support de stockage
US20180246788A1 (en) Method and apparatus for recovering in-memory data processing system
US20220374310A1 (en) Write request completion notification in response to partial hardening of write data
US10956052B1 (en) Online address to hash (A2H) metadata scanner
US9418100B1 (en) System and method for storage management
CN111048141B (zh) 一种错误处理方法、装置、设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17864327

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17864327

Country of ref document: EP

Kind code of ref document: A1