WO2017041654A1 - 用于分布式存储系统的写入数据、获取数据的方法和设备 - Google Patents

用于分布式存储系统的写入数据、获取数据的方法和设备 Download PDF

Info

Publication number
WO2017041654A1
WO2017041654A1 PCT/CN2016/097498 CN2016097498W WO2017041654A1 WO 2017041654 A1 WO2017041654 A1 WO 2017041654A1 CN 2016097498 W CN2016097498 W CN 2016097498W WO 2017041654 A1 WO2017041654 A1 WO 2017041654A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
stop information
written
storage block
file
Prior art date
Application number
PCT/CN2016/097498
Other languages
English (en)
French (fr)
Inventor
杨铭
李文兆
邹忠郁
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2017041654A1 publication Critical patent/WO2017041654A1/zh
Priority to US15/916,192 priority Critical patent/US10599337B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present application relates to the field of computers, and in particular, to a method and a device for writing data and acquiring data for a distributed storage system.
  • a cloud storage system due to the limitation of the user's network environment or the characteristics of the user's scene, the user may need to complete the writing of a data file in multiple times. For example, when a user writes a big data file, an additional write mode is often required. In a distributed cloud storage system, support for additional writes needs to solve problems such as concurrent writes and how to ensure data integrity. Therefore, how to implement data file write-write mode is a key technical solution for cloud storage systems.
  • the technical problem to be solved by the present application is to provide a method and device capable of implementing additional write data and acquiring data in a distributed cloud storage system, including user-specified additional location and additional pending write data for which the user does not specify an additional location. And methods and devices that support reading data in a distributed cloud storage system.
  • the present application provides a method for writing data for a distributed storage system, where the method includes:
  • the acquiring the data to be written by the user and storing the data to be written to the newly allocated data storage block includes:
  • Determining the size of the previous data stored in the data storage block that was last allocated before acquiring the data to be written is If it is less than the preset capacity, when it is less than, the penalty reading step is performed, including combining the previous data with the data to be written, and storing the data to the newly allocated data storage block.
  • the method for the user to specify the write data of the additional location further includes:
  • the request write location of the to-be-written data is obtained, and it is determined whether the requested write location is consistent with the total length of the current data file, and when they are consistent, the to-be-written data is acquired.
  • the method for the user to specify the write data of the additional location further includes:
  • Updating the description data file based on the primary level identification code and the total length of the data file after the data to be written is increased, and the description file updated by using an atomic operation persistence includes:
  • Determining whether the requested write position is consistent with the total length of the current data file if the file length of the data file acquired before the data to be written is acquired is the requested write position of the data to be written, when the time is consistent Updating the description data file based on the primary level identification code and increasing the total length of the data file after the data to be written, and persisting the updated description file by using the atomic operation.
  • the obtaining the data to be written by the user and storing the data to be written to the newly allocated data storage block further includes:
  • the penalty reading step it is determined whether the size of the previous data stored in the data storage block last allocated before the data to be written is less than the preset capacity, and when it is less, the penalty reading step is performed again. And merging the previous data with the data to be written, and re-storing to the data storage block, and updating, in the main-level start-stop information list, start and stop information based on the data storage block, when greater than or When equal to, the data to be written is written into the data to be written and stored in the data storage block, and the start and stop information is generated based on the data storage block in the main level start and stop information list.
  • the total length of the data file is re-acquired, and the file length of the re-acquired data file is the requested write position of the data to be written, and the request write is determined again. Whether the inbound position is consistent with the total length of the current data file, and when they are consistent, updating the description data file based on the main level identification code and the total length of the data file after the data to be written is increased, and the atomic operation is used to persist the Updated description file.
  • the method further includes:
  • All the data start and stop information in the current primary-level start and stop information list is transferred to the slave-level start-stop information list and stored to the newly-assigned slave-level start-and-stop information storage block,
  • the slave start and stop information generated based on the slave start and stop information storage block is stored to the master level start and end information storage block.
  • the method further includes:
  • each of the data storage blocks includes one or more unit storage blocks, wherein the plurality of unit storage blocks have the same capacity.
  • the data start and stop information includes at least one of the following:
  • the number of cells stored in the data storage block is the number of cells stored in the data storage block.
  • a method for acquiring data for a distributed storage system includes:
  • the corresponding data is read from the data storage block corresponding to the data start and stop information.
  • the searching for the corresponding data start-stop information from the primary-level start-stop information list further includes:
  • the corresponding data start and stop information is searched from the slave start and stop information storage block corresponding to the slave start and stop information.
  • the searching for the corresponding data start and stop information from the main level start and stop information list includes:
  • an apparatus for writing data for a distributed storage system where the device includes:
  • Writing means for acquiring data to be written by the user and storing the data to be written to the newly allocated data storage Piece;
  • a start and stop information updating means for generating data start and stop information about the data storage block, and updating the main level start and stop information list based on the data start and stop information, and storing the updated main level start and stop information list to the newly allocated main Level start and stop information storage block;
  • a primary level identification code updating means configured to generate the primary level identification code based on the primary level start and end information storage block
  • a description file updating means for updating the description file based on the main level identification code and increasing the total length of the data file after the data to be written, and persisting the updated description file with an atomic operation.
  • the device further includes:
  • the determining device acquires the request writing position of the data to be written before obtaining the data to be written, and determines whether the request writing position is consistent with the total length of the current data file, and when they are consistent, acquiring the to-be-written data.
  • the writing device includes:
  • a penalty reading unit configured to determine whether a size of a previous data stored in a data storage block that was last allocated before acquiring the data to be written is less than a preset capacity, and when less than, a penalty reading step is performed, including The previous data is merged with the data to be written and stored to the newly allocated data storage block.
  • the writing device further includes:
  • the description file updating apparatus further includes:
  • Determining whether the requested write position is consistent with the total length of the current data file if the file length of the data file acquired before the data to be written is acquired is the requested write position of the data to be written, when the time is consistent Updating the description data file based on the primary level identification code and increasing the total length of the data file after the data to be written, and persisting the updated description file by using the atomic operation.
  • description file updating apparatus is further configured to: for the manner in which the user does not specify the write data of the additional location:
  • the penalty reading step it is determined whether the size of the previous data stored in the data storage block last allocated before the data to be written is less than the preset capacity, and when it is less, the penalty reading step is performed again. And merging the previous data with the data to be written, and re-storing to the data storage block, and updating, in the main-level start-stop information list, start and stop information based on the data storage block, when greater than or When equal to, the data to be written is written into the data to be written and stored in the data storage block, and updated in the main level start and stop information list. Generating start and end information based on the data storage block;
  • the total length of the data file is re-acquired, and the file length of the re-acquired data file is the requested write position of the data to be written, and the request write is determined again. Whether the inbound position is consistent with the total length of the current data file, and when they are consistent, updating the description data file based on the main level identification code and the total length of the data file after the data to be written is increased, and the atomic operation is used to persist the Updated description file.
  • the method further includes:
  • a grading device configured to determine whether the capacity of the primary-level start and stop information list reaches a capacity threshold, or the number of the start-stop information reaches a quantity threshold, and when it is reached, all the data start and stop information in the current primary-level start-stop information list
  • the transfer to the slave start and stop information list is stored to the newly allocated slave start and stop information storage block, and the slave start and stop information generated based on the slave start and stop information storage block is stored to the master start and stop information storage block.
  • the method further includes:
  • a cache device configured to set a primary-level start and stop information storage block corresponding to the primary-level start and stop information list before the update to a readable and non-writable state, and clear the primary before the update after the description file is successfully persisted
  • the main level start and stop information storage block corresponding to the level start and stop information list.
  • each of the data storage blocks includes one or more unit storage blocks, wherein the plurality of unit storage blocks have the same capacity.
  • the data start and stop information includes at least one of the following:
  • the number of cells stored in the data storage block is the number of cells stored in the data storage block.
  • an apparatus for acquiring data for a distributed storage system where the method includes:
  • a main level start and stop information list obtaining means configured to acquire a main level start and stop information list from the main level start and end information storage block corresponding to the main level identification code based on the main level identification code in the description file;
  • a data start and stop information acquiring device configured to search for a corresponding data start and stop information from the main level start and stop information list
  • the data obtaining means is configured to read the corresponding data from the data storage block corresponding to the data start and stop information based on the data start and stop information.
  • the data start and stop information acquiring apparatus is further configured to:
  • the corresponding data start and stop information is searched from the slave start and stop information storage block corresponding to the slave start and stop information.
  • the data start and stop information acquiring means is configured to: find a corresponding data start and stop information by using a binary search method.
  • the method and device for writing data acquiring data for a distributed storage system, in the embodiment of the present application, wherein the method for writing data includes acquiring data to be written by a user And storing the data to be written to the newly allocated data storage block; generating data start and stop information about the data storage block, and updating the main level start and stop information list based on the data start and stop information, and updating the main
  • the start and stop information list is stored to the newly allocated main level start and stop information storage block;
  • the main level identification code is generated based on the main level start and end information storage block;
  • the data file is added based on the main level identification code and the to-be-written data
  • the total length updates the description file and utilizes atomic operations to persist the updated description file.
  • the method does not overwrite the data file, but always updates in the manner of copying to the newly allocated memory block and uses the atomic operation to persist the updated description file, so that in the process of additionally writing data It can ensure that the data files will not be damaged, and maintain the readability and consistency of the data, which is convenient for users.
  • the method for writing data in the embodiment of the present application supports the user to specify the writing of the writing position of the data to be written, and also supports the user not specifying the writing position of the writing position of the data to be written, thereby improving the user's writing data. Flexibility.
  • FIG. 1 shows a schematic diagram of a method for writing data for a distributed storage system in accordance with an aspect of the present application
  • FIG. 2 through 3 illustrate schematic diagrams of write data for a distributed storage system in accordance with a preferred embodiment of the present application
  • FIG. 4 through 5 illustrate schematic diagrams of write data for a distributed storage system in accordance with another preferred embodiment of the present application
  • 6 to 8 are schematic diagrams showing write data for a distributed storage system according to still another preferred embodiment of the present application.
  • FIG. 9 shows a schematic diagram of write data for a distributed storage system in accordance with yet another preferred embodiment of the present application.
  • FIG. 10 shows a schematic diagram of a method for acquiring data for a distributed storage system in accordance with an aspect of the present application
  • FIG. 11 shows a schematic diagram of an apparatus for writing data for a distributed storage system in accordance with an aspect of the present application
  • FIG. 12 shows a schematic diagram of an apparatus for acquiring data for a distributed storage system in accordance with an aspect of the present application.
  • PartIds are unique identifiers. For example, Part 1, Part 2, Part 3, etc., when the user uploads the data block, you must specify which PartId is currently operated, and the PartId can be unique.
  • the local index is stored to a Part, and the Part corresponding to each PartId is stored as a separate data file.
  • the write operation to the PartId overwrites the existing Part data file.
  • the user specifies which PartIds need to be assembled into a data file by submitting the Commit operation.
  • the Commit operation may discard some PartIds, and may also repeat certain PartId, and the corresponding Part is assembled or discarded according to the specified order.
  • the data file is invisible, the user cannot read the data file that wrote a part of the data; and after the Commit operation, the entire data file is visible, but the user can no longer write to the new Part. Or rewrite Part.
  • a user fails to upload a Part, the user needs to rewrite the entire Part and cannot write the Part.
  • the user needs to divide the entire data file in advance.
  • the cloud storage system supports a method of logically forming a large data file for multiple small data files.
  • the user uploads multiple small data files and updates a metadata file in time.
  • the metadata file represents the big data file that the user ultimately wants, and it contains a list of all small data files. Due to the loose relationship between small data files and combined data files, it is impossible to guarantee whether the small data files are deleted or changed, and the integrity of the combined data files is not guaranteed, so the integrity of the data cannot be guaranteed.
  • FIG. 1 illustrates a method for writing data for a distributed storage system in accordance with an aspect of the present application, wherein the method includes:
  • Step S01 acquiring data to be written of the user, and storing the data to be written to the newly allocated data storage block;
  • Step S02 generating data start and stop information about the data storage block, and updating the main level start and stop information list based on the data start and stop information, and storing the updated main level start and stop information list to the newly allocated main level start and stop information storage.
  • Step S03 generating the primary level identification code based on the primary level start and end information storage block
  • Step S04 Update the description file based on the main level identification code and the total length of the data file after the data to be written is increased, and use the atomic operation to persist the updated description file.
  • the data start and stop information is generated based on the data storage block, and is used for recording a start and end address of the data to be written in the data file, and according to the data start and stop information, the corresponding data storage block can be found and found to be corresponding to the data to be written. .
  • the data start and stop information includes: the data storage block identification code; start and end address information of the data in the data storage block; a size of the data to be written; and a number of units stored in the data storage block.
  • the data storage block identification code is used to uniquely identify a data storage block, and the start and end address information of the data in the data storage block includes a start position and an end position of the corresponding to-be-written data in the data file.
  • the description file may include: a total length of the data file, a modification time, and a main level identification code, and the description file is used to facilitate reading data or appending data to be written.
  • each of the data storage blocks includes one or more unit storage blocks, wherein when the data storage blocks have a plurality of unit storage blocks, the capacities of the plurality of unit storage blocks are the same.
  • the method of the present application allocates a data storage block for storing data to be written each time the data to be written is acquired, and allocates a primary-level start and stop information storage block for storing a primary-level start and stop information list having data start and stop information, according to the And updating the description file of the main level identification code of the main level start and stop information storage block and increasing the total length of the data file after the data to be written, and then using the atomic operation to persist the updated description file.
  • the data storage block and the copy-on-write information storage block (Copy-On-Write) are re-allocated, and the process of appending the data to be written does not overwrite the existing data file.
  • the description file is stored in a temporary description file (for example, in a volatile medium, such as a temporary description file in memory) before being persisted, and the description file persistence process is implemented by using atomic operations.
  • atomic operations Preferably, two atomic operations are included: Put-If-Not-Exist and Test-And-Set.
  • Put-If-Not-Exist (FileName, FileInfo) operation can only add a record of the file name to the file metadata if the file name does not exist; Test-And-Set (FileName, NewFileInfo, OldFileInfo) operation
  • the updated description file can only be persisted if the file name exists and the content of the current description file (FileInfo) corresponds to the pre-update description file (OldFileInfo).
  • the current data to be written is invisible to the user before the description file is persisted, that is, the user cannot read the current data to be written (but the existing data file in the cloud storage system is visible to the user), thereby maintaining Data consistency and stability.
  • the acquired data to be written by the user may be the data to be written first submitted by the user to the distributed system, or may be the additional data to be written submitted by the user for the existing data file in the distributed system.
  • the method described in the present application may store the data to be written submitted by the user for the first time and generate a data file, or add the additional data to be written submitted by the user to the existing data file.
  • the underlying storage can be divided into the data element storage area Metastore and the data storage area DataStore, and the MetaStore maintains the file name (Readme.txt).
  • the metadata includes information addressed to the data to be written
  • the DataStore is used to store the data to be written and the related data start and stop information of the data to be written
  • the DataStore is divided into a plurality of fixed size blocks. Each block has a corresponding address, and the contents of the block can be read according to the block address.
  • MetaStore and DataStore are mainly logical distinctions, which may be implemented on the same underlying storage.
  • the data to be written of the user is acquired, and the allocated data storage block D1 stores the data to be written.
  • the data start and stop information Extent is generated according to the data storage block D1, and the data start and stop information is generated.
  • Extent includes the start address Range Start, the end address Range End of the data to be written in the data file, the data storage block identifier OutetID of the data storage block D1, the number of unit storage blocks Nblks in the data storage block D1, and the main level information.
  • Level usually when the level is 0, indicates that the data storage block directly stores the data to be written, and then updates the main-level start and stop information list with the data start and stop information Extent and stores it to the main-level start-stop information storage block RootExtentBlock;
  • a main level identification code RootExtentID is generated based on the main level start and stop information storage block RootExtentBlock;
  • the description file is updated based on the main level identification code RootExtentID and the total length of the data file after the data to be written is increased.
  • Fileinfo and use the atomic operation Put-If-Not-Exist to persist the updated description file Fileinfo
  • the user specifies the write data mode of the additional location and the write data mode in which the user does not specify the additional location, which is described below.
  • the methods described herein are further described in two ways.
  • step S01 the data to be written is acquired.
  • Requesting a write position (Position) determining whether the requested write position of the to-be-written data is consistent with the total length of the current data file, and if they are consistent, acquiring the corresponding data to be written, and when they are inconsistent, rejecting the corresponding
  • the data to be written may simultaneously feed back the total length of the current data file to the user equipment.
  • the concurrent write data request can be prevented from being written to the same write location when the user concurrently writes data, thereby avoiding multiple duplicate write data being assembled into the data file. For example, suppose the write data includes A, B, and C.
  • the current user concurrently writes the data C.
  • the user concurrently uploads two Cs, the two uploaded concurrently. Only one of C will upload successfully, then the data written includes A, B, and C instead of A, B, and two Cs.
  • the manner in which the user specifies that the write data of the additional location is increased can prevent the user from concurrently writing and causing an error in the data file.
  • the data storage block Block D1#0 stores the 20K data to be written, and generates the data start and stop information Extent1 to store the start and stop information.
  • the file length Filelength of the description file FileInfo is 20K
  • the main level identifier RootExtentID is EB-A.
  • the write location of the request is also submitted.
  • the device Before the device obtains the data to be written in step S01, the device first determines whether the write position submitted by the user is equal to the file length of the description file, Filelength. If not equal to, the write request is rejected, and if it is equal to, the data to be written is obtained;
  • step S01 after the data to be written is acquired, it is determined whether the size of the previous data stored in the data storage block last allocated before the data to be written is smaller than a preset.
  • the capacity when less than, performs a penalty read step comprising combining the previous data with the data to be written and storing to the newly allocated data storage block.
  • the preset capacity of the unit storage block of the data storage block is 64K
  • the additional data to be written submitted by the user is 130K
  • the data to be written is added.
  • the size of the previous data stored in the last allocated data storage block is 20K, which is less than the preset capacity of 64K.
  • a penalty reading step is performed, including re-reading the previous data 20K, and the previous data 20K and the current data.
  • the 130K merge of the submitted additional data to be written is stored together in the newly allocated data storage block Block D2, including three unit storage blocks Block D2#0, Block D2#1 and Block D2#2.
  • step S02 According to the newly allocated data storage block Block D2, Generating the data start and stop information Extent2 with respect to the data storage block, and storing it in the main level start and stop information storage block RootExtentBlock EB-B, the data start and stop information Extent2 including the data start address 0 to be written and the end address 150K of the data to be written.
  • the data storage block identification code D2, the unit storage block Nblks number 3, and the level level is 0; in the step S03, the main level identification code EB-B of the data start and stop information RootExtentBlock EB-B is generated.
  • step S04 the description file is updated by using an atomic operation Test-And-Set, including the file length of the updated data file, Filelength is 150K, and the main level identifier RootExtentID is EB-B.
  • the penalty reading step is not performed, and the data to be written is directly stored to the newly allocated data. Memory block.
  • the data to be written 10K submitted by the user again determines the data storage block that was last allocated before the data to be written is acquired. If the size of the previous data stored is 150K greater than the preset capacity 64K, the previous data 150K is not re-read, and the current data to be written 10K is directly stored in the newly allocated data storage block Block D3, and then
  • the generated data start and stop information Extent3 is added to the main-level start and stop information list, and is stored in the newly allocated main-level start-stop information storage block RootExtentBlock EB-C, and the main-level identification code EB-C is generated.
  • the description file Fileinfo is updated based on the main level identification code EB-C and the total length FileLength of the data file is 160K, and the updated description file Fileinfo is persistently updated by the atomic operation Test-And-Set.
  • step S04 the file length of the data file acquired before the data to be written is acquired is the request write position of the data to be written, and the request write is determined.
  • step S01 to the step S03 and before the step S04 if the data file stored in the device does not change, the corresponding data file length and the description file also do not change, then in step S04, the acquisition is performed.
  • the description data file is updated based on the main level identification code and the total length of the data file after the data to be written is increased, and the updated description file is persisted by the atomic operation. If in the step S01 to the step S03 and before the step S04, the data file stored in the device changes (for example, during this period, a concurrent write data is successfully added, resulting in the data file being modified), then corresponding The data file length and the description file are also changed.
  • step S04 the file length of the data file acquired before the data to be written is acquired is the request writing position of the data to be written, and the request is determined. If the write position does not match the total length of the current data file, you need to re-add the data to be written.
  • FIGS. 6 to 8 are schematic diagrams showing write data for a distributed storage system according to still another preferred embodiment of the present application; specifically, for example, in conjunction with FIG. 6, it is assumed that the data file has data 20K to be written, and two data are recorded.
  • Concurrently write data processes Append-1 and Append-2 assuming that Append-1 is added successfully, as shown in Figure 6, Append-2 adds data to be written to fail (for example, Figure 7, Tempend-2 generated temporary description file) Fileinfo is not persisted successfully, that is, Append-2 appends the data to be written to actually fail.
  • Append-3 is executed to re-add the data to be written.
  • Append-3 only needs the total length of the data file, and writes the requested location of the data to be written with the file length of the re-acquired data file, and judges again. Whether the request writing position is consistent with the total length of the current data file, and when they are consistent, updating the description data file based on the main level identification code and increasing the total length of the data file after the data to be written, using the atomic operation Persist the description file for the update.
  • Append-2 has a penalty read, it is determined whether the size of the previous data stored in the data storage block that was last allocated before the data to be written is smaller than the preset capacity. When it is less than, the penalty read step is performed again. And merging the previous data with the data to be written, and re-storing to the data storage block, and updating, in the main-level start-stop information list, start and stop information based on the data storage block, when greater than or When equal to, the data to be written is written into the data to be written and stored in the data storage block, and the start and stop information is generated based on the data storage block in the main level start and stop information list. As shown in FIG. 7, Append-2 has a penalty read, and in combination with FIG. 6 and FIG.
  • each time the user submits the data to be written the user does not need to simultaneously submit the requested write location, and after each time the user's write data is obtained, the write data is appended to the existing data file.
  • the distributed system automatically acquires the total length of the current data file, and sequentially assembles all the received data blocks into the data file according to the order in which the request is received.
  • Write data includes A, B, and C.
  • the existing data files in the distributed system include A and B
  • the user concurrently uploads two Cs when one of the data to be written C is written After the success, the total length of the data file changes.
  • the first write of the second data to be written C may fail to be written. Then the second data to be written C will reacquire the total length of the data file and re- If the attempt is successful, the two Cs uploaded concurrently will be written into the data file in turn.
  • the final data file includes A, B, C, and C.
  • the scheme in which the user does not specify an additional location is suitable for a scenario that does not care about the order between the additional data, such as collection of program logs in a distributed environment.
  • the method further comprises: determining whether the capacity of the primary-level start and stop information list reaches a capacity threshold or the number of the start-stop information reaches a quantity threshold, and when it is reached, the current primary-level start-stop information list All the data start and stop information is transferred to the slave start and stop information list and stored to the newly allocated slave start and stop information storage block, and the slave start and stop information generated based on the slave start and stop information storage block is stored to the master start and stop information storage block. .
  • the method further includes checking a primary-level start and stop information storage block RootExtentBlock, and if the capacity of the primary-level start-stop information list reaches a capacity threshold or the number of start-stop information reaches a quantity threshold, multi-level processing is performed (Level Promotion And all the data start and end information in the current main-level start and stop information list is transferred to the slave-level start-stop information list and stored to the newly-assigned slave-level start-and-stop information storage block, and the slave generated based on the slave-level start-stop information storage block The start and end information is stored to the main start and stop information storage block.
  • FIG. 9 is a schematic diagram showing write data for a distributed storage system according to still another preferred embodiment of the present application; the purpose of Level Promotion is to limit the size of the primary-level start and end information storage block RootExtentBlock to effectively support large files.
  • the purpose of Level Promotion is to limit the size of the primary-level start and end information storage block RootExtentBlock to effectively support large files.
  • the initial main level start and end information storage block RootExtentBlock there is only data start and stop information, that is, multiple start and stop information Extent of Level 0.
  • a certain threshold for example, 1024
  • BlockSize a certain threshold
  • the slave start and stop information storage block ExtentBlock EB-1 stores the slave start and stop information list, including all the Level 0Extents described above, generates a Level 1Extent based on the slave start and stop information storage block ExtentBlock EB-1, and the Level 1Extent is stored in the newly allocated master
  • the data start address RangeStart is the data start address RangeStart of the first data start and stop information Extent included in the start and end information storage block ExtentBlock EB-1
  • the data end address RangeEnd Is included in the start and end information storage block ExtentBlock EB-1 The data end address RangEnd of the last start and end information Extent, and the slave identification code ExtentID corresponding to the slave start and end information storage block ExtentBlock EB-1 is EB-1.
  • the method further includes: setting a primary-level start and stop information storage block corresponding to the primary-level start and stop information list before the update to a readable and non-writable state, and clearing the update after the description file is successfully persisted.
  • the main level start and stop information storage block corresponding to the main level start and stop information list When the method performs the step S01 to the step S04 to write data, the main-level start and stop information storage block corresponding to the main-level start and stop information list before the update is set to a readable and writable state, and is in a persistent office. After the description file succeeds, the main level start and stop information storage block corresponding to the main start and stop information list before the update is cleared. For example, in FIG.
  • the main level start and end information storage block EB-B is set in the process of writing data in FIG. Readable and unwritable state, after the main level start and stop information storage block EB-C update shown in FIG. 4 is completed and the updated description file Fileinfo is completed, the main level start and end information storage block EB-B is cleared; likewise, for FIG. 3
  • the shown data storage block Block D1#0 is cleared after the data storage blocks Block D2#0, D2#1 and D2#2 shown in FIG. 4 are stored and the updated description file Fileinfo is completed, so that the user is added.
  • the data file stored by the device can still provide the user with the service for obtaining the data.
  • the method for writing data in the embodiment of the present application includes acquiring data to be written by a user, and storing the data to be written to a newly allocated data storage block; generating data about the data And storing data start and stop information of the block, and updating the main level start and stop information list based on the data start and stop information, and storing the updated main level start and stop information list to the newly allocated main level start and stop information storage block; starting and ending based on the main level
  • the information storage block generates the primary level identification code; updates the description file based on the primary level identification code and the total length of the data file after the data to be written is increased, and the updated description file is persisted by an atomic operation.
  • the method does not overwrite the data file, but always updates in the manner of copying to the newly allocated memory block and uses the atomic operation to persist the updated description file, so that in the process of additionally writing data It can ensure that the data files will not be damaged, and maintain the readability and consistency of the data, which is convenient for users.
  • the method for writing data in the embodiment of the present application supports the user to specify the writing of the writing position of the data to be written, and also supports the user not specifying the writing position of the writing position of the data to be written, thereby improving the user's writing data. Flexibility.
  • FIG. 10 is a schematic diagram of a method for acquiring data for a distributed storage system according to an aspect of the present disclosure; wherein the method includes:
  • Step S11 acquiring a description file
  • Step S12 Acquire a main level start and stop information list from the main level start and end information storage block corresponding to the main level identification code, based on the main level identification code in the description file;
  • Step S13 searching for the corresponding data start and stop information from the main level start and stop information list;
  • Step S14 Read corresponding data from the data storage block corresponding to the data start and stop information based on the data start and stop information.
  • the step S13 further includes: starting from the slave-level start and stop information, the slave-level start-stop information storage block corresponding to the slave-level start and stop information Find the corresponding data start and stop information.
  • the binary search method is used to find the corresponding data start and stop information.
  • the method for acquiring data in the embodiment of the present application can acquire data in time for the user to read on the basis of the written data, and can read at any time without being written.
  • the impact of the data Convenient for the user's use.
  • the method for writing data in the embodiment of the present application supports the user to specify the writing of the writing position of the data to be written, and also supports the user not specifying the writing position of the writing position of the data to be written, thereby improving the user's writing data. Flexibility.
  • FIG. 11 shows a schematic diagram of an apparatus for writing data for a distributed storage system in accordance with an aspect of the present application, wherein the apparatus 1 includes:
  • the writing device 01 is configured to acquire data to be written by the user, and store the data to be written to the newly allocated data storage block;
  • the start and stop information updating means 02 is configured to generate data start and stop information about the data storage block, and update the main level start and stop information list based on the data start and stop information, and store the updated main level start and stop information list to the newly allocated Main level start and stop information storage block;
  • the main level identification code updating means 03 is configured to generate the main level identification code based on the main level start and end information storage block;
  • the profile updating means 04 is configured to update the description file based on the primary level identification code and the total length of the data file after the data to be written is increased, and the updated description file is persistently performed using an atomic operation.
  • the device 1 includes, but is not limited to, a user equipment, or a device formed by integrating a user equipment and a network device through a network.
  • the user equipment includes, but is not limited to, any mobile electronic product that can interact with a user through a touchpad, such as a smart phone, a PDA, etc., and the mobile electronic product can adopt any operating system, such as an android operating system. iOS operating system, etc.
  • the network device includes a type that can be set in advance Or stored instructions, electronic devices that automatically perform numerical calculations and information processing, including but not limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors (DSPs), embedded Equipment, etc.
  • ASICs application specific integrated circuits
  • FPGAs programmable gate arrays
  • DSPs digital processors
  • embedded Equipment etc.
  • the network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), and the like.
  • the device 1 may also be a script program running on the user equipment, or a user equipment and a network device, a touch terminal, or a device formed by integrating a network device and a touch terminal through a network.
  • the above-mentioned device 1 is only an example, and other existing or future devices 1 may be applicable to the present application, and are also included in the protection scope of the present application, and are hereby incorporated by reference. Included here.
  • the data start and stop information is generated based on the data storage block, and is used for recording a start and end address of the data to be written in the data file, and according to the data start and stop information, the corresponding data storage block can be found and found to be corresponding to the data to be written. .
  • the data start and stop information includes: the data storage block identification code; start and end address information of the data in the data storage block; a size of the data to be written; and a number of units stored in the data storage block.
  • the data storage block identification code is used to uniquely identify a data storage block, and the start and end address information of the data in the data storage block includes a start position and an end position of the corresponding to-be-written data in the data file.
  • the description file may include: a total length of the data file, a modification time, and a main level identification code, and the description file is used to facilitate reading data or appending data to be written.
  • each of the data storage blocks includes one or more unit storage blocks, wherein when the data storage blocks have a plurality of unit storage blocks, the capacities of the plurality of unit storage blocks are the same.
  • the device 1 of the present application allocates a data storage block to store data to be written each time the data to be written is acquired, and allocates a primary-level start-stop information storage block for storing a primary-level start and stop information list having data start and stop information. Updating the description file according to the main level identification code of the main level start and stop information storage block and increasing the total length of the data file after the data to be written, and then using the atomic operation to persist the updated description file.
  • the data storage block and the copy-on-write information storage block (Copy-On-Write) are re-allocated, and the process of appending the data to be written does not overwrite the existing data file.
  • the description file is stored in a temporary description file (for example, in a volatile medium, such as a temporary description file in memory) before being persisted, and the description file persistence process is implemented by using atomic operations.
  • atomic operations Preferably, two atomic operations are included: Put-If-Not-Exist and Test-And-Set.
  • Put-If-Not-Exist (FileName, FileInfo) operation can only add a record of the file name to the file metadata if the file name does not exist; Test-And-Set (FileName, NewFileInfo, OldFileInfo) operation
  • the updated description file can only be persisted if the file name exists and the content of the current description file (FileInfo) corresponds to the pre-update description file (OldFileInfo).
  • the current data to be written is invisible to the user before the description file is persisted, that is, the user cannot read the current data to be written (but the existing data file in the cloud storage system is visible to the user), thereby maintaining Data consistency and stability.
  • the acquired data to be written by the user may be the data to be written first submitted by the user to the distributed system, or may be the additional data to be written submitted by the user for the existing data file in the distributed system.
  • the device 1 described in the present application may store the data to be written submitted by the user for the first time and generate a data file, or add the additional data to be written submitted by the user to the existing data file.
  • the underlying storage can be divided into the data element storage area Metastore and the data storage area DataStore, and the MetaStore maintains the file name (Readme.txt).
  • the metadata includes information addressed to the data to be written
  • the DataStore is used to store the data to be written and the related data start and stop information of the data to be written
  • the DataStore is divided into a plurality of fixed size blocks. Each block has a corresponding address, and the contents of the block can be read according to the block address.
  • MetaStore and DataStore are mainly logical distinctions, which may be implemented on the same underlying storage.
  • the data to be written of the user is acquired, and the distribution data storage block D1 stores the data to be written; in the start and stop information updating device 02, the data start and stop information is generated according to the data storage block D1.
  • the data start and stop information Extent includes the start address of the data to be written in the data file Range Start, the end address Range End, the data storage block identification code ExtentID of the data storage block D1, and the number of unit storage blocks in the data storage block D1 Nblks And the level of the main level information, usually when the level is 0, indicating that the data storage block directly stores the data to be written, and then updates the main level start and stop information list with the data start and end information Extent and stores it to the main level start and stop information storage block RootExtentBlock
  • a main level identification code RootExtentID is generated based on the main level start and end information storage block RootExtentBlock; in the description file updating means 04, based on the main level identification code Root
  • the user specifies the write data mode of the additional location and the write data mode in which the user does not specify the additional location, which is described below.
  • the device 1 described in the present application is further described in two ways.
  • the request write position (Position) of the data to be written is acquired, and the request write position of the data to be written and the current data file are determined. Whether the total length of the data file is consistent. When the data is consistent, the data to be written is obtained. If the data is not consistent, the data to be written is rejected. The total length of the current data file can be fed back to the user.
  • the concurrent write data request can be prevented from being written to the same write location when the user concurrently writes data, thereby avoiding multiple duplicate write data being assembled into the data file. For example, suppose the write data includes A, B, and C.
  • the current user concurrently writes the data C.
  • the user concurrently uploads two Cs, the two uploaded concurrently. Only one of C will upload successfully, then the data written includes A, B, and C instead of A, B, and two Cs.
  • the manner in which the user specifies that the write data of the additional location is increased can prevent the user from concurrently writing and causing an error in the data file.
  • the data storage block Block D1#0 stores the 20K data to be written, and generates the data start and stop information Extent1 to store the start and stop information.
  • the file length Filelength of the description file FileInfo is 20K
  • the main level identifier RootExtentID is EB-A.
  • the write position of the request is also submitted. Before the device obtains the data to be written, the device first determines whether the write position submitted by the user is equal to the file length of the description file. If not equal to, the write request is rejected, and if equal, the data to be written is obtained;
  • the writing device 01 further includes a penalty reading unit, configured to determine, before acquiring the data to be written, before storing the last allocated data storage block before acquiring the data to be written. Whether the size of the secondary data is smaller than the preset capacity, when less than, the penalty reading step is performed, including combining the previous data with the data to be written, and storing the data to the newly allocated data storage block.
  • the preset capacity of the unit storage block of the data storage block is 64K
  • the additional data to be written submitted by the user is 130K
  • the data to be written is added.
  • the size of the previous data stored in the last allocated data storage block is 20K, which is less than the preset capacity of 64K.
  • a penalty reading step is performed, including re-reading the previous data 20K, and the previous data 20K and the current data.
  • the 130K merges are stored together in the newly allocated data storage block Block D2, including three unit storage blocks Block D2#0, Block D2#1 and Block D2#2.
  • the data start and stop information Extent2 regarding the data storage block is generated and stored in the main-level start and stop information storage block RootExtentBlock EB-B, and the data starts and ends.
  • the information Extent2 includes a data start address 0 to be written and an end address 150K of the data to be written, the data storage block identification code D2, the unit storage block Nblks number 3, and the level level 0; the main level identification code is updated.
  • the main level identification code EB-B of the data start and stop information RootExtentBlock EB-B is generated, and in the description file updating device S04, the description file is updated by using an atomic operation Test-And-Set, including the updated The file length of the data file is 150K, and the main identifier RootExtentID is EB-B.
  • the penalty reading step is not performed, and the data to be written is directly stored to the newly allocated data. Memory block.
  • the data to be written 10K submitted by the user of the writing device 01 is again submitted, and it is determined that the data storage block last allocated before the data to be written is stored is stored.
  • the previous data 150K is not re-read, and the data to be written 10K is directly stored in the newly allocated data storage block Block D3, and then the information update device 02 ⁇
  • the main level identification code updating means S03 the generated data start and stop information Extent3 is added to the main level start and stop information list, and stored in the newly allocated main stage start and end information storage block RootExtentBlock EB-C, and the main level identification code EB-C is generated.
  • the profile updating means 04 the profile Fileinfo is updated based on the master-level identification code EB-C and the total length FileLength of the data file is 160K, and the profile file Fileinfo is updated by the atomic operation Test-And-Set.
  • the user when the user does not specify the request write location of the data to be written, each time the user submits the data to be written, the user does not need to simultaneously submit the requested write location, and before the device obtains the data to be written, First, the total length of the data file is acquired; then, the writing device 01 is executed to the main level identification code updating device 03, and the execution process is performed with the writing device 01 to the main level identification code updating device 03 shown in FIGS. 2 to 5.
  • the content is the same or substantially the same, and is included herein for the sake of brevity.
  • the description file update device 04 when the description file update device 04 is executed, the file length of the data file is obtained before the data to be written is acquired.
  • the total length of the file updates the description data file, using the atomic operations to persist the description Document.
  • the writing device 01 to the main level identification code updating device 03 and before the description file updating device 04 if the data file stored in the device has not changed, the corresponding data file length and description file also do not occur.
  • the description file updating device 04 the file length of the data file acquired before the data to be written is acquired is the request writing position of the data to be written, and then the request writing position and current are determined.
  • the total length of the data files is consistent, and the description data file is updated based on the main level identification code and the total length of the data file after the data to be written is increased, and the updated description file is persisted by the atomic operation. If the data file stored in the device changes before the writing device 01 to the main level identification code updating device 03 and before the description file updating device 04 (for example, during this period, a concurrent write data is successfully added) If the data file is modified, and the corresponding data file length and the description file are also changed, in the description file updating device 04, the file length of the data file acquired before the data to be written is acquired is the If the request write location of the write data is determined, it is determined that the request write location does not match the total length of the current data file, and the data to be written needs to be added again.
  • FIGS. 6 to 8 are schematic diagrams showing write data for a distributed storage system according to still another preferred embodiment of the present application; specifically, for example, in conjunction with FIG. 6, it is assumed that the data file has data 20K to be written, and two data are recorded.
  • Concurrently write data processes Append-1 and Append-2 assuming that Append-1 is added successfully, as shown in Figure 6, Append-2 adds data to be written to fail (for example, Figure 7, Tempend-2 generated temporary description file) FileInfo is not persisted successfully, that is, Append-2 appends the data to be written to actually fail.
  • Append-3 is executed to re-add the data to be written.
  • Append-3 only needs the total length of the data file, and writes the requested location of the data to be written with the file length of the re-acquired data file, and judges again. Whether the request writing position is consistent with the total length of the current data file, and when they are consistent, updating the description data file based on the main level identification code and increasing the total length of the data file after the data to be written, using the atomic operation Persist the description file for the update.
  • Append-2 has a penalty read, it is determined whether the size of the previous data stored in the data storage block that was last allocated before the data to be written is smaller than the preset capacity. When it is less than, the penalty read step is performed again. And merging the previous data with the data to be written, and re-storing to the data storage block, and updating, in the main-level start-stop information list, start and stop information based on the data storage block, when greater than or When equal to, the data to be written is written into the data to be written and stored in the data storage block, and is further in the main-level start and stop information list. New start and stop information is generated based on the data storage block. As shown in FIG. 7, Append-2 has a penalty read, and in combination with FIG. 6 and FIG.
  • each time the user submits the data to be written the user does not need to simultaneously submit the requested write location, and after each time the user's write data is obtained, the write data is appended to the existing data file.
  • the distributed system automatically acquires the total length of the current data file, and sequentially assembles all the received data blocks into the data file according to the order in which the request is received.
  • Write data includes A, B, and C.
  • the existing data files in the distributed system include A and B
  • the user concurrently uploads two Cs when one of the data to be written C is written After the success, the total length of the data file changes.
  • the first write of the second data to be written C may fail to be written. Then the second data to be written C will reacquire the total length of the data file and re- If the attempt is successful, the two Cs uploaded concurrently will be written into the data file in turn.
  • the final data file includes A, B, C, and C.
  • the scheme in which the user does not specify an additional location is suitable for a scenario that does not care about the order between the additional data, such as collection of program logs in a distributed environment.
  • the device 1 further includes: determining means, configured to determine whether the capacity of the primary-level start-stop information list reaches a capacity threshold or the number of start-stop information reaches a quantity threshold, and when it is reached, the current primary All the data start and stop information in the start and stop information list is transferred to the slave start and stop information list and stored to the newly allocated slave start and stop information storage block, and the slave start and stop information generated based on the slave start and stop information storage block is stored to the The main level start and stop information storage block.
  • determining means configured to determine whether the capacity of the primary-level start-stop information list reaches a capacity threshold or the number of start-stop information reaches a quantity threshold, and when it is reached, the current primary All the data start and stop information in the start and stop information list is transferred to the slave start and stop information list and stored to the newly allocated slave start and stop information storage block, and the slave start and stop information generated based on the slave start and stop information storage block is stored to the The main level start and stop information storage block.
  • the device 1 further includes: a classifying device, configured to check a primary start and stop information storage block RootExtentBlock, if the capacity of the primary-level start and stop information list reaches a capacity threshold, or the number of start and stop information reaches a quantity threshold, Performing multi-level processing (Level Promotion), transferring all data start and stop information in the current main-level start and stop information list to the slave-level start-stop information list and storing to the newly-assigned slave-level start-and-stop information storage block, based on the slave level
  • the slave start and stop information generated by the start and stop information storage block is stored to the master start and stop information storage block.
  • FIG. 9 is a schematic diagram of write data for a distributed storage system according to still another preferred embodiment of the present application; the purpose of Level Promotion is to limit the size of the primary start and end information storage block RootExtentBlock to be effective Support for large files.
  • the purpose of Level Promotion is to limit the size of the primary start and end information storage block RootExtentBlock to be effective Support for large files.
  • there is only data start and stop information that is, multiple start and stop information Extent of Level 0.
  • a certain threshold for example, 1024
  • BlockSize a new storage block is allocated as the secondary start and stop information storage block ExtentBlock EB.
  • the slave start and stop information storage block ExtentBlock EB-1 stores the slave start and stop information list, including all the Level 0Extents described above, generates a Level 1Extent based on the slave start and stop information storage block ExtentBlock EB-1, and the Level 1Extent is stored in the newly allocated master
  • the data start address RangeStart is the data start address RangeStart of the first data start and stop information Extent included in the start and end information storage block ExtentBlock EB-1 is 0, and the data ends.
  • the address RangeEnd is the data end address RangEnd of the last start and end information Extent included in the start and end information storage block ExtentBlock EB-1, and the slave identification code ExtentID corresponding to the slave start/stop information storage block ExtentBlock EB-1 is EB-1.
  • the device 1 further includes: a cache device, configured to set a primary-level start and stop information storage block corresponding to the primary-level start and stop information list before the update to a readable and writable state, and to persist the description After the file succeeds, the primary-level start and end information storage block corresponding to the primary-level start and stop information list before the update is cleared.
  • a cache device configured to set a primary-level start and stop information storage block corresponding to the primary-level start and stop information list before the update to a readable and writable state, and to persist the description After the file succeeds, the primary-level start and end information storage block corresponding to the primary-level start and stop information list before the update is cleared.
  • the main-level start and stop information storage block corresponding to the main-level start and stop information list before the update is set to a readable and writable state, and the update is cleared after the description file is successfully persisted.
  • the main-level start-stop information storage block corresponding to the previous main-level start-stop information list for example, the main-level start-stop information storage block EB-B in FIG. 3 is set to read and write-free in the process of writing data in FIG.
  • the main level start and stop information storage block EB-C update is completed in FIG. 4 and the updated description file Fileinfo is completed
  • the main level start and end information storage block EB-B is cleared; likewise, the data storage block block shown in FIG. D1#0 is cleared after the data storage blocks Block D2#0, D2#1 and D2#2 shown in FIG. 4 are stored and the updated description file Fileinfo is persisted, so that the device stores when the user adds data to be written.
  • the data file still provides the user with the service to get the data.
  • the device for writing data in the embodiment of the present application does not overwrite the data file, but always updates by using the method of copying to the newly allocated storage block and adopts atomic operation for a long time.
  • the described description file is updated, so that in the process of additionally writing data, the data file can be guaranteed not to be damaged, and the readability and consistency of the data are maintained, which is convenient for the user to use.
  • the device for writing data in the embodiment of the present application supports the user to specify the writing of the writing position of the data to be written, and also supports the user not specifying the writing bit of the data to be written. Set the write to improve the flexibility of the user to write data.
  • FIG. 12 is a schematic diagram of an apparatus for acquiring data for a distributed storage system according to an aspect of the present application; wherein the apparatus 2 includes:
  • the main-level start and stop information list obtaining means 12 is configured to obtain a main-level start-stop information list from the main-level start-stop information storage block corresponding to the main-level identification code based on the main-level identification code in the description file;
  • the data start and stop information obtaining means 13 is configured to search for the corresponding data start and stop information from the main level start and stop information list;
  • the data obtaining means 14 is configured to read the corresponding data from the data storage block corresponding to the data start and stop information based on the data start and stop information.
  • the data start-stop information acquiring device 13 further includes: a slave-level corresponding to the slave-level start and stop information based on the slave-level start and stop information Find the corresponding data start and stop information in the start and stop information storage block.
  • the binary search method is used to find the corresponding data start and stop information.
  • the device 2 includes, but is not limited to, a network device, or a device formed by integrating a user device and a network device through a network.
  • the network device includes an electronic device capable of automatically performing numerical calculation and information processing according to an instruction set or stored in advance, and the hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), and a programmable gate.
  • the network device includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a plurality of servers; wherein the cloud is composed of a large number of computers or network servers based on Cloud Computing Among them, cloud computing is a kind of distributed computing, a virtual supercomputer composed of a group of loosely coupled computers.
  • the network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), and the like.
  • the device 2 may also be a script program running on the network device, or a user device and a network device, or a device formed by integrating a network device and a touch terminal through a network.
  • a script program running on the network device, or a user device and a network device, or a device formed by integrating a network device and a touch terminal through a network.
  • the foregoing device 2 is only an example, and other existing or future network devices 3 may be applicable to the present application, and should also be included in the scope of the present application. The way is included here.
  • the device 2 can be arranged in one device together with the device 1.
  • the device for acquiring data in the embodiment of the present application is based on the data written On the top, the data can be obtained in time for the user to read, and can be read at any time, without being affected by the written data. Convenient for the user's use.
  • the present application can be implemented in software and/or a combination of software and hardware, for example, using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device.
  • the software program of the present application can be executed by a processor to implement the steps or functions described above.
  • the software programs (including related data structures) of the present application can be stored in a computer readable recording medium such as a RAM memory, a magnetic or optical drive or a floppy disk and the like.
  • some of the steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.
  • a portion of the present application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide a method and/or technical solution in accordance with the present application.
  • the program instructions for invoking the method of the present application may be stored in a fixed or removable recording medium, and/or transmitted by a data stream in a broadcast or other signal bearing medium, and/or stored in a The working memory of the computer device in which the program instructions are run.
  • an embodiment in accordance with the present application includes a device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, triggering
  • the apparatus operates based on the aforementioned methods and/or technical solutions in accordance with various embodiments of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Techniques For Improving Reliability Of Storages (AREA)

Abstract

用于分布式存储系统的写入数据、获取数据的方法和设备，所述方法在写入数据过程中，不会对数据文件进行覆盖写，而总是采用拷贝到新分配的存储块的方式进行更新并采用原子操作持久化更新的所述描述文件，从而在追加写入数据的过程中，能保证数据文件不会被损坏，并维持数据的可读性和一致性，方便了用户的使用。该写入数据的方法支持用户指定待写数据的写入位置的写入，也支持用户不指定待写数据的写入位置的写入，提高用户写入数据的灵活性。

Description

用于分布式存储系统的写入数据、获取数据的方法和设备
本申请要求2015年09月09日递交的申请号为201510571183.7、发明名称为“用于分布式存储系统的写入数据、获取数据的方法和设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机领域，尤其涉及一种用于分布式存储系统的写入数据、获取数据的方法和设备。
背景技术
在云存储系统中，由于用户的网络环境限制或者用户场景特点限制，用户可能需要分多次才能完成对一个数据文件的写入，比如用户写大数据文件时，往往需要采用追加写模式。在分布式云存储系统中，支持追加写需要解决用户并发写、如何保证数据完整性等难题，因此如何实现数据文件追加写模式是云存储系统的关键技术方案。
发明内容
本申请要解决的技术问题是，提供一种能够在分布式云存储系统中实现追加的写入数据以及获取数据的方法和设备，包括用户指定追加位置和用户不指定追加位置的追加待写数据，以及支持在分布式云存储系统读取数据的方法和设备。
为解决上述技术问题，本申请提供了一种用于分布式存储系统的写入数据的方法，其中，所述方法包括：
获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；
生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
基于所述主级起止信息存储块生成所述主级识别码；
基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，并利用原子操作持久化更新的所述描述文件。
进一步地，所述获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块包括：
判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是 否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
进一步地，针对用户指定追加位置的写入数据的方式，所述方法还包括：
在获取所述待写数据之前，获取所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，获取所述待写数据。
进一步地，针对用户指定追加位置的写入数据的方式，所述方法还包括：
在获取所述待写数据前，获取所述数据文件的文件长度；
所述基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用原子操作持久化更新的所述描述文件包括：
以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
进一步地，针对用户不指定追加位置的写入数据方式，所述获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块还包括：
所述请求写入位置与当前数据文件的总长度不一致时，则判断在将所述待写数据存储至新分配的数据存储块中是否有所述惩罚读步骤：
若有所述惩罚读步骤，则重新判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息；
若无所述惩罚度步骤，则重新获取所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
进一步地，所述方法还包括：
判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则
将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，
将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
进一步地，所述方法还包括：
将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。
进一步地，每一所述数据存储块包括一个或多个单元存储块，其中，所述多个单元存储块的容量均相同。
进一步地，所述数据起止信息包括以下至少任一项：
所述数据存储块识别码；
所述数据存储块中数据的起止地址信息；
所述数据存储块中单元存储的个数。
根据本申请另一方面提供的一种用于分布式存储系统的获取数据的方法，其中，所述方法包括：
获取描述文件；
基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储块中获取主级起止信息列表；
从所述主级起止信息列表中查找相应的数据起止信息；
基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
进一步地，当所述主级起止信息列表中还包括从级起止信息时，所述从所述主级起止信息列表中查找相应的数据起止信息还包括：
基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。
进一步地，所述从所述主级起止信息列表中查找相应的数据起止信息包括：
采用二分查找法查找相应的数据起止信息。
根据本申请再一方面提供的一种用于分布式存储系统的写入数据的设备，其中，所述设备包括：
写入装置，用于获取用户的待写数据，并将所述待写数据存储至新分配的数据存储 块；
起止信息更新装置，用于生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
主级识别码更新装置，用于基于所述主级起止信息存储块生成所述主级识别码；
描述文件更新装置，用于基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。
进一步地，所述设备还包括：
判断装置，在获取所述待写数据之前，获取所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，获取所述待写数据。
进一步地，所述写入装置包括：
惩罚读单元，用于判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
进一步地，针对用户不指定追加位置的写入数据的方式，所述写入装置还包括：
在获取所述待写数据前，获取所述数据文件的文件长度；
所述描述文件更新装置还包括：
以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
进一步地，针对用户不指定追加位置的写入数据方式，所述描述文件更新装置还用于：
所述请求写入位置与当前数据文件的总长度不一致时，则判断将所述待写数据存储至新分配的数据存储块中是否有所述惩罚读步骤：
若有所述惩罚读步骤，则重新判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更新 基于所述数据存储块生成起止信息；
若无所述惩罚度步骤，则重新获取所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
进一步地，所述方法还包括：
分级装置，用于判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
进一步地，所述方法还包括：
缓存装置，用于将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。
进一步地，每一所述数据存储块包括一个或多个单元存储块，其中，所述多个单元存储块的容量均相同。
进一步地，所述数据起止信息包括以下至少任一项：
所述数据存储块识别码；
所述数据存储块中数据的起止地址信息；
所述数据存储块中单元存储的个数。
根据本申请再一方面提供的一种用于分布式存储系统的获取数据的设备，其中，所述方法包括：
描述文件获取装置，用于获取描述文件；
主级起止信息列表获取装置，用于基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储块中获取主级起止信息列表；
数据起止信息获取装置，用于从所述主级起止信息列表中查找相应的数据起止信息；
数据获取装置，用于基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
进一步地，当所述主级起止信息列表中还包括从级起止信息时，所述数据起止信息获取装置还用于：
基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。
进一步地，所述数据起止信息获取装置用于：采用二分查找法查找相应的数据起止信息。
与现有技术相比，在本申请的实施例中所述的用于分布式存储系统的写入数据、获取数据的方法和设备，其中所述写入数据的方法包括获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；基于所述主级起止信息存储块生成所述主级识别码；基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。所述方法不会对数据文件进行覆盖写，而总是采用拷贝到新分配的存储块的方式进行更新并采用原子操作持久化更新的所述描述文件，从而在在追加写入数据的过程中，能保证数据文件不会被损坏，并维持数据的可读性和一致性，方便了用户的使用。
进一步地，在本申请的实施例的写入数据的方法支持用户指定待写数据的写入位置的写入，也支持用户不指定待写数据的写入位置的写入，提高用户写入数据的灵活性。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：
图1示出根据本申请一个方面的用于分布式存储系统的写入数据的方法示意图；
图2至图3示出根据本申请一优选的实施例的用于分布式存储系统的写入数据的示意图；
图4至图5示出根据本申请另一优选的实施例的用于分布式存储系统的写入数据的示意图；
图6至图8示出本申请再一优选的实施例的用于分布式存储系统的写入数据的示意图；
图9示出根据本申请又一优选的实施例的用于分布式存储系统的写入数据的示意图；
图10示出根据本申请一个方面的用于分布式存储系统的获取数据的方法示意图；
图11示出根据本申请一个方面的用于分布式存储系统的写入数据的设备示意图；
图12示出根据本申请一个方面的用于分布式存储系统的获取数据的设备示意图。
附图中相同或相似的附图标记代表相同或相似的部件。
具体实施方式
下面结合附图对本申请作进一步详细描述。
在现有云存储系统中，通常采用两种方案来实现追加写：1、用户在第一次上传之前，提前将整个数据文件分成多个数据块(Part)和数据块的大小，具体地，每一个Part都被赋予了一个唯一标识，这些唯一标识称为PartId，例如：Part 1、Part 2、Part 3……，在用户上传数据块时必须指定当前对哪个PartId进行操作，通过PartId可以唯一地索引到某个Part，而每个PartId对应的Part作为一个单独的数据文件存储，对该PartId的写操作会覆盖已存在的Part数据文件。当所有的PartId对应的Part都写成功后，用户通过提交操作(Commit)，指定需要将哪些PartId以哪种的顺序组装成一个数据文件，Commit操作可以丢弃某些PartId，也可以重复出现某些PartId，而相应的Part根据指定的顺序组装或者丢弃。用户在commit操作过程中，数据文件处于不可见状态，用户无法对写了一部分数据的数据文件进行读取操作；并且在Commit操作之后，整个数据文件可见了，但是用户无法再写入新的Part或者改写Part。当用户上传某个Part失败时，用户需要重写整个Part，无法追加写该Part；此外，用户需要提前将整个数据文件进行划分，当无法提前预知数据文件大小时，往往会造成每个Part过小，Part数目过多，或者单个Part过大，增加写入失败的概率。2、云存储系统支持对多个小数据文件逻辑上组成一个大数据文件的方法。用户上传多个小数据文件，并及时更新某个元数据文件，其中，元数据文件表示了用户最终想要的大数据文件，它包含所有小数据文件的列表。由于小数据文件和组合数据文件之间松散的关系，无法保证小数据文件是否被删除或者更改，导致组合数据文件的完整性得不到保障，因此无法保证数据的完整性。
结合现有技术中的问题，图1示出根据本申请一个方面的用于分布式存储系统的写入数据的方法，其中，所述方法包括：
步骤S01：获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；
步骤S02：生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
步骤S03：基于所述主级起止信息存储块生成所述主级识别码；
步骤S04：基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。
在此，所述数据起止信息基于所述数据存储块生成，用于记录待写数据在数据文件中的起止地址，根据所述数据起止信息可以找到存储相应数据存储块，进而找到相应待写数据。
优选的，所述数据起止信息包括：所述数据存储块识别码；所述数据存储块中数据的起止地址信息；所述待写数据的大小；所述数据存储块中单元存储的个数。所述数据存储块识别码用于唯一识别数据存储块，所述数据存储块中数据的起止地址信息包括相应待写数据在数据文件中的起始位置和结束位置。
优选地，所述描述文件可以包括：数据文件的总长度，修改时间及主级识别码，所述描述文件用于便于读取数据或追加待写数据。
此外，每一所述数据存储块包括一个或多个单元存储块，其中，当所述数据存储块具有多个单元存储块时，所述多个单元存储块的容量均相同。
本申请所述方法在每次获取待写数据时，通过分配数据存储块用以存储待写数据，并分配主级起止信息存储块用以存储具有数据起止信息的主级起止信息列表，在根据所述主级起止信息存储块的主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，再利用原子操作持久化更新的描述文件。在此，每次追加待写数据时，都重新分配数据存储块和主级起止信息存储块(Copy-On-Write)，保证追加待写数据的过程不会对已有数据文件进行覆盖写，并且，所述描述文件在持久化之前存储在以临时的描述文件形式存在(例如在易失媒介，如内存中的临时的描述文件)，所述描述文件持久化过程是利用原子操作实现的，优选的，包括两种原子操作：Put-If-Not-Exist和Test-And-Set。Put-If-Not-Exist(FileName,FileInfo)操作只有在文件名不存在的情况下，才能新增一条该文件名到文件元数据的记录；Test-And-Set(FileName,NewFileInfo,OldFileInfo)操作只有在文件名存在，且当前描述文件(FileInfo)的内容和更新前的描述文件(OldFileInfo)对应时，才能持久化更新的描述文件。其中，在所述描述文件在持久化之前当前的待写数据对用户不可见，即用户不能读取当前的待写数据(但云存储系统中已有数据文件对用户是可见的)，从而保持数据的一致性和稳定性。
当然，本领域技术人员应能理解上述原子操作指令仅为举例，其他现有的或今后可能出现的其他原子操作或其他持久化方式如可适用于本申请，也应包含在本申请保护范 围以内，并在此以引用方式包含于此。
在此，所获取的用户的待写数据可以是由用户向分布式系统首次提交的待写数据，还可以是由用户针对分布式系统中已有数据文件提交的追加待写数据。本申请所述方法可以将用户首次提交的待写数据存储并生成数据文件，或将用户提交的追加待写数据增加至已有所述数据文件。
图2至图3示出根据本申请一优选的实施例的用于分布式存储系统的写入数据的示意图。当用户首次提交的待写数据时，结合图2，在云存储系统中，根据数据类型，可将底层存储划分为数据元存储区Metastore和数据存储区DataStore，MetaStore维护了文件名(Readme.txt)到元数据的对应关系，所述元数据包含了寻址到待写数据的信息，DataStore用于存储待写数据及待写数据的相关数据起止信息，DataStore被划分为多个固定大小的块，每个块都有相应的地址，根据块地址可以读取块的内容。当MetaStore中有某个文件名时，根据该文件就对用户进行追加写或读取。其中，MetaStore和DataStore主要是逻辑上的区分，它们可能实现在同一个底层存储上。
当然，本领域技术人员应能理解上述根据数据类型进行划分存储区的方式仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
具体地，在所述步骤S01中，获取用户的待写数据，分配数据存储块D1存储待写数据；在所述步骤S02中，依据所述数据存储块D1生成数据起止信息Extent，数据起止信息Extent包括待写数据在数据文件中的起始地址Range Start、结束地址Range End、数据存储块D1的数据存储块识别码ExtentID，数据存储块D1中单元存储块的个数Nblks，以及主级信息level，通常当level为0时，表示直接指向存储待写数据的数据存储块，接着，利用数据起止信息Extent更新主级起止信息列表中并存储至主级起止信息存储块RootExtentBlock；在所述步骤S03中，基于主级起止信息存储块RootExtentBlock生成主级识别码RootExtentID；在所述步骤S04中，基于所述主级识别码RootExtentID和增加所述待写数据后数据文件的总长度FileLength更新描述文件Fileinfo，并利用原子操作Put-If-Not-Exist持久化更新的所述描述文件Fileinfo。
当将用户提交的追加待写数据增加至已有所述数据文件时，可以包括两种方式：用户指定追加位置的写入数据方式和用户不指定追加位置的写入数据方式，以下结合所述两种方式对本申请所述方法进行进一步的描述。
对于用户指定追加位置的写入数据方式，在所述步骤S01中，获取所述待写数据的 请求写入位置(Position)，判断所述待写数据的请求写入位置与当前数据文件的总长度是否一致，当一致时，则获取相应所述待写数据，当不一致时，则拒绝获取相应所述待写数据，可同时向所述用户设备反馈当前数据文件的总长度。针对用户指定追加位置的方案，可以在用户并发待写数据时，防止并发的写入数据请求写入相同写入位置，进而避免多份重复的写入数据组装进数据文件。比如，假设写入数据包括A、B和C，当分布式系统中已有数据文件包括A和B，当前用户并发写入数据C，当用户并发上传了两个C，则并发上传的两个C中只有一个会上传成功，则写入数据包括A、B和C，而不是A、B和两个C。该用户指定追加位置的写入数据的增加方式可以防止用户并发写入导致数据文件出错。
在具体的场景中，结合图3，用户第一次已上传待写数据包括20K，则数据存储块Block D1#0中存储该20K的待写数据，并生成数据起止信息Extent1存储至起止信息存储块RootExtentBlock EB-A中，相应地，所述描述文件FileInfo的文件长度Filelength为20K，所述主级识别码RootExtentID为EB-A。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
用户在提交待写数据时，同时会提交请求的写入位置(Position)，设备在步骤S01获取所述待写数据之前，首先判断用户提交的写入位置是否等于描述文件的文件长度Filelength，若不等于则拒绝写入请求，若等于，则获取所述待写数据；
图4至图5示出根据本申请另一优选的实施例的用于分布式存储系统的写入数据的示意图。进一步地，结合图4，在所述步骤S01中，获取所述待写数据后，判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
接上例，在图2和图3所示基础上，结合图4，假设所述数据存储块的单元存储块的预设容量为64K，用户提交的追加待写数据为130K，追加待写数据前最后一次分配的数据存储块所存储的前次数据的大小为20K，小于预设容量64K，则进行执行一次惩罚读步骤，包括重新读取前次数据20K，并将前次数据20K与当前提交的追加待写数据的130K合并，一同存储至新分配的数据存储块Block D2中，包括三个单元存储块Block D2#0，Block D2#1和Block D2#2。在所述步骤S02中，根据新分配的数据存储块Block D2， 生成关于所述数据存储块的数据起止信息Extent2，并存储至主级起止信息存储块RootExtentBlock EB-B中，数据起止信息Extent2包括待写数据起始地址0和待写数据的结束地址150K，所述数据存储块识别码D2，单元存储块Nblks个数3，及层级level为0；在所述步骤S03中，生成所述数据起止信息RootExtentBlock EB-B的主级识别码EB-B，在所述步骤S04中，采用原子操作Test-And-Set更新所述描述文件，包括更新的数据文件的文件长度Filelength为150K，主级识别码RootExtentID为EB-B，
若获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小或等于预设容量时，则不执行惩罚读步骤，直接将所述待写数据存储至新分配的数据存储块。
继续接上例，在图2至图4的基础上，结合图5，在所述步骤S01中，用户再次提交的待写数据10K，判断获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小为150K大于预设容量64K，则不重新读取前次数据150K，直接将本次待写数据10K存储至新分配的数据存储块Block D3中，接着在所述步骤S02～步骤S03中，生成数据起止信息Extent3增加至主级起止信息列表中，并存储至新分配的主级起止信息存储块RootExtentBlock EB-C，并生成主级识别码EB-C，在所述步骤S04中，基于所述主级识别码EB-C和数据文件的总长度FileLength为160K更新描述文件Fileinfo，并利用原子操作Test-And-Set持久化更新的所述描述文件Fileinfo。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
在另一实施例中，当用户不指定待写数据的请求写入位置时，用户每次提交待写数据时，不需同时提交请求的写入位置，设备在获取所述待写数据前，先获取数据文件的总长度；接着，执行所述步骤S01至步骤S03，执行过程与图2至图5所示的步骤S01至步骤S03的内容相同或基本相同，为简明起见，以引用的方式包含于此，不再赘述，当执行至步骤S04时，以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，则基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述描述文件。在所述步骤S01至步骤S03中且在所述步骤S04之前，设备中存储的数据文件如并未发生改变，则相应数据文件长度及描述文件也未发生改变，则在步骤S04中，在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，则判断所述请求写入位置与当前数据文件的总 长度一致，则基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。若在所述步骤S01至步骤S03中且在所述步骤S04之前，设备中存储的数据文件如发生改变(例如在此期间，一并发写入数据追加成功，导致数据文件被修改)，则相应数据文件长度及描述文件也发生改变，则在步骤S04中，在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，则判断所述请求写入位置与当前数据文件的总长度不一致，则需重新追加待写数据。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
图6至图8示出本申请再一优选的实施例的用于分布式存储系统的写入数据的示意图；具体地，例如，结合图6，设数据文件已有待写数据20K，记两个并发写入数据过程Append-1和Append-2，假设Append-1追加成功，形成如图6所示，Append-2追加待写数据失败，(例如图7，Append-2生成的临时的描述文件Fileinfo并未持久化成功，即Append-2追加待写数据实际失败)，则针对Append-2，执行Append-3以重新追加待写数据。
若Append-2没有发生惩罚读，则Append-3只需所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
Append-2有发生惩罚读，则重新判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息。由图7所示，Append-2有发生惩罚读，结合图6和图8，重新判断在获取所述待写数据前最后一次分配的数据存储块Block D#0所存储的前次数据的大小为20K，小于预设容量64K，则重新读取前次数据20K，与所述待写数据70K合并，并重新存储至所述数据存储块Block D3中。
针对当用户不指定追加位置的方式，用户每次提交待写数据时，不需同时提交请求的写入位置，在每次获取用户的写入数据后，将写入数据追加到已有数据文件之后，当并发待写数据时，由分布式系统自行获取当前数据文件的总长度，并根据接收到请求的顺序，依次将所有接收到的数据块全部组装进数据文件中，接上例，假设写入数据包括A、B和C，当分布式系统中已有数据文件包括A和B，当前用户并发写入数据C，当用户并发上传了两个C，当其中一个待写数据C写入成功后，数据文件的总长度发生变化，第二个待写数据C的第一次写入则可能会写入失败，则第二个待写数据C会重新获取数据文件的总长度，并重新尝试写入成功，则并发上传的两个C会依次写入数据文件中，最终数据文件包括A、B、C、C。所述用户不指定追加位置的方案适合于不关心各次追加数据之间顺序的场景，比如分布式环境下程序日志的收集等。
优选地，所述方法还包括：判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
具体地，所述方法还包括检查主级起止信息存储块RootExtentBlock，如果所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，就进行多级处理(Level Promotion)，将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
图9示出根据本申请又一优选的实施例的用于分布式存储系统的写入数据的示意图；Level Promotion的目的是限制主级起止信息存储块RootExtentBlock的大小，以有效的支持大文件。具体地，起初主级起止信息存储块RootExtentBlock里只有数据起止信息，即Level 0的多个起止信息Extent。结合图9，当这些Level 0Extent数目超过一定阈值(例如1024)，且主级起止信息存储块RootExtentBlock大小大于或等于容量阈值BlockSize时，就分配一个新的存储块作为从级起止信息存储块ExtentBlock EB-1，该从级起止信息存储块ExtentBlock EB-1存储从级起止信息列表，包括上述所有Level 0Extent，基于从级起止信息存储块ExtentBlock EB-1生成Level 1Extent，Level 1Extent存储于新分配的主级起止信息存储块RootExtentBlock EB-D中，其数据起始地址RangeStart是从级起止信息存储块ExtentBlock EB-1所包含的第一个数据起止信息Extent的数据起始地址RangeStart，其数据结束地址RangeEnd是从级起止信息存储块ExtentBlock EB-1所包含 的最后一个起止信息Extent的数据结束地址RangEnd，从级起止信息存储块ExtentBlock EB-1对应的从级识别码ExtentID为EB-1。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
优选地，所述方法还包括：将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。当所述方法执行所述步骤S01至步骤S04以写入数据时，更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块，例如图3中主级起止信息存储块EB-B在图4写入数据过程中，设置为可读不可写状态，待图4所示主级起止信息存储块EB-C更新完成且更新的描述文件Fileinfo持久化完成后，在清除主级起止信息存储块EB-B；同样，对于图3所示的数据存储块Block D1#0在图4所示的数据存储块Block D2#0，D2#1和D2#2存储完成且更新的描述文件Fileinfo持久化完成后清除，以实现用户在追加待写数据时，设备存储的数据文件仍可为用户提供获取数据的服务。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
与现有技术相比，在本申请的实施例中所述写入数据的方法包括获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；基于所述主级起止信息存储块生成所述主级识别码；基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。所述方法不会对数据文件进行覆盖写，而总是采用拷贝到新分配的存储块的方式进行更新并采用原子操作持久化更新的所述描述文件，从而在在追加写入数据的过程中，能保证数据文件不会被损坏，并维持数据的可读性和一致性，方便了用户的使用。
进一步地，在本申请的实施例的写入数据的方法支持用户指定待写数据的写入位置的写入，也支持用户不指定待写数据的写入位置的写入，提高用户写入数据的灵活性。
图10示出根据本申请一个方面的用于分布式存储系统的获取数据的方法示意图；其中，所述方法包括：
步骤S11：获取描述文件；
步骤S12：基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储块中获取主级起止信息列表；
步骤S13：从所述主级起止信息列表中查找相应的数据起止信息；
步骤S14：基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
进一步地，当所述主级起止信息列表中还包括从级起止信息时，所述步骤S13还包括：基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。优选地，采用二分查找法查找相应的数据起止信息。
与现有技术相比，在本申请的实施例中所述获取数据的方法在所述写入数据的基础上，能够及时获取数据以供用户读取，并能够随时读取，不受写入数据的影响。方便了用户的使用。
进一步地，在本申请的实施例的写入数据的方法支持用户指定待写数据的写入位置的写入，也支持用户不指定待写数据的写入位置的写入，提高用户写入数据的灵活性。
图11示出根据本申请一个方面的用于分布式存储系统的写入数据的设备示意图，其中，所述设备1包括：
写入装置01，用于获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；
起止信息更新装置02，用于生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
主级识别码更新装置03，用于基于所述主级起止信息存储块生成所述主级识别码；
描述文件更新装置04，用于基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。
在此，所述设备1包括但不限于用户设备、或用户设备与网络设备通过网络相集成所构成的设备。所述用户设备其包括但不限于任何一种可与用户通过触摸板进行人机交互的移动电子产品，例如智能手机、PDA等，所述移动电子产品可以采用任意操作系统，如android操作系统、iOS操作系统等。其中，所述网络设备包括一种能够按照事先设定 或存储的指令，自动进行数值计算和信息处理的电子设备，其硬件包括但不限于微处理器、专用集成电路(ASIC)、可编程门阵列(FPGA)、数字处理器(DSP)、嵌入式设备等。所述网络包括但不限于互联网、广域网、城域网、局域网、VPN网络、无线自组织网络(Ad Hoc网络)等。优选地，设备1还可以是运行于所述用户设备、或用户设备与网络设备、触摸终端或网络设备与触摸终端通过网络相集成所构成的设备上的脚本程序。当然，本领域技术人员应能理解上述设备1仅为举例，其他现有的或今后可能出现的设备1如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
上述各装置之间是持续不断工作的，在此，本领域技术人员应理解“持续”是指上述各装置分别实时地或者按照设定的或实时调整的工作模式要求，直至所述设备1停止工作。
在此，所述数据起止信息基于所述数据存储块生成，用于记录待写数据在数据文件中的起止地址，根据所述数据起止信息可以找到存储相应数据存储块，进而找到相应待写数据。
优选的，所述数据起止信息包括：所述数据存储块识别码；所述数据存储块中数据的起止地址信息；所述待写数据的大小；所述数据存储块中单元存储的个数。所述数据存储块识别码用于唯一识别数据存储块，所述数据存储块中数据的起止地址信息包括相应待写数据在数据文件中的起始位置和结束位置。
优选地，所述描述文件可以包括：数据文件的总长度，修改时间及主级识别码，所述描述文件用于便于读取数据或追加待写数据。
此外，每一所述数据存储块包括一个或多个单元存储块，其中，当所述数据存储块具有多个单元存储块时，所述多个单元存储块的容量均相同。
本申请所述设备1在每次获取待写数据时，通过分配数据存储块用以存储待写数据，并分配主级起止信息存储块用以存储具有数据起止信息的主级起止信息列表，在根据所述主级起止信息存储块的主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，再利用原子操作持久化更新的描述文件。在此，每次追加待写数据时，都重新分配数据存储块和主级起止信息存储块(Copy-On-Write)，保证追加待写数据的过程不会对已有数据文件进行覆盖写，并且，所述描述文件在持久化之前存储在以临时的描述文件形式存在(例如在易失媒介，如内存中的临时的描述文件)，所述描述文件持久化过程是利用原子操作实现的，优选的，包括两种原子操作：Put-If-Not-Exist和Test-And-Set。Put-If-Not-Exist(FileName,FileInfo)操作只有在文件名不存在的情况下，才能新增一条该 文件名到文件元数据的记录；Test-And-Set(FileName,NewFileInfo,OldFileInfo)操作只有在文件名存在，且当前描述文件(FileInfo)的内容和更新前的描述文件(OldFileInfo)对应时，才能持久化更新的描述文件。其中，在所述描述文件在持久化之前当前的待写数据对用户不可见，即用户不能读取当前的待写数据(但云存储系统中已有数据文件对用户是可见的)，从而保持数据的一致性和稳定性。
在此，所获取的用户的待写数据可以是由用户向分布式系统首次提交的待写数据，还可以是由用户针对分布式系统中已有数据文件提交的追加待写数据。本申请所述设备1可以将用户首次提交的待写数据存储并生成数据文件，或将用户提交的追加待写数据增加至已有所述数据文件。
图2至图3示出根据本申请一优选的实施例的用于分布式存储系统的写入数据的示意图。当用户首次提交的待写数据时，结合图2，在云存储系统中，根据数据类型，可将底层存储划分为数据元存储区Metastore和数据存储区DataStore，MetaStore维护了文件名(Readme.txt)到元数据的对应关系，所述元数据包含了寻址到待写数据的信息，DataStore用于存储待写数据及待写数据的相关数据起止信息，DataStore被划分为多个固定大小的块，每个块都有相应的地址，根据块地址可以读取块的内容。当MetaStore中有某个文件名时，根据该文件就对用户进行追加写或读取。其中，MetaStore和DataStore主要是逻辑上的区分，它们可能实现在同一个底层存储上。
当然，本领域技术人员应能理解上述根据数据类型进行划分存储区的方式仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
具体地，在所述写入装置01中，获取用户的待写数据，分配数据存储块D1存储待写数据；在所述起止信息更新装置02中，依据所述数据存储块D1生成数据起止信息Extent，数据起止信息Extent包括待写数据在数据文件中的起始地址Range Start、结束地址Range End、数据存储块D1的数据存储块识别码ExtentID，数据存储块D1中单元存储块的个数Nblks，以及主级信息level，通常当level为0时，表示直接指向存储待写数据的数据存储块，接着，利用数据起止信息Extent更新主级起止信息列表中并存储至主级起止信息存储块RootExtentBlock；在所述主级识别码更新装置03中，基于主级起止信息存储块RootExtentBlock生成主级识别码RootExtentID；在所述描述文件更新装置04中，基于所述主级识别码RootExtentID和增加所述待写数据后数据文件的总长度FileLength更新描述文件Fileinfo，并利用原子操作Put-If-Not-Exist持久化更新的所述描 述文件Fileinfo。
当将用户提交的追加待写数据增加至已有所述数据文件时，可以包括两种方式：用户指定追加位置的写入数据方式和用户不指定追加位置的写入数据方式，以下结合所述两种方式对本申请所述设备1进行进一步的描述。
对于用户指定追加位置的写入数据方式，在所述写入装置01中，获取所述待写数据的请求写入位置(Position)，判断所述待写数据的请求写入位置与当前数据文件的总长度是否一致，当一致时，则获取相应所述待写数据，当不一致时，则拒绝获取相应所述待写数据，可同时向所述用户反馈当前数据文件的总长度。针对用户指定追加位置的方案，可以在用户并发待写数据时，防止并发的写入数据请求写入相同写入位置，进而避免多份重复的写入数据组装进数据文件。比如，假设写入数据包括A、B和C，当分布式系统中已有数据文件包括A和B，当前用户并发写入数据C，当用户并发上传了两个C，则并发上传的两个C中只有一个会上传成功，则写入数据包括A、B和C，而不是A、B和两个C。该用户指定追加位置的写入数据的增加方式可以防止用户并发写入导致数据文件出错。
在具体的场景中，结合图3，用户第一次已上传待写数据包括20K，则数据存储块Block D1#0中存储该20K的待写数据，并生成数据起止信息Extent1存储至起止信息存储块RootExtentBlock EB-A中，相应地，所述描述文件FileInfo的文件长度Filelength为20K，所述主级识别码RootExtentID为EB-A。
用户在提交待写数据时，同时会提交请求的写入位置(Position)，设备在写入装置01获取所述待写数据之前，首先判断用户提交的写入位置是否等于描述文件的文件长度Filelength，若不等于则拒绝写入请求，若等于，则获取所述待写数据；
图4至图5示出根据本申请另一优选的实施例的用于分布式存储系统的写入数据的示意图。进一步地，结合图4，所述写入装置01还包括惩罚读单元，用于在获取所述待写数据后，判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
接上例，在图2和图3所示基础上，结合图4，假设所述数据存储块的单元存储块的预设容量为64K，用户提交的追加待写数据为130K，追加待写数据前最后一次分配的数据存储块所存储的前次数据的大小为20K，小于预设容量64K，则进行执行一次惩罚读步骤，包括重新读取前次数据20K，并将前次数据20K与当前提交的追加待写数据 的130K合并，一同存储至新分配的数据存储块Block D2中，包括三个单元存储块Block D2#0，Block D2#1和Block D2#2。在所述起止信息更新装置02中，根据新分配的数据存储块Block D2，生成关于所述数据存储块的数据起止信息Extent2，并存储至主级起止信息存储块RootExtentBlock EB-B中，数据起止信息Extent2包括待写数据起始地址0和待写数据的结束地址150K，所述数据存储块识别码D2，单元存储块Nblks个数3，及层级level为0；在所述主级识别码更新装置03中，生成所述数据起止信息RootExtentBlock EB-B的主级识别码EB-B，在所述描述文件更新装置S04中，采用原子操作Test-And-Set更新所述描述文件，包括更新的数据文件的文件长度Filelength为150K，主级识别码RootExtentID为EB-B，
若获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小或等于预设容量时，则不执行惩罚读步骤，直接将所述待写数据存储至新分配的数据存储块。
继续接上例，在图2至图4的基础上，结合图5，写入装置01用户再次提交的待写数据10K，判断获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小为150K大于预设容量64K，则不重新读取前次数据150K，直接将本次待写数据10K存储至新分配的数据存储块Block D3中，接着起止信息更新装置02～主级识别码更新装置S03中，生成数据起止信息Extent3增加至主级起止信息列表中，并存储至新分配的主级起止信息存储块RootExtentBlock EB-C，并生成主级识别码EB-C，描述文件更新装置04中，基于所述主级识别码EB-C和数据文件的总长度FileLength为160K更新描述文件Fileinfo，并利用原子操作Test-And-Set持久化更新的所述描述文件Fileinfo。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
在另一实施例中，当用户不指定待写数据的请求写入位置时，用户每次提交待写数据时，不需同时提交请求的写入位置，设备在获取所述待写数据前，先获取数据文件的总长度；接着，执行所述写入装置01至主级识别码更新装置03，执行过程与图2至图5所示的写入装置01至主级识别码更新装置03的内容相同或基本相同，为简明起见，以引用的方式包含于此，不再赘述，当执行至描述文件更新装置04时，以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，则基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述描 述文件。在所述写入装置01至主级识别码更新装置03中且在所述描述文件更新装置04之前，设备中存储的数据文件如并未发生改变，则相应数据文件长度及描述文件也未发生改变，则在描述文件更新装置04中，在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，则判断所述请求写入位置与当前数据文件的总长度一致，则基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。若在所述写入装置01至主级识别码更新装置03中且在所述描述文件更新装置04之前，设备中存储的数据文件如发生改变(例如在此期间，一并发写入数据追加成功，导致数据文件被修改)，则相应数据文件长度及描述文件也发生改变，则在描述文件更新装置04中，在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，则判断所述请求写入位置与当前数据文件的总长度不一致，则需重新追加待写数据。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
图6至图8示出本申请再一优选的实施例的用于分布式存储系统的写入数据的示意图；具体地，例如，结合图6，设数据文件已有待写数据20K，记两个并发写入数据过程Append-1和Append-2，假设Append-1追加成功，形成如图6所示，Append-2追加待写数据失败，(例如图7，Append-2生成的临时的描述文件FileInfo并未持久化成功，即Append-2追加待写数据实际失败)，则针对Append-2，执行Append-3以重新追加待写数据。
若Append-2没有发生惩罚读，则Append-3只需所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
Append-2有发生惩罚读，则重新判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更 新基于所述数据存储块生成起止信息。由图7所示，Append-2有发生惩罚读，结合图6和图8，重新判断在获取所述待写数据前最后一次分配的数据存储块Block D#0所存储的前次数据的大小为20K，小于预设容量64K，则重新读取前次数据20K，与所述待写数据70K合并，并重新存储至所述数据存储块Block D3中。
针对当用户不指定追加位置的方式，用户每次提交待写数据时，不需同时提交请求的写入位置，在每次获取用户的写入数据后，将写入数据追加到已有数据文件之后，当并发待写数据时，由分布式系统自行获取当前数据文件的总长度，并根据接收到请求的顺序，依次将所有接收到的数据块全部组装进数据文件中，接上例，假设写入数据包括A、B和C，当分布式系统中已有数据文件包括A和B，当前用户并发写入数据C，当用户并发上传了两个C，当其中一个待写数据C写入成功后，数据文件的总长度发生变化，第二个待写数据C的第一次写入则可能会写入失败，则第二个待写数据C会重新获取数据文件的总长度，并重新尝试写入成功，则并发上传的两个C会依次写入数据文件中，最终数据文件包括A、B、C、C。所述用户不指定追加位置的方案适合于不关心各次追加数据之间顺序的场景，比如分布式环境下程序日志的收集等。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
优选地，所述设备1还包括：判断装置，用于判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
具体地，所述设备1还包括：分级装置，用于检查主级起止信息存储块RootExtentBlock，如果所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，就进行多级处理(Level Promotion)，将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
图9示出根据本申请又一优选的实施例的用于分布式存储系统的写入数据的示意图；Level Promotion的目的是限制主级起止信息存储块RootExtentBlock的大小，以有效 的支持大文件。具体地，起初主级起止信息存储块RootExtentBlock里只有数据起止信息，即Level 0的多个起止信息Extent。结合图9，当这些Level 0Extent数目超过一定阈值(例如1024)，且主级起止信息存储块RootExtentBlock大小大于或等于容量阈值BlockSize时，就分配一个新的存储块作为从级起止信息存储块ExtentBlock EB-1，该从级起止信息存储块ExtentBlock EB-1存储从级起止信息列表，包括上述所有Level 0Extent，基于从级起止信息存储块ExtentBlock EB-1生成Level 1Extent，Level 1Extent存储于新分配的主级起止信息存储块RootExtentBlock EB-D中，其数据起始地址RangeStart是从级起止信息存储块ExtentBlock EB-1所包含的第一个数据起止信息Extent的数据起始地址RangeStart为0，其数据结束地址RangeEnd是从级起止信息存储块ExtentBlock EB-1所包含的最后一个起止信息Extent的数据结束地址RangEnd，从级起止信息存储块ExtentBlock EB-1对应的从级识别码ExtentID为EB-1。
优选地，所述设备1还包括：缓存装置，用于将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。当所述设备1执行写入数据时，更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块，例如图3中主级起止信息存储块EB-B在图4写入数据过程中，设置为可读不可写状态，待图4所示主级起止信息存储块EB-C更新完成且更新的描述文件Fileinfo持久化完成后，在清除主级起止信息存储块EB-B；同样，对于图3所示的数据存储块Block D1#0在图4所示的数据存储块Block D2#0，D2#1和D2#2存储完成且更新的描述文件Fileinfo持久化完成后清除，以实现用户在追加待写数据时，设备存储的数据文件仍可为用户提供获取数据的服务。
当然，本领域技术人员应能理解上述具体场景的描述仅为举例，其他现有的或今后可能出现的方式如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。
与现有技术相比，在本申请的实施例中所述写入数据的设备不会对数据文件进行覆盖写，而总是采用拷贝到新分配的存储块的方式进行更新并采用原子操作持久化更新的所述描述文件，从而在在追加写入数据的过程中，能保证数据文件不会被损坏，并维持数据的可读性和一致性，方便了用户的使用。进一步地，在本申请的实施例的写入数据的设备支持用户指定待写数据的写入位置的写入，也支持用户不指定待写数据的写入位 置的写入，提高用户写入数据的灵活性。
图12示出根据本申请一个方面的用于分布式存储系统的获取数据的设备示意图；其中，所述设备2包括：
描述文件获取装置11，用于获取描述文件；
主级起止信息列表获取装置12，用于基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储块中获取主级起止信息列表；
数据起止信息获取装置13，用于从所述主级起止信息列表中查找相应的数据起止信息；
数据获取装置14，用于基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
进一步地，当所述主级起止信息列表中还包括从级起止信息时，所述数据起止信息获取装置13还包括：基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。优选地，采用二分查找法查找相应的数据起止信息。
在此，所述设备2包括但不限于网络设备、或用户设备与网络设备通过网络相集成所构成的设备。其中，所述网络设备包括一种能够按照事先设定或存储的指令，自动进行数值计算和信息处理的电子设备，其硬件包括但不限于微处理器、专用集成电路(ASIC)、可编程门阵列(FPGA)、数字处理器(DSP)、嵌入式设备等。所述网络设备其包括但不限于计算机、网络主机、单个网络服务器、多个网络服务器集或多个服务器构成的云；在此，云由基于云计算(Cloud Computing)的大量计算机或网络服务器构成，其中，云计算是分布式计算的一种，由一群松散耦合的计算机集组成的一个虚拟超级计算机。所述网络包括但不限于互联网、广域网、城域网、局域网、VPN网络、无线自组织网络(Ad Hoc网络)等。优选地，所述设备2还可以是运行于所述网络设备、或用户设备与网络设备、或网络设备与触摸终端通过网络相集成所构成的设备上的脚本程序。当然，本领域技术人员应能理解上述设备2仅为举例，其他现有的或今后可能出现的网络设备3如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。此外，所述设备2可以与设备1共同设置在一个设备中。
上述各装置之间是持续不断工作的，在此，本领域技术人员应理解“持续”是指上述各装置分别实时地或者按照设定的或实时调整的工作模式要求，直至所述设备2停止工作。
与现有技术相比，在本申请的实施例中所述获取数据的设备在所述写入数据的基础 上，能够及时获取数据以供用户读取，并能够随时读取，不受写入数据的影响。方便了用户的使用。
需要注意的是，本申请可在软件和/或软件与硬件的组合体中被实施，例如，可采用专用集成电路(ASIC)、通用目的计算机或任何其他类似硬件设备来实现。在一个实施例中，本申请的软件程序可以通过处理器执行以实现上文所述步骤或功能。同样地，本申请的软件程序(包括相关的数据结构)可以被存储到计算机可读记录介质中，例如，RAM存储器，磁或光驱动器或软磁盘及类似设备。另外，本申请的一些步骤或功能可采用硬件来实现，例如，作为与处理器配合从而执行各个步骤或功能的电路。
另外，本申请的一部分可被应用为计算机程序产品，例如计算机程序指令，当其被计算机执行时，通过该计算机的操作，可以调用或提供根据本申请的方法和/或技术方案。而调用本申请的方法的程序指令，可能被存储在固定的或可移动的记录介质中，和/或通过广播或其他信号承载媒体中的数据流而被传输，和/或被存储在根据所述程序指令运行的计算机设备的工作存储器中。在此，根据本申请的一个实施例包括一个装置，该装置包括用于存储计算机程序指令的存储器和用于执行程序指令的处理器，其中，当该计算机程序指令被该处理器执行时，触发该装置运行基于前述根据本申请的多个实施例的方法和/或技术方案。
对于本领域技术人员而言，显然本申请不限于上述示范性实施例的细节，而且在不背离本申请的精神或基本特征的情况下，能够以其他的具体形式实现本申请。因此，无论从哪一点来看，均应将实施例看作是示范性的，而且是非限制性的，本申请的范围由所附权利要求而不是上述说明限定，因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本申请内。不应将权利要求中的任何附图标记视为限制所涉及的权利要求。此外，显然“包括”一词不排除其他单元或步骤，单数不排除复数。装置权利要求中陈述的多个单元或装置也可以由一个单元或装置通过软件或者硬件来实现。第一，第二等词语用来表示名称，而并不表示任何特定的顺序。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (24)

  1. 一种用于分布式存储系统的写入数据的方法，其中，所述方法包括：
    获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；
    生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
    基于所述主级起止信息存储块生成所述主级识别码；
    基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，并利用原子操作持久化更新的所述描述文件。
  2. 根据权利要求1所述的方法，其中，所述将所述待写数据存储至新分配的数据存储块包括：
    判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
  3. 根据权利要求2所述的方法，其中，所述方法还包括：
    在获取所述待写数据之前，获取所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，获取所述待写数据。
  4. 根据权利要求2所述的方法，其中，所述方法还包括：
    在获取所述待写数据前，获取所述数据文件的文件长度；
    所述基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，并利用原子操作持久化更新的所述描述文件包括：
    以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
  5. 根据权利要求4所述的方法，其中，所述基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，并利用原子操作持久化更新的所述描述文件还包括：
    所述请求写入位置与当前数据文件的总长度不一致时，则判断将所述待写数据存储至新分配的数据存储块中是否有所述惩罚读步骤：
    若有所述惩罚读步骤，则重新判断在获取所述待写数据前最后一次分配的数据存储 块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息；
    若无所述惩罚度步骤，则重新获取所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
  6. 根据权利要求1至5中任一项所述的方法，其中，所述方法还包括：
    判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则
    将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，
    将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
  7. 根据权利要求1至6中任一项所述的方法，其中，所述方法还包括：
    将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。
  8. 根据权利要求1至7中任一项所述的方法，其中，每一所述数据存储块包括一个或多个单元存储块，其中，所述多个单元存储块的容量均相同。
  9. 根据权利要求1至8中任一项所述的方法，其中，所述数据起止信息包括以下至少任一项：
    所述数据存储块识别码；
    所述数据存储块中数据的起止地址信息；
    所述数据存储块中单元存储的个数。
  10. 一种用于分布式存储系统的获取数据的方法，其中，所述方法包括：
    获取描述文件；
    基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储 块中获取主级起止信息列表；
    从所述主级起止信息列表中查找相应的数据起止信息；
    基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
  11. 根据权利要求10所述的方法，其中，当所述主级起止信息列表中还包括从级起止信息时，所述从所述主级起止信息列表中查找相应的数据起止信息还包括：
    基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。
  12. 根据权利要求10或11所述的方法，其中，所述从所述主级起止信息列表中查找相应的数据起止信息包括：
    采用二分查找法查找相应的数据起止信息。
  13. 一种用于分布式存储系统的写入数据的设备，其中，所述设备包括：
    写入装置，用于获取用户的待写数据，并将所述待写数据存储至新分配的数据存储块；
    起止信息更新装置，用于生成关于所述数据存储块的数据起止信息，并基于所述数据起止信息更新主级起止信息列表，并将更新的所述主级起止信息列表存储至新分配的主级起止信息存储块；
    主级识别码更新装置，用于基于所述主级起止信息存储块生成所述主级识别码；
    描述文件更新装置，用于基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述文件，并利用原子操作持久化更新的所述描述文件。
  14. 根据权利要求13所述的设备，其中，所述写入装置包括：
    惩罚读单元，用于判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并存储至新分配的所述数据存储块。
  15. 根据权利要求14所述的设备，其中，所述设备还包括：
    判断装置，在获取所述待写数据之前，获取所述待写数据的请求写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，获取所述待写数据。
  16. 根据权利要求13所述的设备，其中，所述写入装置还用于：
    在获取所述待写数据前，获取所述数据文件的文件长度；
    所述描述文件更新装置还用于：
    以在获取所述待写数据前所获取所述数据文件的文件长度为所述待写数据的请求 写入位置，判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
  17. 根据权利要求16所述的设备，其中，所述描述文件更新装置还用于：
    所述请求写入位置与当前数据文件的总长度不一致时，则判断在将所述待写数据存储至新分配的数据存储块中是否有所述惩罚读步骤：
    若有所述惩罚读步骤，则重新判断在获取所述待写数据前最后一次分配的数据存储块所存储的前次数据的大小是否小于预设容量，当小于时，则再次执行惩罚读步骤，包括将所述前次数据与所述待写数据合并，并重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息，当大于或等于时，将所述待写数据写入所述待写数据重新存储至所述数据存储块，并在所述主级起止信息列表中更新基于所述数据存储块生成起止信息；
    若无所述惩罚度步骤，则重新获取所述数据文件的总长度，并以所重新获取的所述数据文件的文件长度为所述待写数据的请求写入位置，再次判断所述请求写入位置与当前数据文件的总长度是否一致，当一致时，基于所述主级识别码和增加所述待写数据后数据文件的总长度更新描述数据文件，利用所述原子操作持久化所述更新的描述文件。
  18. 根据权利要求13至17中任一项所述的设备，其中，所述方法还包括：
    分级装置，用于判断所述主级起止信息列表的容量是否达到容量阈值或其数量起止信息的数量达到数量阈值，当达到时，则将当前所述主级起止信息列表中的所有数据起止信息转移至从级起止信息列表并存储至新分配的从级起止信息存储块，将基于所述从级起止信息存储块生成的从级起止信息存储至所述主级起止信息存储块。
  19. 根据权利要求13至18中任一项所述的设备，其中，所述方法还包括：
    缓存装置，用于将更新前的所述主级起止信息列表所对应的主级起止信息存储块设置为可读不可写状态，并在持久化所述描述文件成功后清除更新前的所述主级起止信息列表所对应的主级起止信息存储块。
  20. 根据权利要求13至19中任一项所述的设备，其中，每一所述数据存储块包括一个或多个单元存储块，其中，所述多个单元存储块的容量均相同。
  21. 根据权利要求13至20中任一项所述的设备，其中，所述数据起止信息包括以下至少任一项：
    所述数据存储块识别码；
    所述数据存储块中数据的起止地址信息；
    所述数据存储块中单元存储的个数。
  22. 一种用于分布式存储系统的获取数据的设备，其中，所述方法包括：
    描述文件获取装置，用于获取描述文件；
    主级起止信息列表获取装置，用于基于所述描述文件中的主级识别码，从所述主级识别码所对应的主级起止信息存储块中获取主级起止信息列表；
    数据起止信息获取装置，用于从所述主级起止信息列表中查找相应的数据起止信息；
    数据获取装置，用于基于所述数据起止信息，从所述数据起止信息对应的数据存储块中读取相应数据。
  23. 根据权利要求22所述的设备，其中，当所述主级起止信息列表中还包括从级起止信息时，所述数据起止信息获取装置还用于：
    基于所述从级起止信息，从所述从级起止信息对应的从级起止信息存储块中查找相应的数据起止信息。
  24. 根据权利要求22或23所述的设备，其中，所述数据起止信息获取装置用于：
    采用二分查找法查找相应的数据起止信息。
PCT/CN2016/097498 2015-09-09 2016-08-31 用于分布式存储系统的写入数据、获取数据的方法和设备 WO2017041654A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/916,192 US10599337B2 (en) 2015-09-09 2018-03-08 Method and device for writing data and acquiring data in a distributed storage system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510571183.7 2015-09-09
CN201510571183.7A CN106527958B (zh) 2015-09-09 2015-09-09 用于分布式存储系统的写入数据、获取数据的方法和设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/916,192 Continuation US10599337B2 (en) 2015-09-09 2018-03-08 Method and device for writing data and acquiring data in a distributed storage system

Publications (1)

Publication Number Publication Date
WO2017041654A1 true WO2017041654A1 (zh) 2017-03-16

Family

ID=58240575

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/097498 WO2017041654A1 (zh) 2015-09-09 2016-08-31 用于分布式存储系统的写入数据、获取数据的方法和设备

Country Status (3)

Country Link
US (1) US10599337B2 (zh)
CN (1) CN106527958B (zh)
WO (1) WO2017041654A1 (zh)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108664208B (zh) * 2017-03-27 2022-02-15 珠海极海半导体有限公司 基于flash存储器的文件追加写操作方法及装置
CN109947842B (zh) * 2017-07-27 2021-06-18 杭州海康威视数字技术股份有限公司 分布式存储系统中的数据存储方法、装置及系统
CN109032536B (zh) * 2018-08-31 2021-08-10 郑州云海信息技术有限公司 基于分布式集群系统的数据存储方法、装置、系统及设备
CN110597463A (zh) * 2019-08-16 2019-12-20 深圳中兴网信科技有限公司 数据读写方法、计算机设备与计算机可读存储介质
CN110505314B (zh) * 2019-09-26 2022-11-25 浪潮电子信息产业股份有限公司 一种并发追加上传请求的处理方法
CN113051336B (zh) * 2019-12-27 2023-05-26 上海莉莉丝科技股份有限公司 可视化的数据操作方法、系统、设备和介质
CN113377735A (zh) * 2020-02-25 2021-09-10 深信服科技股份有限公司 数据一致性的处理方法、装置、设备及分布式文件系统
CN112214175A (zh) * 2020-10-21 2021-01-12 重庆紫光华山智安科技有限公司 数据处理方法、装置、数据节点及存储介质
CN113568767B (zh) * 2021-06-30 2024-04-30 宁波三星医疗电气股份有限公司 适用于智能用电终端的数据写文件校验方法
CN113703673B (zh) * 2021-07-30 2023-09-22 郑州云海信息技术有限公司 一种单机数据存储方法及相关装置
CN114115735B (zh) * 2021-11-18 2024-02-09 北京奥星贝斯科技有限公司 写入数据的方法及装置
CN114880958B (zh) * 2022-07-12 2022-10-11 南京气象科技创新研究院 一种基于多气象因子智能深度学习的能见度预报方法
CN116700632B (zh) * 2023-08-07 2023-10-24 湖南中盈梦想商业保理有限公司 一种高可靠的金融信息数据存储方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7107419B1 (en) * 2003-02-14 2006-09-12 Google Inc. Systems and methods for performing record append operations
CN102307221A (zh) * 2011-03-25 2012-01-04 国云科技股份有限公司 一种云存储系统及其实现方法
CN102622350A (zh) * 2011-01-26 2012-08-01 腾讯科技（深圳）有限公司 一种写数据的方法、装置及系统
CN104731921A (zh) * 2015-03-26 2015-06-24 江苏物联网研究发展中心 Hadoop分布式文件系统针对日志型小文件的存储和处理方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8069324B2 (en) * 2008-12-19 2011-11-29 Seagate Technology Llc Storage device with manual learning
CN102156700A (zh) * 2010-02-12 2011-08-17 华为技术有限公司 数据库的访问方法、装置及系统
CN102541869B (zh) * 2010-12-07 2016-08-10 腾讯科技（深圳）有限公司 写入文件的方法和装置
CN103136215A (zh) * 2011-11-24 2013-06-05 腾讯科技（深圳）有限公司 存储系统的数据读写方法和装置
US9811455B2 (en) * 2013-03-15 2017-11-07 The Boeing Company Accessing different types of memory by respective distinct command with different timing requirements
JP6044402B2 (ja) * 2013-03-18 2016-12-14 富士通株式会社 情報処理システム、管理装置、管理装置の制御プログラム、および情報処理システムの制御方法
US9053035B1 (en) * 2013-11-25 2015-06-09 Freescale Semiconductor, Inc. Multi-threaded system for performing atomic binary translations
CN103970844B (zh) * 2014-04-28 2017-11-21 北京创世漫道科技有限公司 大数据的写入方法和装置、读取方法和装置及处理系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7107419B1 (en) * 2003-02-14 2006-09-12 Google Inc. Systems and methods for performing record append operations
CN102622350A (zh) * 2011-01-26 2012-08-01 腾讯科技（深圳）有限公司 一种写数据的方法、装置及系统
CN102307221A (zh) * 2011-03-25 2012-01-04 国云科技股份有限公司 一种云存储系统及其实现方法
CN104731921A (zh) * 2015-03-26 2015-06-24 江苏物联网研究发展中心 Hadoop分布式文件系统针对日志型小文件的存储和处理方法

Also Published As

Publication number Publication date
CN106527958A (zh) 2017-03-22
US10599337B2 (en) 2020-03-24
CN106527958B (zh) 2020-03-10
US20180203606A1 (en) 2018-07-19

Similar Documents

Publication Publication Date Title
WO2017041654A1 (zh) 用于分布式存储系统的写入数据、获取数据的方法和设备
US10762038B2 (en) System and method for virtual machine conversion
TWI492077B (zh) 檔案系統的檢查點
US10936441B2 (en) Write-ahead style logging in a persistent memory device
US9280578B1 (en) Combining transactions in a metadata transaction log
US9286320B2 (en) System and method for maintaining consistency among metadata elements of filesystem's logical objects
CN110018998B (zh) 一种文件管理方法、系统及电子设备和存储介质
RU2560786C2 (ru) Корректность без зависимости от упорядоченности
EP2718825A2 (en) Storage architecture for backup application
US10929176B2 (en) Method of efficiently migrating data from one tier to another with suspend and resume capability
US20190087130A1 (en) Key-value storage device supporting snapshot function and operating method thereof
US10430292B2 (en) Snapshot deletion in a distributed storage system
US10977143B2 (en) Mirrored write ahead logs for data storage system
US10776321B1 (en) Scalable de-duplication (dedupe) file system
US10459807B2 (en) Determining modified portions of a RAID storage array
US10430110B2 (en) Implementing a hybrid storage node in a distributed storage system
US11740980B2 (en) Managing snapshot metadata following backup
US11256434B2 (en) Data de-duplication
US10452496B2 (en) System and method for managing storage transaction requests
US10649807B1 (en) Method to check file data integrity and report inconsistencies with bulk data movement
US11023328B2 (en) Redo log for append only storage scheme
US10976959B2 (en) Method and system for accessing virtual machine state while virtual machine restoration is underway
US10452308B2 (en) Encoding tags for metadata entries in a storage system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16843592

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16843592

Country of ref document: EP

Kind code of ref document: A1