WO2017177752A1 - Procédé et dispositif de stockage de fichier - Google Patents

Procédé et dispositif de stockage de fichier Download PDF

Info

Publication number
WO2017177752A1
WO2017177752A1 PCT/CN2017/073080 CN2017073080W WO2017177752A1 WO 2017177752 A1 WO2017177752 A1 WO 2017177752A1 CN 2017073080 W CN2017073080 W CN 2017073080W WO 2017177752 A1 WO2017177752 A1 WO 2017177752A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
stored
keyword
metadata
storage
Prior art date
Application number
PCT/CN2017/073080
Other languages
English (en)
Chinese (zh)
Inventor
丁智勇
汪渭春
陈伟
曹力
Original Assignee
杭州海康威视数字技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视数字技术股份有限公司 filed Critical 杭州海康威视数字技术股份有限公司
Publication of WO2017177752A1 publication Critical patent/WO2017177752A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/164—File meta data generation

Definitions

  • the present application relates to the field of data processing, and in particular, to a file storage method and apparatus.
  • the metadata of the file is generally stored in a metabase of the storage terminal, and the file itself is stored in another location of the storage terminal.
  • the storage terminal determines the storage location of the file according to the metadata stored in the metadata database, thereby obtaining the file.
  • the metadata stored in the metabase is lost, even if the file data corresponding to the lost metadata is read, it cannot be determined that the file data read belongs to those files, thereby causing file loss.
  • the embodiment of the present application discloses a file storage method and device, which ensures metadata recovery and avoids file loss.
  • the embodiment of the present application discloses a file storage method, which is applied to a storage terminal, and the method includes:
  • the first file identifier, the data volume, and the to-be-stored file are stored in the value of the target key according to the preset storage format, including:
  • the first keyword is a keyword that is identified by the file
  • the file identifier length is a number of bytes occupied by the first file identifier.
  • the second keyword is a keyword for the file.
  • the method further includes:
  • the preset metabase is updated according to the first keyword and the second keyword stored in the values of all the keys.
  • the updating the preset metadata database according to the first keyword and the second keyword stored in values of all the keys includes:
  • the obtaining, according to the second file identifier, the target data segment that includes the second file identifier from the values of all the keys including:
  • the data segment is a target data segment, wherein the first keyword W2 is: the second of the values of the key K a first keyword that is identified by the file and that is closest to the second file identifier;
  • the obtaining the metadata to be stored when the file to be stored is stored according to the preset storage format according to the target key includes:
  • the metadata to be stored of the file to be stored is determined according to the calculated offset.
  • the embodiment of the present application further discloses a file storage device, which is applied to a storage terminal, and the device includes: a storage request receiving module, an information determining module, a metadata obtaining module, and a data storage module;
  • the storage request receiving module is configured to receive a storage request for a file to be stored
  • the information determining module is configured to obtain a first file identifier of the file to be stored and a data amount of the file to be stored, and determine a target key for storing the file to be stored;
  • the metadata obtaining module is configured to obtain, according to the target key, metadata to be stored when the file to be stored is stored according to a preset storage format
  • the data storage module is configured to store the first file identifier, the data amount, and the to-be-stored file into a value of the target key according to the preset storage format, and store the
  • the storage metadata is stored in a preset metabase.
  • the data storage module includes:
  • a file storage submodule configured to store the first file identifier, the data amount, and the to-be-stored file into a value of the target key according to the preset storage format
  • a metadata storage submodule configured to store the to-be-stored metadata into a preset metadata database
  • the file storage submodule includes: a file identifier storage unit and a file storage unit;
  • the file identifier storage unit is configured to: the first keyword, the file identifier length, and the first file identifier according to a first keyword, a file identifier length, and an order of the first file identifier And storing, in the value of the target key, where the first keyword is a keyword for a file identifier, where the file identifier length is a number of bytes occupied by the first file identifier;
  • the file storage unit configured to store the second keyword, the data amount, and the file to be stored to the target according to a second keyword, the data amount, and an order of the file to be stored
  • the device further includes: a metabase update module, configured to:
  • the preset metabase is updated according to the first keyword and the second keyword stored in the values of all the keys.
  • the metabase update module includes: a file identifier obtaining submodule, a target data segment determining submodule, and a metabase updating submodule;
  • the file identifier obtaining submodule is configured to obtain a second file identifier of the file with the metadata lost in the case that the metadata of the preset metabase is lost;
  • the target data segment determining submodule is configured to obtain, according to the second file identifier, a target data segment that includes the second file identifier from values of all keys;
  • the meta-database update sub-module is configured to update the preset meta-database according to the second file identifier and the data amount stored in the target data segment.
  • the target data segment determining submodule includes: a key obtaining unit, a keyword determining unit, a first target data segment determining unit, and a second target data segment determining unit;
  • the key obtaining unit configured to obtain, according to the second file identifier, a key K that includes the second file identifier
  • the keyword determining unit is configured to determine whether the first keyword is stored after the second file identifier in the value of the key K, and if yes, triggering the first target data segment determining unit, if No, the second target data segment determining unit is triggered;
  • the first target data segment determining unit is configured to obtain a first keyword W1 that is closest to the second file identifier after the second file identifier in the value of the key K, and determine the first keyword W2 and Place
  • the data segment between the first keywords W1 is a target data segment, wherein the first keyword W2 is: the value of the key K is before the second file identifier and is closest to the second file identifier.
  • the second target data segment determining unit is configured to determine a data segment between the first keyword W2 and a storage location at the end of the value of the key K as a target data segment.
  • the metadata obtaining module includes: a keyword obtaining submodule, an offset calculating submodule, and a metadata determining submodule;
  • the keyword obtaining submodule configured to obtain a first keyword and a second keyword stored in a value of the target key
  • the offset calculation submodule is configured to calculate, according to the obtained first keyword and the obtained second keyword, an offset of the file to be stored in a value of the target key;
  • the metadata determining submodule is configured to determine metadata to be stored of the file to be stored according to the calculated offset.
  • an embodiment of the present application further discloses a storage terminal, where the storage terminal includes: a casing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed in the casing Inside the space, the processor and the memory are disposed on the circuit board; the power circuit is configured to supply power to each circuit or device of the storage terminal; and the memory is used to store executable program code
  • the processor executes the following steps by running executable program code stored in the memory:
  • an embodiment of the present application further discloses an executable program code, where the executable program code is configured to perform the following steps at runtime:
  • the embodiment of the present application further discloses a storage medium for storing executable program code, and the executable program code is executed to perform the following steps:
  • the storage terminal after receiving the storage request for the file to be stored, the storage terminal obtains the first file identifier of the file to be stored and the data amount of the file to be stored, and determines to store the file to be stored.
  • the target key is obtained according to the target key, and the metadata to be stored when the file to be stored is stored according to the preset storage format is obtained, and the first file identifier, the data volume, and the file to be stored are stored in the target key according to the preset storage format. And storing the to-be-stored metadata in a preset metabase.
  • the first file identifier, the data amount, and the stored text of the stored file in the value of the key may be determined according to a preset storage format.
  • the offset of the value in the key, combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata and avoiding Loss of documents.
  • FIG. 1 is a schematic flowchart diagram of a file storage method according to an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of a TLV storage format
  • FIG. 3 is a schematic flowchart diagram of another file storage method according to an embodiment of the present disclosure.
  • FIG. 4 is a schematic flowchart diagram of another file storage method according to an embodiment of the present disclosure.
  • FIG. 5 is a schematic flowchart diagram of another file storage method according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram of a file storage arrangement according to an embodiment of the present application.
  • FIG. 7 is a schematic flowchart diagram of another file storage method according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram of a file storage device according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure.
  • FIG. 10 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure.
  • FIG. 12 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure.
  • FIG. 1 is a schematic flowchart of a file storage method according to an embodiment of the present disclosure, which is applied to a storage terminal, and the method may include the following steps:
  • S101 Receive a storage request for a file to be stored
  • the storage request includes a file to be stored, and may further include an identifier of the file to be stored, so that the storage terminal can quickly obtain the identifier of the file to be stored, and then store the metadata of the file to be stored; or, the storage request may also be
  • the storage terminal does not include the identifier of the file to be stored, and the storage terminal can parse the identifier of the file to be stored in the file to be stored, so that the storage request is simpler and clearer.
  • S102 Obtain a first file identifier of the file to be stored and a data amount of the file to be stored, and determine a target key for storing the file to be stored;
  • the first file identifier and the data amount of the file to be stored are part of the metadata of the file to be stored, and may be directly obtained when the storage request is obtained, and therefore, the storage is to be stored. Before the file, the first file identifier and the amount of data of the file to be stored are obtained first.
  • the massive data storage with high reliability and scalability is a huge challenge for the current storage terminal, and the traditional database is often difficult to meet the requirement. In this case, the relationship is used. Databases will make storing and reading data inefficient, and scaling databases will be a big problem. In such cases, using key storage would be a good choice. Key storage has the following advantages:
  • the key consists of the name Key and the value Value, and a Key corresponds to a Value.
  • the metadata of the file to be stored generally includes metadata information such as an identifier of the file to be stored in the target key, an amount of data, and an offset from a start storage location of the target key.
  • S103 can be implemented by the following process:
  • the storage location of the file to be stored in the target key may be determined according to the storage format. At this time, it may be determined that the file to be stored is stored relative to the target key.
  • the offset of the location such that the metadata of the file to be stored, that is, the metadata to be stored, can be obtained by combining the obtained first file identifier, the amount of data, the offset, and the name of the target key.
  • the preset storage format may be a format of TLV (TAG_LENGTH_VALUE, mark_length_value), as shown in FIG. 2, and FIG. 2 is a schematic structural diagram of a TLV storage format. For the sake of simplicity.
  • S104 Store the first file identifier of the file to be stored, the data volume of the file to be stored, and the file to be stored in the value of the target key according to the preset storage format, and store the metadata to be stored in the preset metabase. in.
  • the preset metabase is used to store metadata of the file.
  • the storage terminal may store a plurality of files having a small capacity relative to the value of the key, such as: the value of the key has a capacity of 1 Mbytes, and the size of the file to be stored is 100 Kbytes.
  • the value of a key is Storing only one file in it will easily cause a waste of storage space, so you can consider storing multiple complete files in the value of one key.
  • the storage device obtains the first file identifier of the file to be stored, the data amount of the file to be stored, and the file to be stored, and stores the value in the target key according to a preset storage format, so that If the meta-database has metadata loss, the data belonging to the same file may be determined in the value of the key according to the storage format, and according to the storage format, the obtained data is obtained.
  • the metadata information such as the file identifier and the data amount of the file is stored, and the metadata stored in the metadata database is restored.
  • metadata information such as the offset amount in the value of the target key when the data to be stored is stored according to the preset storage format can be obtained, and integrated.
  • the data identifier of the file to be stored may be determined by the first file identifier and the data volume of the file to be stored, and the metadata to be stored is stored in a preset metadata database. In this way, when data access is performed, the storage terminal can perform data access according to the metadata in the metadata database.
  • the storage terminal after successfully storing the file to be stored, stores the metadata of the file to be stored, thereby ensuring the accuracy of the metadata in the metadata database, and avoiding unsuccessful storage to be stored.
  • the file stores the metadata of the file to be stored, and the storage terminal cannot correctly read the metadata stored in the metadata database, and the storage terminal library crashes.
  • step of storing the file to be stored may be performed simultaneously with the step of storing the metadata to be stored, or may be performed after storing the metadata to be stored before S102, or may be performed after storing the metadata to be stored, this application This is not limited.
  • the storage terminal after receiving the storage request for the file to be stored, the storage terminal obtains the first file identifier of the file to be stored and the data amount of the file to be stored, and determines a target key for storing the file to be stored. According to the target key, the metadata to be stored when the file to be stored is stored according to the preset storage format is obtained, and the first file identifier, the data volume, and the file to be stored are stored in the target key according to the preset storage format, and The metadata to be stored is stored in a preset metabase.
  • the first file identifier, the data amount, and the amount of the stored file in the value of the key can be determined according to a preset storage format.
  • the offset of the stored file in the value of the key, combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata. Avoid the loss of files.
  • FIG. 3 is a schematic flowchart of another file storage method according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • step S104 may include the following steps:
  • the first keyword is a keyword for the file identifier
  • the file identifier length is the number of bytes occupied by the first file identifier
  • the TAG may be used to store the first keyword
  • the first keyword may be TAG_FILE_NAME, and other characters may be used to represent the first keyword, which is not limited in this embodiment of the present application.
  • S1042 Store the second keyword, the data volume, and the file to be stored in the value of the target key according to the second keyword, the data volume of the file to be stored, and the order of the file to be stored;
  • the second keyword is a keyword for the file.
  • the TAG may also be used to store the second keyword
  • the second keyword may be TAG_FILE_DATA
  • the other keywords may be used to represent the second keyword, which is not limited in this embodiment of the present application.
  • S1043 Store the metadata to be stored into a preset metabase.
  • the storage terminal stores the first keyword, the file identifier length, and the first file identifier of the file to be stored according to the first keyword, the file identifier length, and the first file identifier of the file to be stored.
  • the metadata is stored in a preset metabase.
  • FIG. 4 is a schematic flowchart of another file storage method according to an embodiment of the present disclosure, which is applied to a storage terminal, and the method may include the following steps:
  • each time the metadata database is updated a first check value is obtained, and a second check value may be obtained again before the next time the metadata is stored, when the first check is performed.
  • the check value can be obtained according to a CRC (Cyclic Redundancy Check) check algorithm or other check algorithm.
  • the identifier and data of the stored file in the value of the key may be obtained according to the first keyword and the second keyword stored in the values of all the keys.
  • the amount of metadata information, and then the metadata database is updated according to the obtained metadata information, which ensures the recovery of the metadata in the metadata database and avoids file loss.
  • step S105 may occur before any of steps S101, S102, S103, and S104, and may also occur after any of steps S101, S102, S103, and S104, and may also be followed by step S101. Any of the steps S102, S103, and S104 occur at the same time, which is not limited by the examples in the present application.
  • the storage terminal obtains the identifier of the stored file and the stored file according to the preset storage format when the metadata is lost from the preset key database.
  • the amount of data, according to the first keyword and the second keyword stored in the values of all the keys, obtaining the offset of the stored file in the value of the key, and updating the preset metabase according to the obtained information, can effectively Restore metadata in the metabase to avoid file loss.
  • FIG. 5 is a schematic flowchart of another file storage method according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • step S105 may include the following steps:
  • S1051 Obtain a second file identifier of the file in which the metadata is lost in the case that the metadata of the preset metabase is lost.
  • the metadata lost in the metabase may be only a small part. In this case, if all the metadata stored in the metabase is updated, the workload of the storage terminal will be increased, and the metadata is The recovery speed is slow. If only the missing metadata in the metabase is updated, the workload of the storage terminal will be greatly reduced, and the metadata recovery speed will be fast.
  • the metadata of the file in the metadata database may be considered to be lost, thereby obtaining the second file identifier of the file.
  • the second file identification of the file can be obtained from a read request when the file is read.
  • S1052 Obtain a target data segment that includes the second file identifier from the values of all the keys according to the second file identifier of the lost file;
  • the obtaining, according to the second file identifier of the lost file, the target data segment that includes the second identifier from the file stored by the value of all the keys may include:
  • the first keyword is read in the value of all the keys, and then the corresponding file identifier length is determined according to the first keyword, and then the file identifier is determined according to the file identifier length, and the second file is obtained from the determined file identifier.
  • the same file identifier and the key corresponding to the file identifier are identified, and the key obtained here is the key containing the second file identifier.
  • the above-mentioned key K may be a plurality of keys, or may be only one key.
  • the above-mentioned key K is a plurality of keys, it is explained that one file is stored in a plurality of keys.
  • step S2 determining whether the first key is stored in the value of the key K after the second file identification, if yes, proceeding to step S3, if not, executing step S4;
  • the identifier of the file, the amount of data, and the order of the file itself may be: first store the identifier of the file, then store the data amount of the file and the file itself, and the first keyword is a data segment storing a file. Starting storage location.
  • FIG. 6 is a schematic diagram of a file storage arrangement according to an embodiment of the present application. As can be seen from FIG. 6, two files are stored in the value of the key named Key1, and file2 is stored after file1, and no file is stored after file2.
  • the first keyword A (TAG_FILE_NAME) corresponding to file1 and
  • the space between the first key B (TAG_FILE_NAME) corresponding to file2 is the data segment storing file1
  • the first key corresponding to file2 The space between the word B and the end of the value of the key named Key1 is the data segment in which file2 is stored.
  • the first keyword W2 is: a first keyword in the value of the key K before the second file identifier and closest to the second file identifier;
  • the first keyword is a data segment starting storage location for storing a file
  • a file identifier corresponds to a first keyword
  • a space between two adjacent first keywords is a data segment
  • the file The identifier corresponding to the first keyword is: the first keyword before the file identifier and closest to the file identifier.
  • the file identifier length may be obtained according to the first keyword stored in the target data segment, and then the file identifier is determined according to the file identifier length; and the second identifier may be stored according to the target data segment.
  • the keyword gets the amount of data for the file and gets the file.
  • the file identifier and the data amount of the file are metadata, so that the metadata in the metadata database can be restored according to the metadata obtained in the value of the key.
  • the first keyword W2 corresponding to the file identifier and the last storage location of the value of the key K where the file identifier is located may be determined.
  • the data segment between them is the target data segment.
  • S1053 Update the preset metabase according to the second file identifier of the lost file stored in the target data segment and the data amount of the lost file.
  • the storage terminal only needs to obtain the second file identifier of the file whose metadata is lost, and obtain the second identifier from the file stored in the value of all the keys according to the second file identifier of the lost file.
  • the target data segment is finally updated according to the second file identifier of the lost file stored in the target data segment and the data amount of the lost file, and the default metadata database is updated, which is reduced.
  • the workload of the storage terminal improves the efficiency of the metadata database update.
  • FIG. 7 is a schematic flowchart of another file storage method according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • step S103 may include the following steps:
  • S1031 Obtain a first keyword and a second keyword stored in a value of the target key
  • the number of bytes occupied by the TAG and LENGTH in the file stored in the TLV format is a fixed size, so only the first keyword and the second keyword stored in the value of the target key are required (also That is, TAG), it is determined according to the number of the first keyword and the second keyword and the file identifier length corresponding to the first keyword and the data amount corresponding to the second keyword (ie, LENGTH), to determine the to-be-stored The offset of the file in the value of the target key.
  • S1032 Calculate, according to the obtained first keyword and the obtained second keyword, an offset of the file to be stored in the value of the target key;
  • the number of bytes occupied by TAG and LENGTH is 4Kbytes
  • the identifier length of file1 is 10Kbytes
  • the data amount of file1 is 100Kbytes
  • the identification length of file2 is 12Kbytes
  • the data volume of file2 is 200Kbytes
  • file1 and The storage order of file2 is as follows: first store file1 and then store file2.
  • the metadata of file1 and file2 obtained at this time are shown in Table 1.
  • the offset of file1 and file2 in the value of the key is calculated by storing the file1 in the value of the target key named Key1, the value of the target key is empty, and at this time, it can be determined
  • the starting storage location of the value of the target key has a first keyword and a second keyword between the file headers of the file1 file (this is the first keyword and the default storage format carried by the TLV itself)
  • the obtained second keyword that is, there are 2 TAGs, corresponding to 2 LENGTH, from LENGTH, the file length of file1 can be 10Kbytes, and the data amount of file1 is 100Kbytes. Therefore, the value of file1 at Key1 can be determined.
  • the first keyword carried and the obtained second keyword that is, there are 4 TAGs, corresponding to 4 LENGTH, and the file length of file1 is 10Kbytes, the data amount of file1 is 100Kbytes, and file2 is obtained from LENGTH.
  • the identifier length is 12Kbytes, and the data volume of file2 is 200Kbytes.
  • S1033 Determine, according to the calculated offset, the metadata to be stored of the file to be stored;
  • the offset of the file in the value of the key is also part of the metadata of the file, so that the storage terminal quickly obtains the storage location of the file according to the offset.
  • the metadata of the file to be stored is determined in combination with the first file identifier and the data amount of the file to be stored.
  • the storage terminal obtains the first keyword and the second keyword stored in the value of the target key, and calculates the file to be stored in the target key according to the obtained first keyword and the obtained second keyword.
  • the offset in the value and based on the calculated offset, determine the metadata to be stored of the file to be stored.
  • the offset of the file is calculated according to the first keyword and the second keyword stored in the value of the target key, thereby determining the metadata to be stored of the file to be stored, without having to count the storage location and target of the file to be stored one by one.
  • the amount of data between the starting storage locations of the values of the keys to determine the offset of the file, the determination of the metadata is faster and more convenient.
  • FIG. 8 is a schematic structural diagram of a file storage device according to an embodiment of the present disclosure, which is applied to a storage terminal, and the device may include: a storage request receiving module 801, an information determining module 802, a metadata obtaining module 803, and data.
  • the storage request receiving module 801 is configured to receive a storage request for a file to be stored
  • the information determining module 802 is configured to obtain a first file identifier of the file to be stored and a file to be stored. The amount of data, and determine the target key for storing the file to be stored;
  • the metadata obtaining module 803 is configured to obtain, according to the target key, metadata to be stored when the file to be stored is stored according to a preset storage format
  • the data storage module 804 is configured to store the first file identifier, the data amount, and the to-be-stored file into a value of the target key according to the preset storage format, and store the to-be-stored
  • the metadata is stored in a preset metabase, wherein a preset metabase is used to store metadata of the file.
  • the storage terminal after receiving the storage request for the file to be stored, the storage terminal obtains the first file identifier of the file to be stored and the data amount of the file to be stored, and determines a target key for storing the file to be stored. According to the target key, the metadata to be stored when the file to be stored is stored according to the preset storage format is obtained, and the first file identifier, the data volume, and the file to be stored are stored in the target key according to the preset storage format, and The metadata to be stored is stored in a preset metabase.
  • the first file identifier, the data amount, and the amount of the stored file in the value of the key can be determined according to a preset storage format.
  • the offset of the stored file in the value of the key, combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata. Avoid the loss of documents.
  • FIG. 9 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • the data storage module 804 may include: a file identifier storage submodule 8041 and a metadata storage device. Module 8042;
  • a file storage sub-module 8041 configured to store the first file identifier, the data amount, and the to-be-stored file into a value of the target key according to the preset storage format
  • a metadata storage sub-module 8042 configured to store the to-be-stored metadata into a preset metadata database
  • the file storage submodule 8041 includes: a file identifier storage unit 8041a and a file storage unit 8041b;
  • the file identifier storage unit 8041a is configured to store the first keyword, the file identifier length, and the first file identifier to the target key according to the first keyword, the file identifier length, and the first file identifier.
  • the value of the first keyword is a keyword for the file identifier, and the file identifier length is the number of bytes occupied by the first file identifier;
  • the file storage unit 8041b is configured to store the second keyword, the data amount, and the file to be stored in the value of the target key according to the second keyword, the data amount, and the order of the file to be stored, where the second keyword is for The keyword of the file.
  • the storage terminal stores the first keyword, the file identifier length, and the first file identifier of the file to be stored according to the first keyword, the file identifier length, and the first file identifier of the file to be stored.
  • the storage terminal stores the first keyword, the file identifier length, and the first file identifier of the file to be stored according to the first keyword, the file identifier length, and the first file identifier of the file to be stored.
  • the metadata is stored in a preset metabase.
  • FIG. 10 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure, which is applied to a storage terminal, and the device may further include: a metabase update module 805;
  • metabase update module 805 is configured to:
  • the preset metabase is updated according to the first keyword and the second keyword stored in the values of all the keys.
  • the storage terminal obtains the identifier of the stored file and the stored file according to the preset storage format when the data is read from each key in the case that the metadata is lost in the preset metabase.
  • the amount of data, according to the first keyword and the second keyword stored in the values of all the keys, determine the offset of each stored file in the value of each key, and update the preset metabase according to the obtained information, which can be effective Restore metadata in the metabase to avoid file loss.
  • FIG. 11 is a schematic structural diagram of another file storage device according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • the meta-database update module 805 may include: a file identifier obtaining sub-module 8051 and a target data segment. Determining submodule 8052 and metabase update submodule 8053;
  • the file identifier obtaining sub-module 8051 is configured to obtain a second file identifier of the file in which the metadata is lost if the metadata of the preset meta-database is lost.
  • a target data segment determining sub-module 8052 configured to obtain, according to the second file identifier, a target data segment that includes the second file identifier from values of all the keys;
  • the metabase update submodule 8053 is configured to update the preset metabase according to the second file identifier and the amount of data stored in the target data segment.
  • the target data segment determining submodule 8052 may include: a key obtaining unit, a keyword determining unit, a first target data segment determining unit, and a second target data segment determining unit (not shown in FIG. 11 show);
  • the key obtaining unit is configured to obtain, according to the second file identifier, a key K that includes the second file identifier;
  • a keyword determining unit configured to determine whether a first keyword is stored after the second file identifier in the value of the key K, and if yes, triggering the first target data segment determining unit, and if not, triggering the second target data Segment determination unit;
  • a first target data segment determining unit configured to obtain a first keyword W1 that is closest to the second file identifier after the second file identifier in the value of the key K, and determine data between the first keyword W2 and the first keyword W1
  • the segment is a target data segment, where the first keyword W2 is: a first keyword in the value of the key K before the second file identifier and closest to the second file identifier;
  • the second target data segment determining unit is configured to determine the data segment between the first keyword W2 and the storage location at the end of the value of the key K as the target data segment.
  • the storage terminal only needs to obtain the second file identifier of the file whose metadata is lost, and obtain the second identifier from the file stored in the value of all the keys according to the second file identifier of the lost file.
  • the target data segment finally updates the preset metabase according to the second file identifier of the lost file stored in the target data segment and the data amount of the lost file, thereby reducing the workload of the storage terminal and improving the metabase. Updated efficiency.
  • FIG. 12 is a schematic structural diagram of another file storage apparatus according to an embodiment of the present disclosure, which is applied to a storage terminal.
  • the metadata obtaining module 803 may include: a keyword obtaining submodule 8031, and an offset. a quantity calculation sub-module 8032, a metadata determination sub-module 8033;
  • the keyword obtaining submodule 8031 is configured to obtain the first stored in the value of the target key. Keyword and second keyword;
  • the offset calculation sub-module 8032 is configured to calculate, according to the obtained first keyword and the obtained second keyword, an offset of the file to be stored in the value of the target key;
  • the metadata determining sub-module 8033 is configured to determine metadata to be stored of the file to be stored according to the calculated offset.
  • the storage terminal obtains the first keyword and the second keyword stored in the value of the target key, and calculates the file to be stored in the target key according to the obtained first keyword and the obtained second keyword.
  • the offset in the value and based on the calculated offset, determine the metadata to be stored of the file to be stored.
  • the offset of the file is calculated according to the first keyword and the second keyword stored in the value of the target key, thereby determining the metadata to be stored of the file to be stored, without having to count the storage location and target of the file to be stored one by one.
  • the amount of data between the starting storage locations of the values of the keys to determine the offset of the file, the determination of the metadata is faster and more convenient.
  • the embodiment of the present application provides a storage terminal, including: a casing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside a space enclosed by the casing, The processor and the memory are disposed on the circuit board; the power supply circuit is configured to supply power to each circuit or device of the storage terminal; the memory is configured to store executable program code; Executing executable program code stored in the memory to perform the following steps:
  • the storage terminal After receiving the storage request for the file to be stored, the storage terminal Obtaining a first file identifier of the file to be stored and a data volume of the file to be stored, and determining a target key for storing the file to be stored, and obtaining metadata to be stored when the file to be stored is stored according to a preset storage format according to the target key And storing the first file identifier, the data volume, and the file to be stored in the target key according to the preset storage format, and storing the to-be-stored metadata in a preset metabase.
  • the metadata loss of the metadata database the first file identifier, the data amount of the stored file in the value of the key, and the stored file in the value of the key may be determined according to a preset storage format.
  • the offset combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata and avoiding file loss.
  • the storage terminal exists in various forms including, but not limited to:
  • Mobile communication devices These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication.
  • Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.
  • Ultra-mobile personal computer equipment This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access.
  • Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.
  • Portable entertainment devices These devices can display and play multimedia content. Such devices include: audio, video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.
  • the server consists of a processor, a hard disk, a memory, a system bus, etc.
  • the server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.
  • Applying the storage request for the file to be stored obtaining the first file identifier of the file to be stored and the data amount of the file to be stored, and determining a target key for storing the file to be stored, according to the target key.
  • obtaining the to-be-stored metadata when the file to be stored is stored according to the preset storage format, storing the first file identifier, the data volume, and the file to be stored in the target key according to the preset storage format, and storing the to-be-stored element
  • the data is stored in a preset metabase.
  • the metadata loss of the metadata database the first file identifier, the data amount of the stored file in the value of the key, and the stored file in the value of the key may be determined according to a preset storage format.
  • the offset combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata and avoiding file loss.
  • Embodiments of the present application provide a storage medium for storing executable program code, the executable program code being executed to perform the following steps:
  • the application After the storage request for the file to be stored is received, the application is obtained. And storing the first file identifier of the file and the data volume of the file to be stored, and determining a target key for storing the file to be stored, and obtaining the metadata to be stored when the file to be stored is stored according to the preset storage format according to the target key, according to the target key
  • the preset storage format stores the first file identifier, the data volume, and the file to be stored in the target key, and stores the to-be-stored metadata in a preset metabase.
  • the metadata loss of the metadata database the first file identifier, the data amount of the stored file in the value of the key, and the stored file in the value of the key may be determined according to a preset storage format.
  • the offset combined with the name of the key, determines the metadata of the stored file, and then restores the metadata in the metadata database according to the metadata, thereby ensuring recovery of the metadata and avoiding file loss.
  • the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un procédé et un dispositif de stockage de fichier, le procédé consistant : à recevoir une requête de stockage pour un fichier à stocker (S101) ; à obtenir un premier identificateur de fichier du fichier à stocker et une quantité de données du fichier à stocker, et à déterminer une clé cible utilisée pour stocker le fichier à stocker (S102) ; selon la clé cible, à obtenir des métadonnées à stocker lorsque le fichier à stocker est stocké selon un format de stockage prédéfini (S103) ; selon le format de stockage prédéfini, à stocker le premier identificateur de fichier, la quantité de données et le fichier à stocker dans une valeur de la clé cible, et à stocker les métadonnées à stocker dans une bibliothèque de métadonnées préétablie (S104). Le procédé et le dispositif garantissent de manière efficace la récupération de métadonnées, permettant ainsi d'empêcher une perte de fichier.
PCT/CN2017/073080 2016-04-14 2017-02-08 Procédé et dispositif de stockage de fichier WO2017177752A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610230618.6 2016-04-14
CN201610230618.6A CN107301177B (zh) 2016-04-14 2016-04-14 一种文件存储方法及装置

Publications (1)

Publication Number Publication Date
WO2017177752A1 true WO2017177752A1 (fr) 2017-10-19

Family

ID=60041372

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/073080 WO2017177752A1 (fr) 2016-04-14 2017-02-08 Procédé et dispositif de stockage de fichier

Country Status (2)

Country Link
CN (1) CN107301177B (fr)
WO (1) WO2017177752A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110659334A (zh) * 2019-09-20 2020-01-07 浪潮电子信息产业股份有限公司 元数据路径信息存取方法、装置、设备及可读存储介质
CN110968549A (zh) * 2019-11-18 2020-04-07 Oppo（重庆）智能科技有限公司 文件存储的方法、装置、电子设备及介质
CN112416858A (zh) * 2020-11-09 2021-02-26 深圳市珍爱捷云信息技术有限公司 文档存储方法、装置、电子设备和计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110173162A1 (en) * 2010-01-14 2011-07-14 Anderson Eric A Scrubbing procedure for a data storage system
CN103186668A (zh) * 2013-03-11 2013-07-03 北京京东世纪贸易有限公司 基于键值数据库的数据处理方法与装置以及数据存储系统
CN103853714A (zh) * 2012-11-28 2014-06-11 中国移动通信集团河南有限公司 一种数据处理方法和装置
CN103902479A (zh) * 2014-03-27 2014-07-02 浪潮电子信息产业股份有限公司 一种基于元数据日志的元数据缓存快速重建机制

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8325924B2 (en) * 2009-02-19 2012-12-04 Microsoft Corporation Managing group keys
CN103327052B (zh) * 2012-03-22 2018-04-03 深圳市腾讯计算机系统有限公司 数据存储方法和系统以及数据访问方法和系统
CN103595797B (zh) * 2013-11-18 2017-01-18 上海爱数信息技术股份有限公司 一种分布式存储系统中的缓存方法
CN103699585B (zh) * 2013-12-06 2017-04-19 华为技术有限公司 文件的元数据存储以及文件恢复的方法、装置和系统
CN104156278B (zh) * 2014-08-01 2017-06-27 江苏大学 一种文件版本控制系统及其方法
CN104239438B (zh) * 2014-08-29 2017-11-10 北京大学深圳研究生院 基于分离存储的文件信息存储方法和文件信息读写方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110173162A1 (en) * 2010-01-14 2011-07-14 Anderson Eric A Scrubbing procedure for a data storage system
CN103853714A (zh) * 2012-11-28 2014-06-11 中国移动通信集团河南有限公司 一种数据处理方法和装置
CN103186668A (zh) * 2013-03-11 2013-07-03 北京京东世纪贸易有限公司 基于键值数据库的数据处理方法与装置以及数据存储系统
CN103902479A (zh) * 2014-03-27 2014-07-02 浪潮电子信息产业股份有限公司 一种基于元数据日志的元数据缓存快速重建机制

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110659334A (zh) * 2019-09-20 2020-01-07 浪潮电子信息产业股份有限公司 元数据路径信息存取方法、装置、设备及可读存储介质
CN110968549A (zh) * 2019-11-18 2020-04-07 Oppo（重庆）智能科技有限公司 文件存储的方法、装置、电子设备及介质
CN110968549B (zh) * 2019-11-18 2024-03-29 Oppo（重庆）智能科技有限公司 文件存储的方法、装置、电子设备及介质
CN112416858A (zh) * 2020-11-09 2021-02-26 深圳市珍爱捷云信息技术有限公司 文档存储方法、装置、电子设备和计算机可读存储介质

Also Published As

Publication number Publication date
CN107301177B (zh) 2020-02-18
CN107301177A (zh) 2017-10-27

Similar Documents

Publication Publication Date Title
US10579831B2 (en) Verification of data set components using digitally signed probabilistic data structures
CN112765271B (zh) 区块链交易索引的存储方法、装置、计算机设备及介质
US9552382B2 (en) Reference counter integrity checking
WO2017201977A1 (fr) Procédé et appareil d'écriture et de lecture de données, et grappe de stockage d'objet distribuée
WO2017185616A1 (fr) Procédé de stockage de fichiers et équipement électronique
WO2019085474A1 (fr) Procédé de mise en œuvre d'un moteur de calcul, dispositif électronique et support d'informations
WO2020211236A1 (fr) Procédé et appareil de résolution de conflit de lecture-écriture utilisant un arbre b+, et support de stockage
WO2017032312A1 (fr) Procédé et appareil de présentation de données à nettoyer, et dispositif électronique
WO2017118171A1 (fr) Procédé et appareil de migration de données
WO2017177752A1 (fr) Procédé et dispositif de stockage de fichier
US20200065074A1 (en) Devices, systems, and methods of program identification, isolation, and profile attachment
CN109597707B (zh) 克隆卷数据拷贝方法、装置及计算机可读存储介质
CN110347900B (zh) 一种关键词的重要度计算方法、装置、服务器及介质
JP6788002B2 (ja) モバイル端末のためのデータ記憶方法及び装置
CN112559913B (zh) 一种数据处理方法、装置、计算设备及可读存储介质
US8527478B1 (en) Handling bulk and incremental updates while maintaining consistency
CN112363814A (zh) 任务调度方法、装置、计算机设备及存储介质
WO2019071907A1 (fr) Procédé d'identification d'informations d'aide d'après une page d'opération, et serveur d'applications
US11126520B2 (en) Skew detector for data storage system
US20170169044A1 (en) Property retrieval apparatus, method and system
CN109542860B (zh) 基于hdfs的业务数据管理方法、终端设备
WO2020258652A1 (fr) Procédé et système de remplacement de caractères, appareil informatique et support d'informations lisible par ordinateur
WO2018094689A1 (fr) Procédé, appareil et dispositif permettant d'améliorer l'expérience de navigation
CN110636042B (zh) 一种服务端已验证块高的更新方法、装置及设备
US11132401B1 (en) Distributed hash table based logging service

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17781728

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17781728

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17781728

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03/05/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 17781728

Country of ref document: EP

Kind code of ref document: A1