CN115576956B - Data processing method, system, equipment and storage medium - Google Patents

Data processing method, system, equipment and storage medium Download PDF

Info

Publication number
CN115576956B
CN115576956B CN202211562654.4A CN202211562654A CN115576956B CN 115576956 B CN115576956 B CN 115576956B CN 202211562654 A CN202211562654 A CN 202211562654A CN 115576956 B CN115576956 B CN 115576956B
Authority
CN
China
Prior art keywords
key value
key
value pair
value
pba
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211562654.4A
Other languages
Chinese (zh)
Other versions
CN115576956A (en
Inventor
刚亚州
王见
徐洪志
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202211562654.4A priority Critical patent/CN115576956B/en
Publication of CN115576956A publication Critical patent/CN115576956A/en
Application granted granted Critical
Publication of CN115576956B publication Critical patent/CN115576956B/en
Priority to PCT/CN2023/104398 priority patent/WO2024119797A1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2246Trees, e.g. B+trees
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data processing method, which relates to the field of storage and comprises the following steps: in response to receiving a write request, calculating a fingerprint value; inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value; in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA; and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair. The invention also discloses a system, a computer device and a readable storage medium. According to the scheme provided by the invention, the combination of the PBA and the LBA is used as the key value, so that the number of the LBA corresponding to the PBA does not need to be inquired in advance, the KV organization mode of the node is optimized, the overflow tolerance is supported, and the condition that the same P value overflows is fundamentally solved.

Description

Data processing method, system, equipment and storage medium
Technical Field
The present invention relates to the field of storage, and in particular, to a data processing method, system, device, and storage medium.
Background
Metadata (Mete data) refers to data (data about data) describing data, and can be understood as data with a wider scope than general meaning, and not only represents information such as type, name, value and the like of the data, but also further provides context information of the data, such as a domain to which the data belongs, a data source and the like. In a data storage system, metadata is the basis for information storage, being the smallest unit of data. In recent years, with the development of information technology, a huge amount of data is generated, but how to effectively manage and organize the huge amount of data has become a prominent problem. For a large amount of stored data, the query analyzes the data content and data meaning in the stored data, so that the data can be utilized more effectively. Efficient organization and management of metadata in a storage system is an effective means to solve this problem, and can support the management and maintenance of data by the system. In short, only valid management metadata, data becomes more valuable. Therefore, how to effectively manage metadata and use metadata is a very worth of discussion.
The online data deduplication characteristic is the most important and necessary characteristic in the full flash memory storage system, because the rear end of the full flash memory storage system uses the SSD hard disk as a storage medium, in view of the value of the SSD disk, the full flash memory storage system requires online data deduplication to achieve reduction of the actual storage space of the rear end disk. In order to realize the full flash online deduplication, the metadata management is crucial, and the metadata management mainly manages L-P mapping, P-L mapping and H-P mapping relations. Compared with the traditional characteristic that online deduplication is not supported, the metadata management has two metadata of P-L mapping and H-P mapping, and the metadata management is more stressed in terms of massive, high-concurrency and short-delay data access. In some special scenes, such as a write scene with high local deduplication data, the deduplication data is mainly concentrated in a few or few data, and in such a case, the number of P-L mapping relationships corresponding to the same PBA is very large, so that management becomes difficult, and the number of mappings of L corresponding to a single P must be limited, so that a mode of pre-query before insertion into a B + tree is usually used, but this method must queue up the same PBA to perform serialized query, and if a threshold value is exceeded, the remaining queued requests need to be processed again, and enter the longest IO processing path, that is, the current PBA needs to be rewound, and metadata is inserted again after a new PBA is generated, so that processing delay is greatly increased, and performance of the full flash storage system and a system deduplication rate are reduced.
Disclosure of Invention
In view of this, in order to overcome at least one aspect of the above problems, an embodiment of the present invention provides a data processing method, including the following steps:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data processing system, including:
a calculation module configured to calculate a fingerprint value in response to receiving a write request;
the query module is configured to query a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
a write module configured to write the data in the write request to a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
and the inserting module is configured to establish a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the insertion module is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, the query module is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the recording module is configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, the insertion module is further configured to:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, the insertion module is further configured to:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, the insertion module is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH quick look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer apparatus, including:
at least one processor; and
a memory storing a computer program operable on the processor, wherein the processor executes the program to perform the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA carried in the PBA + write request determined according to the corresponding HP key value pair as a key value and the value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH quick look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA carried in the PBA + write request determined according to the corresponding HP key value pair as a key value and the value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The invention has one of the following beneficial technical effects: the invention does not need to inquire the number of LBAs corresponding to the PBA in advance, optimizes the node KV organization mode, supports overflow tolerance, and has the core that the PBA and the LBA are combined to be used as a key value, thereby fundamentally solving the overflow condition of the same P value, and simultaneously increasing an overflow feedback mechanism. Therefore, the condition that the LBA corresponding to a single PBA is too much, the height of an individual PBA tree is too high, and the balance is influenced is avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of full flash access metadata provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a modified B + tree structure according to an embodiment of the present invention;
FIG. 4 is a block flow diagram of a data processing method provided by an embodiment of the invention;
FIG. 5 is a block diagram of a data processing system according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In the embodiment of the invention, the LBA, logical block address, and the logical block address;
PBA, physical block address;
METADATA, METADATA;
LP, LBA- > PBA, key value pair, LP mapping Key value pair;
PL, PBA- > LBA, key value pair, PL mapping Key value pair;
HP, HASH KEY- > PBA, key value pair, HP map KEY value pair.
According to an aspect of the present invention, an embodiment of the present invention provides a data processing method, as shown in fig. 1, which may include the steps of:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
According to the scheme provided by the invention, the PBA and the LBA are combined to be used as the key value, so that the number of the LBA corresponding to the PBA does not need to be inquired in advance, the KV organization mode of the node is optimized, the overflow tolerance is supported, and the overflow condition of the same P value is fundamentally solved.
In some embodiments, the full flash metadata management has three mapping relationships of LP, PL, and HP, which correspond to an LP tree, a PL tree, and an HP tree, respectively. The LP tree is a mapping organization of the L-P, and mainly has the functions of mapping a logical address LBA of a volume to a physical address PBA of a physical pool, and performing read-write and garbage recycling of non-deleted data for a user host; the PL tree is a P-L mapping organization, and mainly has the functions of mapping the physical address of the pool to the logical address of the volume, so that whether the physical address PBA is used or not is inquired for garbage collection; the HP tree is a mapping organization of H-P and is used by a deduplication module, H represents a fingerprint value of data and mainly has the function of mapping a data fingerprint to a physical address of a pool, when the deduplication function is started, the fingerprint value of newly written data is calculated firstly, then HP mapping is inquired, and if the condition that P is inquired shows that the same data exists in the physical pool, the physical address does not need to be allocated.
As shown in fig. 2, when a data query request is to query data, first querying metadata to find an L- > P mapping relationship, first accessing a metadata cache, if corresponding metadata is found in the cache, directly performing metadata verification and returning to a query request, otherwise, accessing metadata on an SSD disk, then returning to the query request, and finally, accessing data corresponding to a PBA by the query request; it can be seen that the access efficiency of metadata is very important for data reading and writing, and the organization method of metadata is very important for the access efficiency of metadata.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
Specifically, as shown in fig. 3, in order to reduce the flow, a new data organization structure needs to be designed, the node KV organization mode needs to be optimized, and overflow tolerance is supported.
The PL tree is characterized in that a data organization mode of the PL tree is optimized, the overflow of a K value is avoided, KV of the PL tree is formed, the original modes of K [ PBA ] and V [ LBA ] are changed into the modes of K [ PBA & LBA ] and V [ null ], the core is that the PBA and the LBA are combined to be used as the K value, one or more LBAs are still found in the PL tree from the PBA, but the LBA is not the same under the condition that the PBA is the same, the K value corresponding to the K value is the same, and the K value is changed into different K values, so that the overflow of the same K value is fundamentally solved. In this case, the data organization mode in the intermediate node is also changed from K [ PBA ], V [ Pointer ] to K [ PBA & LBA ], V [ Pointer ].
The non-leaf nodes of the B + tree after reconstruction: k (P-L), V (lower node address); leaf node size and length did not change: k (P-L), V (null, 0 bytes).
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
Specifically, when an individual PBA is found to exceed a certain limit, an asynchronous HP deleting task is dispatched, and when data of the same content falls to the disk again, a new PBA is generated. Therefore, the condition that the LBA corresponding to a single PBA is too much, the height of an individual PBA tree is too high, and the balance is influenced is avoided.
As shown in fig. 4, in the process, a PH fast look-up table needs to be added in the memory, and the fast look-up table is used to implement mapping from PBA to fingerprint value H, so that fingerprint values H corresponding to PBAs exceeding a threshold number are found through the table, and the corresponding mapping is deleted in the HP.
The implementation process can be as follows:
1. when the host write request reaches the deduplication module, a deduplication fingerprint value H is calculated, and then the metadata module is removed by the fingerprint value H to inquire the HP;
2. inquiring PBA according to H, if yes, turning to 3, otherwise, turning to 4;
3. if the PBA is inquired, the data can be deleted again, and LP and PL mapping is directly inserted into the metadata module;
4. if the H does not inquire the PBA, the data is not deleted again, the data is firstly written on a disc, and after the PBA is distributed, the mapping relation of LP, PL and HP is inserted into the metadata module;
5. inserting or updating a PH quick look-up table in the process of inserting the mapping relation into the metadata cache;
6. when the metadata mapping relation is organized into a B + tree disk-brushing state, whether the number of LBAs corresponding to the same PBA exceeds a threshold value or not is judged;
7. if the threshold value is exceeded, an HP deleting task is derived to delete the PBA mapping relation corresponding to the H;
8. after the mapping relation is deleted, the PH record is deleted from the PH fast look-up table, so that the newly deleted new data can be distributed to new PBAs;
according to the process, the number of the L corresponding to the P does not need to be inquired in advance, a new data organization structure needs to be designed, a node KV organization mode is optimized, overflow tolerance is supported, the core is that the PBA and the LBA are combined to be used as the K value, the problem that the same P value overflows is solved fundamentally, meanwhile, an overflow feedback mechanism is added, when the fact that the individual P exceeds a certain limit is found, a task of asynchronously deleting the HP is dispatched, and when the data with the same content falls to the disk again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data processing system 400, as shown in fig. 5, including:
a calculation module 401 configured to calculate a fingerprint value in response to receiving a write request;
a query module 402 configured to query a corresponding HP key value pair in a B + tree recording HP key value pairs according to the fingerprint value;
a write module 403, configured to write the data in the write request into a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
an inserting module 404 configured to establish a PL key-value pair with the LBA carried in the allocated PBA + the write request as a key value and the value as null, and insert the PL key-value pair into a B + tree in which the PL key-value pair is recorded.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, the insertion module 404 is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the insertion module 404 is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, the query module 402 is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, the insertion module 404 is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the recording module is configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, the insertion module 404 is further configured to:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, the insertion module 404 is further configured to:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, the insertion module 404 is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The scheme provided by the invention does not need to pre-query the number of L corresponding to P any more, needs to design a new data organization structure, optimizes a node KV organization mode, supports overflow tolerance, and has the core that PBA and LBA are combined to be used as a K value, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, a task of asynchronously deleting HP is dispatched, and when data with the same content falls to the disk again, a new P is generated. Therefore, the situation that the balance is influenced due to the fact that the height of an individual P tree is too high due to too much L corresponding to a single P is avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 6, an embodiment of the present invention further provides a computer apparatus 501, including:
at least one processor 520; and
a memory 510, the memory 510 storing a computer program 511 operable on the processor, the processor 520 when executing the program performing the steps of:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The scheme provided by the invention does not need to inquire the number of L corresponding to P in advance, needs to design a new data organization structure, optimizes a node KV organization mode and supports overflow tolerance, and the core is to combine PBA and LBA as a value K, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, an asynchronous HP deleting task is dispatched, and when the data with the same content falls down again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the repeated deletion rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 7, an embodiment of the present invention further provides a computer-readable storage medium 601, the computer-readable storage medium 601 stores a computer program 610, and the computer program 610 performs the following steps when executed by a processor:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating a task to asynchronously delete an HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH quick look-up table.
The scheme provided by the invention does not need to inquire the number of L corresponding to P in advance, needs to design a new data organization structure, optimizes a node KV organization mode and supports overflow tolerance, and the core is to combine PBA and LBA as a value K, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, an asynchronous HP deleting task is dispatched, and when the data with the same content falls down again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Finally, it should be noted that, as understood by those skilled in the art, all or part of the processes in the methods of the embodiments described above may be implemented by instructing relevant hardware by a computer program, and the program may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims (24)

1. A data processing method, comprising the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value, wherein the key value of the HP key value pair is the fingerprint value, and the value is PBA;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the distributed PBA and the LBA carried in the write request as key values and value values as null values, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
2. The method of claim 1, wherein in the B + tree that records PL key-value pairs, non-leaf nodes have key values of PBA and LBA, value values are lower node addresses, leaf nodes have key values of PBA and LBA, and value values are null.
3. The method of claim 1, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
4. The method of claim 1, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
5. The method of claim 1, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
6. The method of claim 5, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
7. The method of claim 5, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
8. The method of claim 7, further comprising:
and establishing a PL key value pair by taking the PBA determined according to the corresponding HP key value pair and the LBA carried in the write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
9. The method of claim 8, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
10. The method of claim 9, wherein generating the task to asynchronously delete the HP key-value pair, further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
11. The method of claim 10, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
12. A data processing system, comprising:
a calculation module configured to calculate a fingerprint value in response to receiving a write request;
the query module is configured to query a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value, wherein the key value of the HP key value pair is a fingerprint value, and the value is a PBA;
a write module configured to write the data in the write request to a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
and the inserting module is configured to establish a PL key value pair by taking the allocated PBA and the LBA carried in the write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
13. The system of claim 12, wherein in the B + tree that records PL key-value pairs, non-leaf nodes have key values of PBA and LBA, value values are lower node addresses, leaf nodes have key values of PBA and LBA, and value values are null.
14. The system of claim 12, wherein the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
15. The system of claim 12, wherein the insertion module is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
16. The system of claim 12, wherein the query module is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
17. The system of claim 16, wherein the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
18. The system of claim 16, further comprising a logging module configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
19. The system of claim 18, wherein the insertion module is further configured to:
and establishing a PL key value pair by taking the PBA determined according to the corresponding HP key value pair and the LBA carried in the write request as key values and the value values as null values, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
20. The system of claim 19, wherein the insertion module is further configured to:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
21. The system of claim 20, wherein the insertion module is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
deleting the corresponding HP key-value pair.
22. The system of claim 21, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH fast look-up table.
23. A computer device, comprising:
at least one processor; and
memory storing a computer program operable on the processor, characterized in that the processor executes the program to perform the steps of the method according to any of claims 1-11.
24. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1-11.
CN202211562654.4A 2022-12-07 2022-12-07 Data processing method, system, equipment and storage medium Active CN115576956B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202211562654.4A CN115576956B (en) 2022-12-07 2022-12-07 Data processing method, system, equipment and storage medium
PCT/CN2023/104398 WO2024119797A1 (en) 2022-12-07 2023-06-30 Data processing method and system, device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211562654.4A CN115576956B (en) 2022-12-07 2022-12-07 Data processing method, system, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN115576956A CN115576956A (en) 2023-01-06
CN115576956B true CN115576956B (en) 2023-03-10

Family

ID=84590632

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211562654.4A Active CN115576956B (en) 2022-12-07 2022-12-07 Data processing method, system, equipment and storage medium

Country Status (2)

Country Link
CN (1) CN115576956B (en)
WO (1) WO2024119797A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115576956B (en) * 2022-12-07 2023-03-10 苏州浪潮智能科技有限公司 Data processing method, system, equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103019953A (en) * 2012-12-28 2013-04-03 华为技术有限公司 Construction system and construction method for metadata
CN105183399A (en) * 2015-09-30 2015-12-23 北京奇艺世纪科技有限公司 Data writing and reading method and device based on elastic block storage
CN110795031A (en) * 2019-10-17 2020-02-14 北京浪潮数据技术有限公司 Data deduplication method, device and system based on full flash storage
CN111125447A (en) * 2019-12-22 2020-05-08 北京浪潮数据技术有限公司 Metadata access method, device and equipment and readable storage medium
CN113867626A (en) * 2021-08-29 2021-12-31 苏州浪潮智能科技有限公司 Method, system, equipment and storage medium for optimizing performance of storage system
CN113885789A (en) * 2021-08-20 2022-01-04 苏州浪潮智能科技有限公司 Method, system, device and medium for verifying data consistency after metadata repair

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10083062B2 (en) * 2015-07-31 2018-09-25 Cisco Technology, Inc. Data suppression for faster migration
CN108008918A (en) * 2017-11-30 2018-05-08 联想(北京)有限公司 Data processing method, memory node and distributed memory system
CN113867627B (en) * 2021-08-29 2023-08-22 苏州浪潮智能科技有限公司 Storage system performance optimization method and system
CN113535708A (en) * 2021-09-17 2021-10-22 苏州浪潮智能科技有限公司 Data deduplication method, system, storage medium and equipment
CN115576956B (en) * 2022-12-07 2023-03-10 苏州浪潮智能科技有限公司 Data processing method, system, equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103019953A (en) * 2012-12-28 2013-04-03 华为技术有限公司 Construction system and construction method for metadata
CN105183399A (en) * 2015-09-30 2015-12-23 北京奇艺世纪科技有限公司 Data writing and reading method and device based on elastic block storage
CN110795031A (en) * 2019-10-17 2020-02-14 北京浪潮数据技术有限公司 Data deduplication method, device and system based on full flash storage
CN111125447A (en) * 2019-12-22 2020-05-08 北京浪潮数据技术有限公司 Metadata access method, device and equipment and readable storage medium
CN113885789A (en) * 2021-08-20 2022-01-04 苏州浪潮智能科技有限公司 Method, system, device and medium for verifying data consistency after metadata repair
CN113867626A (en) * 2021-08-29 2021-12-31 苏州浪潮智能科技有限公司 Method, system, equipment and storage medium for optimizing performance of storage system

Also Published As

Publication number Publication date
CN115576956A (en) 2023-01-06
WO2024119797A1 (en) 2024-06-13

Similar Documents

Publication Publication Date Title
US9043334B2 (en) Method and system for accessing files on a storage system
WO2017201977A1 (en) Data writing and reading method and apparatus, and distributed object storage cluster
CN102629247B (en) Method, device and system for data processing
CN107766374B (en) Optimization method and system for storage and reading of massive small files
KR20200122994A (en) Key Value Append
CN110888837B (en) Object storage small file merging method and device
CN104054071A (en) Method for accessing storage device and storage device
CN113535670B (en) Virtual resource mirror image storage system and implementation method thereof
CN115576956B (en) Data processing method, system, equipment and storage medium
CN113867627B (en) Storage system performance optimization method and system
WO2013075306A1 (en) Data access method and device
CN109407985B (en) Data management method and related device
CN113535708A (en) Data deduplication method, system, storage medium and equipment
CN112416880A (en) Method and device for optimizing storage performance of mass small files based on real-time merging
CN102346783B (en) Data retrieval method and device
CN103942301A (en) Distributed file system oriented to access and application of multiple data types
CN113821477A (en) Metadata caching method, system, equipment and medium
KR101936364B1 (en) Memory management system using flash memory and method thereof
CN115437579B (en) Metadata management method and device, computer equipment and readable storage medium
US20200019539A1 (en) Efficient and light-weight indexing for massive blob/objects
CN116610670A (en) State data storage method and device based on block chain
CN110781101A (en) One-to-many mapping relation storage method and device, electronic equipment and medium
CN104537023A (en) Storage method and device for reverse index records
CN111444114B (en) Method, device and system for processing data in nonvolatile memory
CN113505086B (en) Storage system capacity statistical method, device, storage medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant