CN115576956B - Data processing method, system, equipment and storage medium - Google Patents
Data processing method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN115576956B CN115576956B CN202211562654.4A CN202211562654A CN115576956B CN 115576956 B CN115576956 B CN 115576956B CN 202211562654 A CN202211562654 A CN 202211562654A CN 115576956 B CN115576956 B CN 115576956B
- Authority
- CN
- China
- Prior art keywords
- key value
- key
- value pair
- value
- pba
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 230000004044 response Effects 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 claims description 23
- 238000003780 insertion Methods 0.000 claims description 19
- 230000037431 insertion Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000008520 organization Effects 0.000 abstract description 20
- 229920001485 poly(butyl acrylate) polymer Polymers 0.000 description 130
- 238000012005 ligant binding assay Methods 0.000 description 68
- 238000013507 mapping Methods 0.000 description 26
- 238000007726 management method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000008713 feedback mechanism Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 230000005055 memory storage Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, which relates to the field of storage and comprises the following steps: in response to receiving a write request, calculating a fingerprint value; inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value; in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA; and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair. The invention also discloses a system, a computer device and a readable storage medium. According to the scheme provided by the invention, the combination of the PBA and the LBA is used as the key value, so that the number of the LBA corresponding to the PBA does not need to be inquired in advance, the KV organization mode of the node is optimized, the overflow tolerance is supported, and the condition that the same P value overflows is fundamentally solved.
Description
Technical Field
The present invention relates to the field of storage, and in particular, to a data processing method, system, device, and storage medium.
Background
Metadata (Mete data) refers to data (data about data) describing data, and can be understood as data with a wider scope than general meaning, and not only represents information such as type, name, value and the like of the data, but also further provides context information of the data, such as a domain to which the data belongs, a data source and the like. In a data storage system, metadata is the basis for information storage, being the smallest unit of data. In recent years, with the development of information technology, a huge amount of data is generated, but how to effectively manage and organize the huge amount of data has become a prominent problem. For a large amount of stored data, the query analyzes the data content and data meaning in the stored data, so that the data can be utilized more effectively. Efficient organization and management of metadata in a storage system is an effective means to solve this problem, and can support the management and maintenance of data by the system. In short, only valid management metadata, data becomes more valuable. Therefore, how to effectively manage metadata and use metadata is a very worth of discussion.
The online data deduplication characteristic is the most important and necessary characteristic in the full flash memory storage system, because the rear end of the full flash memory storage system uses the SSD hard disk as a storage medium, in view of the value of the SSD disk, the full flash memory storage system requires online data deduplication to achieve reduction of the actual storage space of the rear end disk. In order to realize the full flash online deduplication, the metadata management is crucial, and the metadata management mainly manages L-P mapping, P-L mapping and H-P mapping relations. Compared with the traditional characteristic that online deduplication is not supported, the metadata management has two metadata of P-L mapping and H-P mapping, and the metadata management is more stressed in terms of massive, high-concurrency and short-delay data access. In some special scenes, such as a write scene with high local deduplication data, the deduplication data is mainly concentrated in a few or few data, and in such a case, the number of P-L mapping relationships corresponding to the same PBA is very large, so that management becomes difficult, and the number of mappings of L corresponding to a single P must be limited, so that a mode of pre-query before insertion into a B + tree is usually used, but this method must queue up the same PBA to perform serialized query, and if a threshold value is exceeded, the remaining queued requests need to be processed again, and enter the longest IO processing path, that is, the current PBA needs to be rewound, and metadata is inserted again after a new PBA is generated, so that processing delay is greatly increased, and performance of the full flash storage system and a system deduplication rate are reduced.
Disclosure of Invention
In view of this, in order to overcome at least one aspect of the above problems, an embodiment of the present invention provides a data processing method, including the following steps:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data processing system, including:
a calculation module configured to calculate a fingerprint value in response to receiving a write request;
the query module is configured to query a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
a write module configured to write the data in the write request to a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
and the inserting module is configured to establish a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the insertion module is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, the query module is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the recording module is configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, the insertion module is further configured to:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, the insertion module is further configured to:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, the insertion module is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH quick look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer apparatus, including:
at least one processor; and
a memory storing a computer program operable on the processor, wherein the processor executes the program to perform the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA carried in the PBA + write request determined according to the corresponding HP key value pair as a key value and the value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH quick look-up table.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA carried in the PBA + write request determined according to the corresponding HP key value pair as a key value and the value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The invention has one of the following beneficial technical effects: the invention does not need to inquire the number of LBAs corresponding to the PBA in advance, optimizes the node KV organization mode, supports overflow tolerance, and has the core that the PBA and the LBA are combined to be used as a key value, thereby fundamentally solving the overflow condition of the same P value, and simultaneously increasing an overflow feedback mechanism. Therefore, the condition that the LBA corresponding to a single PBA is too much, the height of an individual PBA tree is too high, and the balance is influenced is avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of full flash access metadata provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a modified B + tree structure according to an embodiment of the present invention;
FIG. 4 is a block flow diagram of a data processing method provided by an embodiment of the invention;
FIG. 5 is a block diagram of a data processing system according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In the embodiment of the invention, the LBA, logical block address, and the logical block address;
PBA, physical block address;
METADATA, METADATA;
LP, LBA- > PBA, key value pair, LP mapping Key value pair;
PL, PBA- > LBA, key value pair, PL mapping Key value pair;
HP, HASH KEY- > PBA, key value pair, HP map KEY value pair.
According to an aspect of the present invention, an embodiment of the present invention provides a data processing method, as shown in fig. 1, which may include the steps of:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
According to the scheme provided by the invention, the PBA and the LBA are combined to be used as the key value, so that the number of the LBA corresponding to the PBA does not need to be inquired in advance, the KV organization mode of the node is optimized, the overflow tolerance is supported, and the overflow condition of the same P value is fundamentally solved.
In some embodiments, the full flash metadata management has three mapping relationships of LP, PL, and HP, which correspond to an LP tree, a PL tree, and an HP tree, respectively. The LP tree is a mapping organization of the L-P, and mainly has the functions of mapping a logical address LBA of a volume to a physical address PBA of a physical pool, and performing read-write and garbage recycling of non-deleted data for a user host; the PL tree is a P-L mapping organization, and mainly has the functions of mapping the physical address of the pool to the logical address of the volume, so that whether the physical address PBA is used or not is inquired for garbage collection; the HP tree is a mapping organization of H-P and is used by a deduplication module, H represents a fingerprint value of data and mainly has the function of mapping a data fingerprint to a physical address of a pool, when the deduplication function is started, the fingerprint value of newly written data is calculated firstly, then HP mapping is inquired, and if the condition that P is inquired shows that the same data exists in the physical pool, the physical address does not need to be allocated.
As shown in fig. 2, when a data query request is to query data, first querying metadata to find an L- > P mapping relationship, first accessing a metadata cache, if corresponding metadata is found in the cache, directly performing metadata verification and returning to a query request, otherwise, accessing metadata on an SSD disk, then returning to the query request, and finally, accessing data corresponding to a PBA by the query request; it can be seen that the access efficiency of metadata is very important for data reading and writing, and the organization method of metadata is very important for the access efficiency of metadata.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
Specifically, as shown in fig. 3, in order to reduce the flow, a new data organization structure needs to be designed, the node KV organization mode needs to be optimized, and overflow tolerance is supported.
The PL tree is characterized in that a data organization mode of the PL tree is optimized, the overflow of a K value is avoided, KV of the PL tree is formed, the original modes of K [ PBA ] and V [ LBA ] are changed into the modes of K [ PBA & LBA ] and V [ null ], the core is that the PBA and the LBA are combined to be used as the K value, one or more LBAs are still found in the PL tree from the PBA, but the LBA is not the same under the condition that the PBA is the same, the K value corresponding to the K value is the same, and the K value is changed into different K values, so that the overflow of the same K value is fundamentally solved. In this case, the data organization mode in the intermediate node is also changed from K [ PBA ], V [ Pointer ] to K [ PBA & LBA ], V [ Pointer ].
The non-leaf nodes of the B + tree after reconstruction: k (P-L), V (lower node address); leaf node size and length did not change: k (P-L), V (null, 0 bytes).
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
Specifically, when an individual PBA is found to exceed a certain limit, an asynchronous HP deleting task is dispatched, and when data of the same content falls to the disk again, a new PBA is generated. Therefore, the condition that the LBA corresponding to a single PBA is too much, the height of an individual PBA tree is too high, and the balance is influenced is avoided.
As shown in fig. 4, in the process, a PH fast look-up table needs to be added in the memory, and the fast look-up table is used to implement mapping from PBA to fingerprint value H, so that fingerprint values H corresponding to PBAs exceeding a threshold number are found through the table, and the corresponding mapping is deleted in the HP.
The implementation process can be as follows:
1. when the host write request reaches the deduplication module, a deduplication fingerprint value H is calculated, and then the metadata module is removed by the fingerprint value H to inquire the HP;
2. inquiring PBA according to H, if yes, turning to 3, otherwise, turning to 4;
3. if the PBA is inquired, the data can be deleted again, and LP and PL mapping is directly inserted into the metadata module;
4. if the H does not inquire the PBA, the data is not deleted again, the data is firstly written on a disc, and after the PBA is distributed, the mapping relation of LP, PL and HP is inserted into the metadata module;
5. inserting or updating a PH quick look-up table in the process of inserting the mapping relation into the metadata cache;
6. when the metadata mapping relation is organized into a B + tree disk-brushing state, whether the number of LBAs corresponding to the same PBA exceeds a threshold value or not is judged;
7. if the threshold value is exceeded, an HP deleting task is derived to delete the PBA mapping relation corresponding to the H;
8. after the mapping relation is deleted, the PH record is deleted from the PH fast look-up table, so that the newly deleted new data can be distributed to new PBAs;
according to the process, the number of the L corresponding to the P does not need to be inquired in advance, a new data organization structure needs to be designed, a node KV organization mode is optimized, overflow tolerance is supported, the core is that the PBA and the LBA are combined to be used as the K value, the problem that the same P value overflows is solved fundamentally, meanwhile, an overflow feedback mechanism is added, when the fact that the individual P exceeds a certain limit is found, a task of asynchronously deleting the HP is dispatched, and when the data with the same content falls to the disk again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data processing system 400, as shown in fig. 5, including:
a calculation module 401 configured to calculate a fingerprint value in response to receiving a write request;
a query module 402 configured to query a corresponding HP key value pair in a B + tree recording HP key value pairs according to the fingerprint value;
a write module 403, configured to write the data in the write request into a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
an inserting module 404 configured to establish a PL key-value pair with the LBA carried in the allocated PBA + the write request as a key value and the value as null, and insert the PL key-value pair into a B + tree in which the PL key-value pair is recorded.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is a lower node address, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, the insertion module 404 is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the insertion module 404 is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, the query module 402 is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, the insertion module 404 is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, the recording module is configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, the insertion module 404 is further configured to:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, the insertion module 404 is further configured to:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, the insertion module 404 is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The scheme provided by the invention does not need to pre-query the number of L corresponding to P any more, needs to design a new data organization structure, optimizes a node KV organization mode, supports overflow tolerance, and has the core that PBA and LBA are combined to be used as a K value, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, a task of asynchronously deleting HP is dispatched, and when data with the same content falls to the disk again, a new P is generated. Therefore, the situation that the balance is influenced due to the fact that the height of an individual P tree is too high due to too much L corresponding to a single P is avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 6, an embodiment of the present invention further provides a computer apparatus 501, including:
at least one processor 520; and
a memory 510, the memory 510 storing a computer program 511 operable on the processor, the processor 520 when executing the program performing the steps of:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in a B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating the task to asynchronously delete the HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
The scheme provided by the invention does not need to inquire the number of L corresponding to P in advance, needs to design a new data organization structure, optimizes a node KV organization mode and supports overflow tolerance, and the core is to combine PBA and LBA as a value K, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, an asynchronous HP deleting task is dispatched, and when the data with the same content falls down again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the repeated deletion rate of the whole storage system.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 7, an embodiment of the present invention further provides a computer-readable storage medium 601, the computer-readable storage medium 601 stores a computer program 610, and the computer program 610 performs the following steps when executed by a processor:
s1, responding to a received writing request, and calculating a fingerprint value;
s2, inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
s3, in response to the condition that the corresponding HP key value pair is not inquired, writing the data in the writing request into a data disc and distributing PBA;
and S4, establishing a PL key value pair by taking the LBA carried in the distributed PBA + write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, in the B + tree recording PL key-value pairs, the key value of a non-leaf node is PBA + LBA, the value is the address of the lower node, and the key value of a leaf node is PBA + LBA, and the value is null.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
In some embodiments, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
In some embodiments, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
In some embodiments, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
In some embodiments, further comprising:
and establishing a PL key value pair by taking the LBA + carried in the write request determined according to the corresponding HP key value pair as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
In some embodiments, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
In some embodiments, generating a task to asynchronously delete an HP key-value pair further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
In some embodiments, further comprising:
and deleting the corresponding PH key value pair in the PH quick look-up table.
The scheme provided by the invention does not need to inquire the number of L corresponding to P in advance, needs to design a new data organization structure, optimizes a node KV organization mode and supports overflow tolerance, and the core is to combine PBA and LBA as a value K, so that the overflow condition of the same P value is fundamentally solved, an overflow feedback mechanism is added, when an individual P exceeds a certain limit, an asynchronous HP deleting task is dispatched, and when the data with the same content falls down again, a new P is generated. Therefore, the problems that the L corresponding to a single P is too much, the height of an individual P tree is too high and the balance is influenced are avoided. So as to achieve the win-win of the performance and the deduplication rate of the whole storage system.
Finally, it should be noted that, as understood by those skilled in the art, all or part of the processes in the methods of the embodiments described above may be implemented by instructing relevant hardware by a computer program, and the program may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (24)
1. A data processing method, comprising the steps of:
in response to receiving a write request, calculating a fingerprint value;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value, wherein the key value of the HP key value pair is the fingerprint value, and the value is PBA;
in response to not inquiring the corresponding HP key-value pair, writing the data in the write request into a data disc and distributing PBA;
and establishing a PL key value pair by taking the distributed PBA and the LBA carried in the write request as key values and value values as null values, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
2. The method of claim 1, wherein in the B + tree that records PL key-value pairs, non-leaf nodes have key values of PBA and LBA, value values are lower node addresses, leaf nodes have key values of PBA and LBA, and value values are null.
3. The method of claim 1, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
4. The method of claim 1, further comprising:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
5. The method of claim 1, further comprising:
and responding to the corresponding HP key-value pair inquired, and determining a PBA according to the corresponding HP key-value pair.
6. The method of claim 5, further comprising:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
7. The method of claim 5, further comprising:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
8. The method of claim 7, further comprising:
and establishing a PL key value pair by taking the PBA determined according to the corresponding HP key value pair and the LBA carried in the write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
9. The method of claim 8, further comprising:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
10. The method of claim 9, wherein generating the task to asynchronously delete the HP key-value pair, further comprises:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring corresponding HP key value pairs in the B + tree for recording the HP key value pairs according to the fingerprint values;
deleting the corresponding HP key-value pair.
11. The method of claim 10, further comprising:
and deleting the corresponding PH key value pair in the PH fast look-up table.
12. A data processing system, comprising:
a calculation module configured to calculate a fingerprint value in response to receiving a write request;
the query module is configured to query a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value, wherein the key value of the HP key value pair is a fingerprint value, and the value is a PBA;
a write module configured to write the data in the write request to a data disk and allocate a PBA in response to not querying the corresponding HP key value pair;
and the inserting module is configured to establish a PL key value pair by taking the allocated PBA and the LBA carried in the write request as a key value and taking a value as null, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
13. The system of claim 12, wherein in the B + tree that records PL key-value pairs, non-leaf nodes have key values of PBA and LBA, value values are lower node addresses, leaf nodes have key values of PBA and LBA, and value values are null.
14. The system of claim 12, wherein the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and the PBA as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
15. The system of claim 12, wherein the insertion module is further configured to:
and establishing an HP key value pair by taking the fingerprint value as a key value and the PBA as a value, and inserting the HP key value pair into the B + tree for recording the HP key value pair.
16. The system of claim 12, wherein the query module is further configured to:
and responding to the corresponding HP key-value pair inquired, and determining the PBA according to the corresponding HP key-value pair.
17. The system of claim 16, wherein the insertion module is further configured to:
and establishing an LP key value pair by taking the LBA carried in the write request as a key value and taking the PBA determined according to the corresponding HP key value pair as a value, and inserting the LP key value pair into a B + tree for recording the LP key value pair.
18. The system of claim 16, further comprising a logging module configured to:
and establishing a PH key value pair by taking the PBA determined according to the corresponding HP key value pair as a key value and the fingerprint value as a value, and recording the PH key value pair into a PH quick look-up table.
19. The system of claim 18, wherein the insertion module is further configured to:
and establishing a PL key value pair by taking the PBA determined according to the corresponding HP key value pair and the LBA carried in the write request as key values and the value values as null values, and inserting the PL key value pair into a B + tree for recording the PL key value pair.
20. The system of claim 19, wherein the insertion module is further configured to:
judging whether the number of PL key value pairs of the same PBA in the B + tree for recording the PL key value pairs is larger than a threshold value or not;
in response to being greater than the threshold, a task is generated to asynchronously delete the HP key-value pair.
21. The system of claim 20, wherein the insertion module is further configured to:
inquiring a corresponding fingerprint value in the PH quick look-up table according to the same PBA;
inquiring a corresponding HP key value pair in a B + tree for recording the HP key value pair according to the fingerprint value;
deleting the corresponding HP key-value pair.
22. The system of claim 21, further comprising a deletion module configured to:
and deleting the corresponding PH key value pair in the PH fast look-up table.
23. A computer device, comprising:
at least one processor; and
memory storing a computer program operable on the processor, characterized in that the processor executes the program to perform the steps of the method according to any of claims 1-11.
24. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1-11.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211562654.4A CN115576956B (en) | 2022-12-07 | 2022-12-07 | Data processing method, system, equipment and storage medium |
PCT/CN2023/104398 WO2024119797A1 (en) | 2022-12-07 | 2023-06-30 | Data processing method and system, device, and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211562654.4A CN115576956B (en) | 2022-12-07 | 2022-12-07 | Data processing method, system, equipment and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115576956A CN115576956A (en) | 2023-01-06 |
CN115576956B true CN115576956B (en) | 2023-03-10 |
Family
ID=84590632
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211562654.4A Active CN115576956B (en) | 2022-12-07 | 2022-12-07 | Data processing method, system, equipment and storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115576956B (en) |
WO (1) | WO2024119797A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115576956B (en) * | 2022-12-07 | 2023-03-10 | 苏州浪潮智能科技有限公司 | Data processing method, system, equipment and storage medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103019953A (en) * | 2012-12-28 | 2013-04-03 | 华为技术有限公司 | Construction system and construction method for metadata |
CN105183399A (en) * | 2015-09-30 | 2015-12-23 | 北京奇艺世纪科技有限公司 | Data writing and reading method and device based on elastic block storage |
CN110795031A (en) * | 2019-10-17 | 2020-02-14 | 北京浪潮数据技术有限公司 | Data deduplication method, device and system based on full flash storage |
CN111125447A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Metadata access method, device and equipment and readable storage medium |
CN113867626A (en) * | 2021-08-29 | 2021-12-31 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium for optimizing performance of storage system |
CN113885789A (en) * | 2021-08-20 | 2022-01-04 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for verifying data consistency after metadata repair |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10083062B2 (en) * | 2015-07-31 | 2018-09-25 | Cisco Technology, Inc. | Data suppression for faster migration |
CN108008918A (en) * | 2017-11-30 | 2018-05-08 | 联想(北京)有限公司 | Data processing method, memory node and distributed memory system |
CN113867627B (en) * | 2021-08-29 | 2023-08-22 | 苏州浪潮智能科技有限公司 | Storage system performance optimization method and system |
CN113535708A (en) * | 2021-09-17 | 2021-10-22 | 苏州浪潮智能科技有限公司 | Data deduplication method, system, storage medium and equipment |
CN115576956B (en) * | 2022-12-07 | 2023-03-10 | 苏州浪潮智能科技有限公司 | Data processing method, system, equipment and storage medium |
-
2022
- 2022-12-07 CN CN202211562654.4A patent/CN115576956B/en active Active
-
2023
- 2023-06-30 WO PCT/CN2023/104398 patent/WO2024119797A1/en unknown
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103019953A (en) * | 2012-12-28 | 2013-04-03 | 华为技术有限公司 | Construction system and construction method for metadata |
CN105183399A (en) * | 2015-09-30 | 2015-12-23 | 北京奇艺世纪科技有限公司 | Data writing and reading method and device based on elastic block storage |
CN110795031A (en) * | 2019-10-17 | 2020-02-14 | 北京浪潮数据技术有限公司 | Data deduplication method, device and system based on full flash storage |
CN111125447A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Metadata access method, device and equipment and readable storage medium |
CN113885789A (en) * | 2021-08-20 | 2022-01-04 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for verifying data consistency after metadata repair |
CN113867626A (en) * | 2021-08-29 | 2021-12-31 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium for optimizing performance of storage system |
Also Published As
Publication number | Publication date |
---|---|
CN115576956A (en) | 2023-01-06 |
WO2024119797A1 (en) | 2024-06-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9043334B2 (en) | Method and system for accessing files on a storage system | |
WO2017201977A1 (en) | Data writing and reading method and apparatus, and distributed object storage cluster | |
CN102629247B (en) | Method, device and system for data processing | |
CN107766374B (en) | Optimization method and system for storage and reading of massive small files | |
KR20200122994A (en) | Key Value Append | |
CN110888837B (en) | Object storage small file merging method and device | |
CN104054071A (en) | Method for accessing storage device and storage device | |
CN113535670B (en) | Virtual resource mirror image storage system and implementation method thereof | |
CN115576956B (en) | Data processing method, system, equipment and storage medium | |
CN113867627B (en) | Storage system performance optimization method and system | |
WO2013075306A1 (en) | Data access method and device | |
CN109407985B (en) | Data management method and related device | |
CN113535708A (en) | Data deduplication method, system, storage medium and equipment | |
CN112416880A (en) | Method and device for optimizing storage performance of mass small files based on real-time merging | |
CN102346783B (en) | Data retrieval method and device | |
CN103942301A (en) | Distributed file system oriented to access and application of multiple data types | |
CN113821477A (en) | Metadata caching method, system, equipment and medium | |
KR101936364B1 (en) | Memory management system using flash memory and method thereof | |
CN115437579B (en) | Metadata management method and device, computer equipment and readable storage medium | |
US20200019539A1 (en) | Efficient and light-weight indexing for massive blob/objects | |
CN116610670A (en) | State data storage method and device based on block chain | |
CN110781101A (en) | One-to-many mapping relation storage method and device, electronic equipment and medium | |
CN104537023A (en) | Storage method and device for reverse index records | |
CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
CN113505086B (en) | Storage system capacity statistical method, device, storage medium and equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |