CN115509446A - Metadata garbage identification method, device and equipment - Google Patents
Metadata garbage identification method, device and equipment Download PDFInfo
- Publication number
- CN115509446A CN115509446A CN202211127823.1A CN202211127823A CN115509446A CN 115509446 A CN115509446 A CN 115509446A CN 202211127823 A CN202211127823 A CN 202211127823A CN 115509446 A CN115509446 A CN 115509446A
- Authority
- CN
- China
- Prior art keywords
- storage
- key value
- metadata
- key
- garbage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a device and equipment for identifying metadata garbage, which are used for solving the technical problems of identification and marking of garbage metadata in a storage system adopting a redirection write ROW technology. The invention uses the database storage metadata realized based on the structured data merging storage engine, adds the processing of identifying and marking the garbage metadata in the data merging process, and realizes the high-efficiency identification and marking of the garbage metadata by utilizing the structured data merging storage engine to execute the data merging process, thereby improving the utilization rate of storage space and the storage performance.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a metadata garbage identification method, a metadata garbage identification device and metadata garbage identification equipment.
Background
The Structured Log Merge (LSM) storage architecture is a generic name of a data structure and a method designed by making full use of the characteristic that batch sequential writing is much higher than random writing performance, changing a large amount of random accesses to storage from random to sequential, integrating data written into storage equipment in a background, and releasing the occupied space of invalid data.
The LSM key structure LSM-Tree (LSM-Tree) is composed of two or more storage structures, for example, one storage structure resident in the memory, called C0 Tree, and may be any convenient key-value pair lookup data structure, such as a red-black Tree, a map, or even a skip list. Another storage structure resides in a non-volatile storage device, such as a disk, called a C1 tree, which is structured like a B-tree. C1 all nodes are 100% full, the size of the node being the size of the basic storage block (e.g., disk block) of the device.
In a database based on an LSM storage architecture, data is divided into two parts, namely baseline data (such as SST) and incremental data (such as MemTable), the baseline data is usually stored in a disk in the form of an ordered key value pair aggregate file (such as SST file), and is loaded into a memory cache of the database when reading is needed, when the data is continuously inserted (or modified) into the incremental data (such as MemTable) cached in the memory, when the incremental data reaches a preset threshold, the incremental data is refreshed into a nonvolatile storage device, and when the incremental data on the storage device reaches a certain threshold, the threshold data and the baseline data on the storage device are merged. The core design idea of the LSM architecture is to keep the modification increments for data in memory, and write these modification operations in bulk to the non-volatile storage device after reaching a specified size limit (configurable), thereby greatly improving performance. Regarding the node data structure of the Tree, different databases can be designed according to specific service characteristics when realizing a specific storage engine based on the LSM-Tree concept.
With the continuous decline of the price of the flash disk and the great performance advantage of the flash disk relative to the mechanical disk, the enterprise data storage of the full flash memory is accepted by more and more enterprises. However, the existing storage systems are designed based on mechanical disks or mixed flash, and the performance of the full flash disk cannot be fully exerted at all. Regardless of a flash disk or a mechanical disk, the bandwidth performance of random Write small IO (Input/Output) is different from that of sequential large IO by a large amount, and the technology of Redirect On Write (ROW) can combine a plurality of random small IOs into a sequential large IO, so that the multi-Write operation of the nonvolatile storage device is converted into Write-once operation, and the Write performance of the system can be effectively improved.
The ROW is applicable to both a mechanical disk and a flash disk, but the ROW also has an additional benefit for the flash disk, that is, the sequential IO can effectively reduce the overhead of garbage collection (referring to the process of collecting the storage space which is no longer used) of the flash disk, and further improve the performance of the flash disk. Therefore, ROW is an essential function of flash memory. After the ROW function is realized, a mechanism for recovering invalid data (or old data) in the ROW and releasing the space of the storage system is needed to be realized, the effective utilization rate of the system space is improved, and the competitiveness of a product is improved.
The premise of garbage collection is that it is required to accurately identify which data is garbage data, the garbage data is identified by processing the metadata of the data, whether the data block in the current system is still used is identified by the metadata, and if the data block is not used any more, the data can be identified as garbage data.
The invention provides a storage system supporting a ROW function, and aims to efficiently identify and clean garbage data through metadata and improve the utilization rate and storage performance of a storage space.
Disclosure of Invention
In view of this, the present invention provides a metadata garbage recognition method, apparatus and device, which are used to solve the technical problem of recognizing and marking garbage metadata in a storage system that employs a redirect-on-write (ROW) technique.
Based on one aspect of the embodiments of the present invention, the present invention provides a metadata spam recognition method, including:
storing metadata in the distributed storage system by adopting a structured data merging storage engine;
in the process of merging the upper-layer storage into the lower-layer storage performed by the structured data merging storage engine, the junk metadata in the lower-layer storage is identified and marked through the traversal matching of the keywords of the key value pairs in the upper-layer storage and the lower-layer storage.
Further, the method for identifying and marking the garbage metadata in the lower storage through the traversal matching of the keywords of the key-value pairs in the upper storage and the lower storage comprises the following steps:
screening out a related ordered key value pair set with intersection of all key values in the lower storage based on each ordered key value pair set in the upper storage when a data merging process from the upper storage to the lower storage is triggered;
sequentially acquiring current key value pairs from each ordered key value pair set in the upper-layer storage, and performing keyword traversal matching in all relevant ordered key value pair sets in the lower-layer storage;
if the key value pairs with the same key value exist in the lower-layer storage, the key value pairs corresponding to the same key word in the lower-layer storage are marked as garbage metadata;
and adding the current key-value pair in the upper-layer storage into a cache queue to be merged.
Further, the method further comprises:
and after the key word traversal matching processing of the key value pairs in all the ordered key value pair sets in the upper-layer storage is completed, merging the key value pairs in the cache queues to be merged with the lower-layer storage.
Further, the storing metadata in the distributed storage system by using the structured data merging storage engine refers to storing metadata in the distributed storage system by using a database of the structured log merging LSM storage engine.
Further, the database of the structured log merge LSM storage engine is rocksdb;
the ordered set of key-value pairs is a sorted string table SST.
Based on another aspect of the embodiments of the present invention, the present invention further provides a metadata garbage recognition apparatus, which may be implemented in software, hardware, or a combination of software and hardware. When implemented in a software module, when the program code of the software module is loaded into the storage medium of the electronic device, the program code in the storage medium is read by the processor for execution, so as to implement the functions of each component module in the apparatus. The device includes:
the metadata storage module is used for storing metadata in the distributed storage system by adopting a structured data merging storage engine;
and the garbage identification module is used for identifying and marking the garbage metadata in the lower storage through traversing and matching the keywords of the key value pairs in the upper storage and the lower storage in the process of executing the merging from the upper storage to the lower storage by the structured data merging storage engine.
Further, the garbage recognition module includes:
the screening module is used for screening out a related ordered key value pair set with intersection of all key values in the lower-layer storage based on each ordered key value pair set in the upper-layer storage when the data merging process from the upper-layer storage to the lower-layer storage is triggered;
the traversing marking module is used for sequentially acquiring the current key value pair from each ordered key value pair set in the upper-layer storage and performing key word traversing matching in all relevant ordered key value pair sets in the lower-layer storage; if the key value pairs with the same key value exist in the lower-layer storage, the key value pairs corresponding to the same key word in the lower-layer storage are marked as garbage metadata; and adding the current key-value pair in the upper-layer storage into a cache queue to be merged.
Further, after the garbage recognition module completes the key word traversal matching processing of the key value pairs in all the ordered key value pair sets in the upper-layer storage, the structured data merging storage engine in the metadata storage module merges the key value pairs in the cache queue to be merged with the lower-layer storage.
The invention uses the database storage metadata realized based on the structured data merging storage engine, adds the processing of identifying and marking the garbage metadata in the data merging process, and realizes the high-efficiency identification and marking of the garbage metadata by utilizing the structured data merging storage engine to execute the data merging process, thereby improving the utilization rate of storage space and the storage performance.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments of the present invention or the description in the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and it is also possible for those skilled in the art to obtain other drawings according to the drawings of the embodiments of the present invention.
FIG. 1 is a diagram illustrating redirecting write structured metadata when writing in a structured data merged storage architecture database, in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram of a structural process for implementing metadata garbage recognition by using rocksdb according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating steps of a metadata spam recognition method according to an embodiment of the present invention;
fig. 4 is a diagram illustrating an example of an electronic device structure for implementing the metadata spam recognition method provided by the present invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. As used in this embodiment of the invention, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Depending on the context, moreover, the word "if" may be used to be interpreted as "at 8230; \8230when" or "when 8230; \8230when" or "in response to a determination".
When writing data into a nonvolatile storage device, a storage system supporting redirection during writing generally writes the data into the storage device in a form of data blocks of a fixed size, records and manages the written data through metadata, and combines the repeated data in order to improve the utilization rate of a storage space. For distributed storage systems, metadata is typically stored in separate storage spaces for centralized management and maintenance.
All write operations of a database (LSM database for short) of a storage engine realized based on an LSM tree are written in a new write mode, and original data cannot be modified, so that the cost brought by modifying metadata can be obviously reduced by using the LSM database to store the metadata. The LSM database has a data merging function, and after the capacity of the database reaches a certain proportion, data merging (compact) can be started, wherein the data merging is performed from an upper level storage layer to a lower level storage layer.
In the storage system supporting the ROW redirection function during writing, the garbage recognition can also complete the storage and maintenance of metadata by means of other databases of a non-LSM storage architecture, so as to achieve the purpose of garbage recognition marking. For example, mysql and the like are used, when determining whether data is garbage data, mysql needs to be queried in the mysql firstly, and if the data exists, the data needs to be immediately marked as garbage data in an IO flow. Therefore, the way of storing metadata by using the database of the LSM storage mechanism cannot satisfy the scenario with high performance requirement.
Based on the analysis, the invention creatively provides a method for storing metadata of a distributed or non-distributed storage system by using a database based on a structured data merging storage architecture, and adding a recognition repeated key value key in the data merging process to efficiently mark junk data. The non-volatile storage media employed by the distributed or non-distributed storage system may be magnetic disks or flash memory disks.
The Structured Log (Log-Structured) is a specific example of Structured data, and the database of the Structured data merging storage architecture in the present invention refers to all databases that store data in a Structured data structure, support redirection during writing, and support hierarchical merging of data, including databases of an LSM storage architecture, such as rocksdb, levelDB, tikv database, and the like.
For convenience of description, the implementation process of the present invention is described in detail below by taking the database rocksdb of the LSM storage architecture as an example, but it can be understood by those skilled in the art that the technical solution of the present invention is also applicable to databases of other structured data merging storage architectures.
FIG. 1 is a diagram illustrating redirecting write structured metadata during a write in a structured data merge storage architecture database, according to an embodiment of the invention. As shown in the figure, metadata is mostly stored in a structured data structure, when a user metadata object (userboj) located in a volatile storage medium (for short, a memory) is written in a non-volatile storage medium (for example, a magnetic disk, a flash disk, or the like, for short, an external storage) in a ROW manner, for multiple writes of the same userboj, the userboj is written to a new storage location when the ROW function is implemented, and metadata already written in the external storage is not modified.
Taking rocksdb as an example for storing metadata, when the metadata in the memory is successfully written into the rocksdb based on a redirect-on-write (ROW) mechanism, one key-value key value pair is written into the rocksdb every time, the same user metadata object (UserObj) is identified by the same key, and when the same UserObj is written for multiple times, the key values are the same and the key values are different when the rocksdb is written, so that two or more different key value pairs may exist in one key in the rocksdb. After a large amount of metadata is written in the rocksdb, the rocksdb triggers a data merging (Compact) process, and in the process of merging the large amount of metadata from Level1 to Level2, if it is identified that the metadata object pointed by the userboj 1_ LBA1 is redirected to the storage location of the RowObj2_ LBA2, the metadata stored at the location of the RowObj1_ LBA1 before redirection is identified as garbage data.
In the LSM storage architecture, the amount of data stored in each storage layer level may be configured, and the difference in storage capacity between general levels is set to 10 times, which may be configured in a more specific case.
Fig. 2 is a schematic diagram of a structural process for implementing metadata garbage recognition by using rocksdb in an embodiment of the present invention. In a rocksdb database system, data is divided into two parts, namely baseline data (stored in an SST format) and incremental data, the baseline data is stored in a nonvolatile storage medium (such as a disk, a flash disk, and the like) in a structural form of a Sorted String Table (SST, a form of an ordered key-value pair set), when data in a library needs to be read and written, latest baseline data is loaded into a memory and cached in the memory in a form of a memory Table (Mem Table), data in the Mem Table is written into the baseline data in a manner of incremental data for modifications (including insert, delete, and modify) of the data in the Mem Table. Data in the memory table is provided with atomicity and persistence support in a Write-ahead logging (WAL) mechanism before formal writing to the nonvolatile storage medium.
With the increment data cached in the memory increasing continuously, when the increment data reaches a preset threshold value, the increment data is refreshed into the nonvolatile storage device, and the increment data and the baseline data are merged. The core design idea of the LSM storage architecture is to keep the modification increment of the data in a memory, and write the modification operation into the nonvolatile storage device in batch after reaching the specified size limit (configurable), thereby greatly improving the performance. Different databases can be designed according to specific service characteristics when a specific storage engine is realized based on the LSM-Tree idea.
As shown in the example of FIG. 2, in the database implemented by the LSM storage engine, after a large amount of metadata is written in the upper storage Level1, a data merging (Compact) process is triggered. If key value pairs with the same key word (UserObj 1_ LBA 1) and different values exist in the Level1 and the Level2 of the user metadata object UserObj1 in the merging process, the key value pairs (UserObj 1_ LBA1 and RowObj1_ LBA 1) in the lower storage Level2 are marked as garbage metadata.
Fig. 3 is a flowchart illustrating steps of a metadata garbage recognition method according to an embodiment of the present invention. The method is applied to a distributed storage system that stores metadata by using a structured data merge storage engine (such as an LSM storage engine), and the metadata garbage recognition method of this embodiment is described in detail below with reference to fig. 2, where the method includes:
and (4) supposing that the current data merging process of storing the Level0 layer to the Level1 layer storage layer is triggered, and adding a metadata garbage mark processing task in the data merging process. In the metadata spam tag processing task, the relevant SST in Level1 is screened against the key range of each ordered set of key-value pairs SST00, SST01 and SST02 in Level 0. Since the key-value pairs contained in SST are stored in key order, for example, assume that:
the key range of SST00 is key00=10000 to 11000,
…
the key range of SST10 is key10= 9500-10500,
the key range of SST11 is key11= 10600-11600,
the key range of SST12 is key12= 12000-13000,
…
and matching the key ranges of the SST00 and the SST 10-SST 14 one by one in Level1 according to the key range of the SST00, and screening out the related SST of the SST00 as SST10 and SST11.
The above-described screening process is performed for each SST in Level0, so that all the related SSTs that intersect with all the key values in the underlying storage related to each SST in Level0 are screened from Level 1.
In RocksDB, a Sorted String Table (SST) is a form of a set of ordered key-value pairs, and each SST is stored in a file. The SST file in RocksDB is an ordered, immutable, persistent map structure, in which key-value pairs of a map may be strings of arbitrary bytes, supporting the use of a specified key to find values or traverse all key-value pairs through a given key range.
taking SST 00-SST 02 in Level0 as an example, sequentially obtaining each key value pair from SST00 as a current key value pair, using the key words of the current key value pair to perform key word traversal matching in SST related to SST00 in Level1, if the key value pair of the same key word is matched in related SST in Level1, marking the key value pair corresponding to the current key word in SST related to SST00 in Level1 as garbage metadata, and adding the key value pair corresponding to the current key word in SST00 into a cache queue to be merged. And if the key value pair of the same key word is not matched in the lower Level1, adding the key value pair corresponding to the current key word in the SST00 into a cache queue to be merged.
After the traversal matching of each key value pair in the SST00 is completed, each key value pair in the SST01 and the SST02 is sequentially subjected to the same processing until the processing of all the key value pairs in the SST in the Level0 is completed.
And step 306, after the key word traversal matching processing of the key value pairs in all the ordered key value pair sets in the upper-layer storage is completed, carrying out merging Compact processing on the key value pairs in the cache queues to be merged and the lower-layer storage, namely executing merging processing from the upper-layer storage to the lower-layer storage.
Key values marked as garbage metadata are subsequently cleaned up by a garbage handler.
Fig. 4 is a diagram illustrating an example of an electronic device implementing a metadata spam recognition method provided by the present invention, where the device 400 includes: a processor 410 such as a Central Processing Unit (CPU), a communication bus 420, a communication interface 440, and a storage medium 430. Wherein the processor 410 and the storage medium 430 can communicate with each other through a communication bus 420. The storage medium 430 stores therein a computer program that, when executed by the processor 410, implements the functions of one or more steps of the metadata garbage recognition method provided by the present invention.
The storage medium may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. In addition, the storage medium may be at least one memory device located remotely from the processor. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Further, operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A metadata garbage recognition method is characterized by comprising the following steps:
storing metadata in the distributed storage system by adopting a structured data merging storage engine;
in the process of merging the upper-layer storage into the lower-layer storage performed by the structured data merging storage engine, the garbage metadata in the lower-layer storage is identified and marked through the traversing matching of the keywords of the key value pairs in the upper-layer storage and the lower-layer storage.
2. The method of claim 1, wherein identifying and marking the garbage metadata located in the lower storage through keyword traversal matching of the key-value pairs in the upper storage and the lower storage is by:
screening out a related ordered key value pair set with intersection of all key values in the lower storage based on each ordered key value pair set in the upper storage when a data merging process from the upper storage to the lower storage is triggered;
sequentially acquiring a current key value pair from each ordered key value pair set in the upper-layer storage, and performing keyword traversal matching on all relevant ordered key value pair sets in the lower-layer storage;
if the key value pairs with the same key value exist in the lower-layer storage, the key value pairs corresponding to the same key word in the lower-layer storage are marked as garbage metadata;
and adding the current key-value pair in the upper-layer storage into a cache queue to be merged.
3. The method of claim 2, further comprising:
and after the key word traversal matching processing of the key value pairs in all the ordered key value pair sets in the upper-layer storage is completed, merging the key value pairs in the cache queues to be merged with the lower-layer storage.
4. The method of claim 3,
the step of storing the metadata in the distributed storage system by adopting the structured data merging storage engine refers to the step of storing the metadata in the distributed storage system by adopting a database of the structured log merging LSM storage engine.
5. The method of claim 4,
the database of the structured log merging LSM storage engine is rocksdb;
the ordered set of key-value pairs is a sorted string table SST.
6. A metadata spam recognition apparatus, comprising:
the metadata storage module is used for storing metadata in the distributed storage system by adopting a structured data merging storage engine;
and the garbage identification module is used for identifying and marking the garbage metadata in the lower storage through traversing and matching the keywords of the key value pairs in the upper storage and the lower storage in the process of executing the merging from the upper storage to the lower storage by the structured data merging storage engine.
7. The apparatus of claim 6, wherein the garbage recognition module comprises:
the screening module is used for screening out a related ordered key value pair set with intersection of all key values in the lower-layer storage based on each ordered key value pair set in the upper-layer storage when a data merging process from the upper-layer storage to the lower-layer storage is triggered;
the traversal marking module is used for sequentially acquiring the current key value pair from each ordered key value pair set in the upper-layer storage and performing keyword traversal matching on all related ordered key value pair sets in the lower-layer storage; if the key value pairs with the same key value exist in the lower-layer storage, the key value pairs corresponding to the same key word in the lower-layer storage are marked as garbage metadata; and adding the current key-value pair in the upper-layer storage into a cache queue to be merged.
8. The apparatus of claim 7,
and after the garbage recognition module completes the key word traversal matching processing of the key value pairs in all the ordered key value pair sets in the upper-layer storage, merging the key value pairs in the cache queue to be merged with the lower-layer storage by a structured data merging storage engine in the metadata storage module.
9. An electronic device is characterized by comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium are communicated with each other through the communication bus;
a storage medium for storing a computer program;
a processor for performing the method steps of any one of claims 1-5 when executing a computer program stored on a storage medium.
10. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211127823.1A CN115509446A (en) | 2022-09-16 | 2022-09-16 | Metadata garbage identification method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211127823.1A CN115509446A (en) | 2022-09-16 | 2022-09-16 | Metadata garbage identification method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115509446A true CN115509446A (en) | 2022-12-23 |
Family
ID=84504185
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211127823.1A Pending CN115509446A (en) | 2022-09-16 | 2022-09-16 | Metadata garbage identification method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115509446A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117009439A (en) * | 2023-10-07 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Data processing method, device, electronic equipment and storage medium |
-
2022
- 2022-09-16 CN CN202211127823.1A patent/CN115509446A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117009439A (en) * | 2023-10-07 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN117009439B (en) * | 2023-10-07 | 2024-01-23 | 腾讯科技（深圳）有限公司 | Data processing method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10162598B2 (en) | Flash optimized columnar data layout and data access algorithms for big data query engines | |
| US11403321B2 (en) | System and method for improved performance in a multidimensional database environment | |
| CN111399777A (en) | Differentiated key value data storage method based on data value classification | |
| KR102564170B1 (en) | Method and device for storing data object, and computer readable storage medium having a computer program using the same | |
| CN110109910A (en) | Data processing method and system, electronic equipment and computer readable storage medium | |
| US9280551B2 (en) | De-duplication deployment planning | |
| EP3242227A1 (en) | Page querying method and data processing node in oltp cluster database | |
| CN112463058B (en) | Fragmented data sorting method and device and storage node | |
| US20180011897A1 (en) | Data processing method having structure of cache index specified to transaction in mobile environment dbms | |
| CN115509446A (en) | Metadata garbage identification method, device and equipment | |
| Fevgas et al. | LB-Grid: An SSD efficient grid file | |
| CN107846327A (en) | A kind of processing method and processing device of network management performance data | |
| US20070255772A1 (en) | Method for fast deletion of physically clustered data | |
| US10114878B2 (en) | Index utilization in ETL tools | |
| US20240220470A1 (en) | Data storage device and storage control method based on log-structured merge tree | |
| EP3698254B1 (en) | Memory allocation in a data analytics system | |
| CN113544683A (en) | Data generalization device, data generalization method, and program | |
| KR101744017B1 (en) | Method and apparatus for indexing data for real time search | |
| JPWO2012081165A1 (en) | Database management apparatus and database management method | |
| CN112597070B (en) | Object recovery method and device | |
| Lee et al. | Iterator Interface Extended LSM-tree-based KVSSD for Range Queries | |
| US11544197B2 (en) | Random-access performance for persistent memory | |
| CN114841806A (en) | Mode matching system under real-time transfer link | |
| US8977814B1 (en) | Information lifecycle management for binding content | |
| JP5626561B2 (en) | Information processing system and data management method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |