CN114896215A - Metadata storage method and device - Google Patents
Metadata storage method and device Download PDFInfo
- Publication number
- CN114896215A CN114896215A CN202210432316.2A CN202210432316A CN114896215A CN 114896215 A CN114896215 A CN 114896215A CN 202210432316 A CN202210432316 A CN 202210432316A CN 114896215 A CN114896215 A CN 114896215A
- Authority
- CN
- China
- Prior art keywords
- metadata
- message queue
- file
- engine
- sst
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The application provides a method and a device for storing metadata, wherein the method comprises the following steps: receiving a storage request of metadata of a message queue; merging the metadata of the message queues into a limited sorting character string table (SST) file; and calling a key value KV engine to store the SST file in a target storage position of a storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing. By the mode, the metadata are combined into the SST file and are additionally written into the storage device by the KV engine, so that the reading and writing performance of the metadata of the message queue is improved.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method and an apparatus for storing metadata.
Background
Message queues are important components of distributed systems, and are typically used to solve problems such as application coupling, asynchronous messages, traffic clipping, etc. With more and more processing mechanisms for accessing the message queue by the application, the problem of storing the metadata of the message queue needs to be solved urgently.
In the related art, the metadata of the message queue is generally stored through a local disk of the server. When the metadata of the message queue stored in the local disk needs to be maintained, operations such as data reading and writing, data deletion, updating of consumption point positions and the like can be performed through the local file system.
However, if the metadata of the message queue is stored in the local disk of the server, the metadata needs to be updated and persisted to the local file system in a full amount each time the metadata is changed, and then the metadata file of the message queue is serialized and persisted to the local disk. When the amount of metadata in the message queue is large, there may be a problem that the read-write performance of the metadata in the message queue is not high.
Disclosure of Invention
The embodiment of the application provides a method and a device for storing metadata, which aim to solve the problem that the read-write performance of the metadata of a message queue in the prior art is not high.
In a first aspect, an embodiment of the present application provides a method for storing metadata, where the method includes:
receiving a storage request of metadata of a message queue;
merging the metadata of the message queues into a limited Sorting String Table (SST) file;
and calling a key value KV engine to store the SST file in a target storage position of a storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing.
In an optional embodiment, before said merging the metadata of the message queue into a limited ordering string table SST file, the method further comprises:
determining whether a data amount of metadata of the message queue is greater than a data amount threshold;
the merging the metadata of the message queues into a limited sorting string table SST file includes:
and if the data volume of the metadata of the message queue is larger than the data volume threshold value, merging the metadata of the message queue into a limited sorting character string table (SST) file.
In an alternative embodiment, the merging the metadata of the message queue into a limited sorting string table SST file includes:
calling a logging structure of the KV engine and an LSM tree, and merging the metadata of the message queue into an SST file of a limited sequencing character string table;
wherein the LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.
In an optional embodiment, after the determining whether the data amount of the metadata of the message queue is greater than the data amount threshold, the method further includes:
and if the data volume of the metadata of the message queue is less than or equal to the data volume threshold value, calling a message queue model in a distributed message middleware engine, and storing the SST file corresponding to the metadata of the message queue in a local file system.
In an optional embodiment, before the invoking the key-value KV engine to store the SST file in a target storage location of a storage device, the method further comprises:
and determining the target storage position according to the header data of the message queue, the identifier of the message queue and the offset data of the message queue.
In an optional embodiment, after the invoking the key-value KV engine to store the SST file in a target storage location of a storage device, the method further comprises:
adding a deletion identifier on the SST file according to the storage time information and the range information of the SST file;
and calling a filter of the KV engine to delete the SST file containing the deletion identification.
In an alternative embodiment, the metadata of the message queue comprises index data of the message queue.
In a second aspect, an embodiment of the present application provides a device for storing metadata, where the device includes:
the receiving module is used for receiving a storage request of the metadata of the message queue;
the processing module is used for merging the metadata of the message queue into a limited sorting character string table (SST) file;
and the storage module is used for calling a key value KV engine to store the SST file in a target storage position of the storage device, and the writing mode of the KV engine on the storage device is parallel additional writing.
In an optional implementation manner, the processing module is specifically configured to determine whether a data amount of metadata of the message queue is greater than a data amount threshold; and if the data volume of the metadata of the message queue is larger than the data volume threshold value, merging the metadata of the message queue into a limited sorting character string table (SST) file.
In an optional implementation manner, the processing module is specifically configured to invoke a journaling structure of the KV engine to merge an LSM tree, and merge metadata of the message queue into a limited sorting string table SST file;
wherein the LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.
In an optional implementation manner, the processing module is further configured to invoke a message queue model in a distributed message middleware engine if the data amount of the metadata of the message queue is less than or equal to the data amount threshold, and store the metadata of the message queue in a local file system.
In an optional implementation manner, the processing module is further configured to determine the target storage location according to header data of the message queue, an identifier of the message queue, and offset data of the message queue.
In an optional implementation manner, the processing module is further configured to add a deletion identifier to the SST file according to the storage time information and the range information of the SST file; and calling a filter of the KV engine to delete the SST file containing the deletion identification.
In an alternative embodiment, the metadata of the message queue comprises index data of the message queue.
In a third aspect, the present application further provides an electronic device, including: a processor, and a memory; the memory is for storing a computer program for the processor; the processor is configured to implement any one of the possible methods of the first aspect by executing the computer program.
In a fourth aspect, the present invention also provides a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform any one of the possible methods of the first aspect.
In a fifth aspect, the present invention also provides a computer program for causing a computer to perform the method of the first aspect.
In a sixth aspect, the disclosed embodiments provide a computer program product comprising a computer program that, when executed by a processor, implements a method as described above in the first aspect and in various possible designs of the first aspect.
According to the metadata storage method and device provided by the embodiment of the application, a server firstly receives a metadata storage request of a message queue. The server then merges the metadata of the message queues into a limited ordering string table SST file. And finally, the server calls a key value KV engine to store the SST file in a target storage position of the storage device, and the KV engine writes in the storage device in a parallel additional writing mode. By the method, the small files can be combined into the SST file, so that the read-write bottleneck of a large number of small files is solved. Moreover, the SST is additionally written into the target storage position in parallel by using the KV engine, so that the distribution and serialization of the large object are reduced, and the memory recovery pressure of the large object is reduced. Based on this, the method and the device for storing the metadata provided by the embodiment of the application improve the reading and writing performance of the metadata of the message queue.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings needed to be used in the description of the embodiments or the prior art, and obviously, the drawings in the following description are some embodiments of the present invention, and those skilled in the art can obtain other drawings according to the drawings without inventive labor.
Fig. 1 is a schematic application scenario diagram of a metadata storage method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a metadata storage method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of another metadata storage method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a further metadata storage method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a metadata storage device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein is intended to be open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It is noted that references to "a" or "an" in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will appreciate that references to "one or more" are intended to be exemplary and not limiting unless the context clearly indicates otherwise.
First, technical names according to embodiments of the present application will be described below.
KV engine: an assembly for providing high performance reading and writing to KV data structures.
And (3) producing a terminal: a terminal device for generating a message to a message queue.
A consumption terminal: a terminal device for obtaining a message from a consumption queue for consumption.
Message: a data information carrying communication.
LMS tree: a memory structure converts random writes to sequential writes.
SST documents: a file containing arbitrary length sorted by key value.
Message queues are an important component of distributed systems, and are typically used to solve problems such as application coupling, asynchronous messages, traffic clipping, etc. With the increasing application of the processing mechanism of the access message queue, the problem of storing the metadata of the message queue needs to be solved.
Two common ways of storing metadata for a message queue are provided below.
In the first message queue storage method, the metadata of the message queue is usually stored through a local disk of the server. When the metadata of the message queue stored in the local disk needs to be maintained, operations such as data reading and writing, data deletion, updating of consumption point positions and the like can be performed through the local file system.
When the metadata of the message queue of the distributed message middleware is stored in the local disk through the local file system, various loading modes can be supported, for example, the loading modes such as NIOFS, MMAP, SIMPLEFS, SMB and the like. In some cases, the metadata file of the message queue can also be directly snapped into the MMAP of the (LOCK) memory.
Among other things, the distributed message middleware may be, for example, rockmq, Kafka, etc.
However, for storing the metadata of the message queue on the local disk of the server, the following problems may exist:
first, each metadata change requires a full update and persistence to the local file system, and then JSON serializes the metadata file of the message queue and persistence to the local disk. The large object allocation has a large pressure on memory garbage recovery of the message queue system, so that the read-write performance of the metadata of the message queue is not high.
Secondly, due to the fact that a message queue model of the distributed message middleware is used, when metadata of a message queue are large, a single machine uses MMAP mapping to read and write files, and the problems that the number of the files is limited and the reading and writing performance is sharply reduced due to a large number of small files exist.
And thirdly, the metadata of the message queue is stored in a local disk of the server, so that the failure recovery time of the message system is long. Moreover, a large number of message queues may appear as a large number of file descriptors in the local file system, thereby occupying a large amount of system resources.
In a second message queue storage approach, metadata for all message queues may be persistently stored in a centralized database node. However, since the database is made a single point of the entire message queuing system, a single point bottleneck is easily formed. When directed to a large number of small files, the performance of the database system is limited.
Based on this, the storage modes of the two message queues have low reading and writing performance of the metadata of the message queues.
In order to solve the above problem, embodiments of the present application provide a method and an apparatus for storing metadata, where metadata in a message queue is merged into a finite String Table (SST) file, and a Key Value (KV) engine is called to store the SST file in a target storage location of a storage device. By the method, small files are combined into the SST file, so that the read-write bottleneck of a large number of small files is solved. Moreover, the SST is additionally written into the target storage position in parallel by using the KV engine, so that the distribution and serialization of the large object are reduced, and the memory recovery pressure of the large object is reduced. Based on this, the method and the device for storing the metadata provided by the embodiment of the application improve the reading and writing performance of the metadata of the message queue.
An application scenario of the metadata storage method according to the embodiment of the present application is described below.
Fig. 1 is a schematic application scenario diagram of a metadata storage method according to an embodiment of the present application. As shown in fig. 1, the terminal device 101 can access metadata of a message queue in the server 102. When terminal device 101 sends a storage request for metadata to server 102, server 102 may merge the metadata of the message queue into an SST file and invoke the KV engine to store the SST file in a target storage location of server 102.
The terminal device 101 may be a tablet computer (pad), a Virtual Reality (VR) terminal device, an Augmented Reality (AR) terminal device, a wireless terminal in a self driving (self driving), a wireless terminal in a remote surgery (remote medical supply), a wireless terminal in a smart grid (smart grid), a wireless terminal in a smart home (smart home), and the like.
It is understood that the storage method of the metadata is implemented by the storage device of the metadata provided by the embodiment of the present application, and the storage device of the metadata may be part or all of a certain device, for example, a server or a processor of the server.
The following takes a server integrated or installed with relevant execution codes as an example, and details the technical solution of the embodiment of the present application with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 2 is a schematic flowchart of a metadata storage method according to an embodiment of the present application. The present embodiment relates to a specific process of how a server stores metadata of a message queue in a storage device. As shown in fig. 2, the method for storing metadata includes:
s201, receiving a storage request of metadata of the message queue.
In the application, when the terminal device accesses the message queue of the access server, a storage request of the metadata of the message queue can be sent to the server.
It should be understood that the terminal device sending the storage request of the metadata of the message queue is not limited in the embodiments of the present application, and in some embodiments, the terminal device may be a producer terminal and may be a consumer terminal. The consumer terminal is a terminal device capable of generating a message to a message queue, and the consumer terminal is a terminal device capable of acquiring the message from the message queue for consumption.
And the messages in the message queue are used for carrying data information of communication.
Accordingly, the metadata of the message queue is used to describe the messages in the message queue. It should be noted that, in the embodiment of the present application, there is no limitation on the type of metadata to be stored, and for example, the metadata may be index data, consumption site data, title (Topic) data, or the like.
S202, merging the metadata of the message queues into a limited sorting character string table SST file.
In this step, when the server receives the metadata of the message queue sent by the terminal device, the metadata of the message queue may be merged into the SST file, thereby reducing the read-write pressure of the storage device of the server.
The SST file is a file which internally contains a sequence with any length according to key values.
It should be understood that the embodiment of the present application is not limited to how the server merges the metadata of the message queue into the limited ordered string table SST file, and in some embodiments, the server may call a Log Structured Merge Tree (LSM) Tree of the KV engine to Merge the metadata of the message queue into the limited ordered string table SST file.
The LSM tree is used for converting the metadata of the message queue from random writing into sequential writing, so that the writing speed is greatly improved, and a high-efficiency KV engine is provided.
It should be understood that the present embodiments are not limited to the type of KV engine, and in some embodiments, the KV engine may include levelDB, rocksDB, badger, and the like.
In some embodiments, the server may also determine whether the data amount of the metadata (e.g., Topic data) of the message queue is greater than a data amount threshold. And if the data volume of the metadata of the message queue is larger than a volume threshold, merging the metadata of the message queue into an SST file of a limited sorting character string table, and calling a KV engine to store the SST file in a target storage position of the storage device in a parallel additional writing mode. If the data quantity of the metadata of the message Queue is smaller than or equal to the quantity threshold value, calling a message Queue (Queue) model in a distributed message middleware engine, and storing the SST file corresponding to the metadata of the message Queue in a local file system. Among other things, the distributed message middleware may be, for example, rockmq, Kafka, etc.
It should be noted that, in the embodiment of the present application, the data amount threshold is not limited, and may be specifically set according to actual situations. For example, when the metadata exceeds a preset order of magnitude, the metadata can be merged and stored by using a KV engine, and for example, when the metadata does not exceed the preset order of magnitude, the Queue model in the distributed message middleware engine can be used for storage.
S203, calling a key value KV engine to store the SST file in a target storage position of the storage device.
In this step, after the server merges the metadata of the message queue into the limited sorting character string table SST file, the KV engine may be invoked to store the SST file in the target storage location of the storage device.
It should be understood that the present application is not limited to determining the target storage location, and in some embodiments, the server may determine the target storage location according to header (Topic) data of the message queue, identifier (queue Id) of the message queue, and offset (offset) data of the message queue.
The header data of the message queue and the identifier of the message queue can be used for determining the message queue of the metadata to be stored, and the offset data is used for determining the storage position of the metadata to be stored in the metadata of the message queue.
It should be understood that the KV engine according to the embodiment of the present application writes in parallel additional writing on the storage device. By customizing the capacity of the equal KV engine of the RocksDB, the equal KV engine can only be additionally written in parallel without updating data in place, so that the merging (compact) period of the KV engine is greatly reduced, and the data reading capacity is improved.
In addition, because the KV engine is used for additionally writing the metadata of the queue messages into the storage device in parallel, the related operations of the distributed database do not need to be strictly controlled, and the performance and the throughput of the whole writing of the metadata of the queue messages of the message system are improved.
It should be noted that the bottom layer of the KV engine may use a file system to store the metadata of the message queue, and when the file of the metadata is damaged, the file of the stand-alone message queue may be recovered by submitting a log (commit log) file.
In other embodiments, after the KV engine is invoked to store the SST file in the target storage location of the storage device, the server may further set storage validity period and range information to be regulated for the SST file stored on the storage device. And adding a deletion identifier on the SST file according to the storage time information and the range information of the SST file, so as to call a filter (compact filter) of the KV engine to delete the SST file containing the deletion identifier.
The filter is used for cleaning out expired data before the KV engine merges the metadata.
It should be understood that the storage period of the SST file in the embodiment of the application is not limited, and may be specifically set according to actual situations, and may be, for example, 1 day, 3 days, 7 days, and the like.
For example, if the range information to be regulated of the SST file is 1 st to 100 th and the validity period is 7 days, the SST mark with the storage time exceeding 7 days in the 1 st to 100 th SSTs may be marked with a deletion mark, so as to invoke the filter of the KV engine to delete the SST file containing the deletion mark
In the application, the storage period and the range information are set for the metadata of the queue message, so that the expired metadata can be deleted, and the reading and writing performance is improved. Meanwhile, the linear growth of small file data cannot be caused along with the growth of the Topic data, and the performance of the system is ensured.
According to the metadata storage method provided by the embodiment of the application, the server firstly receives the metadata storage request of the message queue. The server then merges the metadata of the message queues into a limited ordered string table SST file. And finally, the server calls a key value KV engine to store the SST file in a target storage position of the storage device, and the writing mode of the KV engine on the storage device is parallel additional writing. By the method, the small files can be combined into the SST file, so that the read-write bottleneck of a large number of small files is solved. Moreover, the SST is additionally written into the target storage position in parallel by using the KV engine, so that the distribution and serialization of the large object are reduced, and the memory recovery pressure of the large object is reduced. Based on this, the method and the device for storing the metadata provided by the embodiment of the application improve the reading and writing performance of the metadata of the message queue.
On the basis of the above embodiment, the storage manner of the metadata of the queue message may also be determined based on the data amount. Fig. 3 is a schematic flowchart of another metadata storage method according to an embodiment of the present application. As shown in fig. 3, the method for storing metadata includes:
s301, receiving a storage request of the metadata of the message queue.
S302, whether the data volume of the metadata of the message queue is larger than a data volume threshold value is determined.
If yes, go to step S304, otherwise go to step S303.
S303, calling a message queue model in the distributed message middleware engine, and storing the SST file corresponding to the metadata of the message queue in the local file system.
S304, merging the metadata of the message queue into a limited sorting character string table SST file.
And S305, calling a key value KV engine to store the SST file in a target storage position of the storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing.
The technical terms, technical effects, technical features and optional embodiments of steps S301 to S305 can be understood by referring to steps S201 to S203 shown in fig. 2, and repeated information will not be described herein.
On the basis of the above embodiments, the metadata storage method provided in the embodiments of the present application also performs periodic cleaning on the stored metadata. Fig. 4 is a schematic flowchart of another metadata storage method according to an embodiment of the present application. As shown in fig. 4, the method for storing metadata includes:
s401, receiving a storage request of the metadata of the message queue.
S402, merging the metadata of the message queues into a limited sorting character string table SST file.
And S403, calling a key value KV engine to store the SST file in a target storage position of the storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing.
The technical terms, technical effects, technical features and optional embodiments of steps S401 to S403 can be understood with reference to steps S201 to S203 shown in fig. 2, and repeated information will not be described herein.
S404, adding a deletion identifier on the SST file according to the storage time information and the range information of the SST file.
It should be understood that, in the embodiment of the present application, the storage life of the SST file is not limited, and may be specifically set according to actual situations, and for example, the storage life may be 1 day, 3 days, 7 days, and the like.
In some embodiments, the deletion identifier may specifically include a tombstone identifier. Which may be added when metadata of a message queue is merged into the SST to be stored on the storage device, the server detects that a tombstone of the SST stored on the storage device is identified at a preset time interval (e.g., 10 minutes, 30 minutes, 1 hour, etc.) or at a preset time point (e.g., 24 dots per day), and may delete the SST file if it is determined that the SST file exceeds the storage expiration date and is within the range to be regulated indicated by the range information.
In other embodiments, the server may further set the file of interest to the user outside the range indicated by the range information, thereby saving the file for a long time.
In the application, the storage validity period is set for the metadata of the queue message, so that the expired metadata can be deleted, and the reading and writing performance is improved. Meanwhile, the linear growth of the small file data cannot be caused along with the growth of the Topic data, and the performance of the system is ensured.
S405, calling a filter of the KV engine to delete the SST file containing the deletion identification.
The filter is used for cleaning out expired data before merging of the metadata.
According to the metadata storage method provided by the embodiment of the application, the server firstly receives the metadata storage request of the message queue. The server then merges the metadata of the message queues into a limited ordered string table SST file. And finally, the server calls a key value KV engine to store the SST file in a target storage position of the storage device, and the writing mode of the KV engine on the storage device is parallel additional writing. By the method, the small files can be combined into the SST file, so that the read-write bottleneck of a large number of small files is solved. Moreover, the SST is additionally written into the target storage position in parallel by using the KV engine, so that the distribution and serialization of the large object are reduced, and the memory recovery pressure of the large object is reduced. Based on this, the method and the device for storing the metadata provided by the embodiment of the application improve the reading and writing performance of the metadata of the message queue.
Those of ordinary skill in the art will understand that: all or part of the steps of implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer-readable storage medium, and when executed, executes the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Fig. 5 is a schematic structural diagram of a metadata storage device according to an embodiment of the present application. The storage device of the metadata may be implemented by software, hardware, or a combination of both to execute the storage method of the metadata in the above embodiments. As shown in fig. 5, the metadata storage device 500 includes: a receiving module 501, a processing module 502 and a storage module 503.
In an optional embodiment, the processing module is specifically configured to determine whether a data amount of metadata of the message queue is greater than a data amount threshold; and if the data quantity of the metadata of the message queue is larger than a quantity threshold value, merging the metadata of the message queue into a limited sorting character string table SST file.
In an optional implementation manner, the processing module is specifically configured to invoke a journaling structure of the KV engine and merge the LSM tree, and merge metadata of the message queue into a limited sorting string table SST file;
wherein the LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.
In an optional embodiment, the processing module is further configured to, if it is determined that the data amount of the metadata of the message queue is less than or equal to the amount threshold, invoke a message queue model in the distributed message middleware engine, and store the metadata of the message queue in the local file system.
In an optional implementation manner, the processing module is further configured to determine the target storage location according to header data of the message queue, an identifier of the message queue, and offset data of the message queue.
In an optional implementation manner, the processing module is further configured to add a deletion identifier to the SST file according to the storage time information and the range information of the SST file; and calling a filter of the KV engine to delete the SST file containing the deletion identifier.
In an alternative embodiment, the metadata of the message queue comprises index data of the message queue.
It should be noted that the metadata storage apparatus provided in the embodiment shown in fig. 5 may be used to execute the metadata storage method provided in any of the above embodiments, and the specific implementation and technical effects are similar and will not be described again here.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 6, the electronic device may include: a plurality of processors 601 and a memory 602. Fig. 6 shows an electronic device as an example of a processor.
A memory 602 for storing programs. In particular, the program may include program code comprising computer operational instructions configured to:
receiving a storage request of metadata of a message queue;
merging the metadata of the message queues into a limited sorting character string table SST file;
and calling a key value KV engine to store the SST file in a target storage position of the storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing.
The memory 602 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as a number of disk memories.
The processor 601 is used for executing computer-executable instructions stored in the memory 602 to implement the above-mentioned metadata storage method;
the processor 601 may be a processor (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement the embodiments of the present Application.
Alternatively, in a specific implementation, if the communication interface, the memory 602 and the processor 601 are implemented independently, the communication interface, the memory 602 and the processor 601 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. Buses may be classified as address buses, data buses, control buses, etc., but do not represent only one bus or type of bus.
Alternatively, in a specific implementation, if the communication interface, the memory 602 and the processor 601 are integrated into a chip, the communication interface, the memory 602 and the processor 601 may complete communication through an internal interface.
The electronic device provided by the embodiment of the application can be the server.
The embodiment of the application also provides a chip which comprises a processor and an interface. Wherein the interface is used for inputting and outputting data or instructions processed by the processor. The processor is used for executing the storage method of the metadata provided in the above method embodiment. The chip may be applied in a storage of metadata.
An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, are specifically, the computer-readable storage medium stores program information, and the program information is used for the storage method of the metadata.
Embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the metadata storage method as described above.
The embodiment of the application also provides a computer program, and the computer program enables a computer to execute the storage method of the metadata.
The embodiment of the application also provides a storage system of the queue message, which comprises a server and terminal equipment, wherein the server executes the storage method of the metadata.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for storing metadata, the method comprising:
receiving a storage request of metadata of a message queue;
merging the metadata of the message queues into a limited Sorting String Table (SST) file;
and calling a key value KV engine to store the SST file in a target storage position of a storage device, wherein the writing mode of the KV engine on the storage device is parallel additional writing.
2. The method as claimed in claim 1, wherein prior to said merging metadata of said message queue into a limited ordering string table (SST) file, said method further comprises:
determining whether a data amount of metadata of the message queue is greater than a data amount threshold;
the merging the metadata of the message queues into a limited sorting string table SST file includes:
and if the data volume of the metadata of the message queue is greater than the data volume threshold value, merging the metadata of the message queue into the SST file.
3. The method as claimed in claim 2, wherein said merging the metadata of the message queue into the SST file comprises:
calling a journaling structure of the KV engine and merging an LSM (local switch management) tree, and merging the metadata of the message queue into the SST file;
wherein the LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.
4. The method of claim 2, wherein after the determining whether the amount of data of the metadata of the message queue is greater than a data amount threshold, the method further comprises:
and if the data volume of the metadata of the message queue is less than or equal to the data volume threshold value, calling a message queue model in a distributed message middleware engine, and storing the SST file corresponding to the metadata of the message queue in a local file system.
5. The method of claim 1, wherein prior to said invoking the key-value KV engine to store the SST file at a target storage location of a storage device, the method further comprises:
and determining the target storage position according to the header data of the message queue, the identifier of the message queue and the offset data of the message queue.
6. The method of claim 1, wherein after said invoking the key-value KV engine to store the SST file at a target storage location of a storage device, the method further comprises:
adding a deletion identifier on the SST file according to the storage time information and the range information of the SST file;
and calling a filter of the KV engine to delete the SST file containing the deletion identification.
7. The method of any of claims 1-6, wherein the metadata of the message queue comprises index data of the message queue.
8. An apparatus for storing metadata, the apparatus comprising:
the receiving module is used for receiving a storage request of the metadata of the message queue;
the processing module is used for merging the metadata of the message queues into a limited sorting character string table SST file;
and the storage module is used for calling a key value KV engine to store the SST file in a target storage position of the storage device, and the writing mode of the KV engine on the storage device is parallel additional writing.
9. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of any of claims 1-7.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210432316.2A CN114896215A (en) | 2022-04-22 | 2022-04-22 | Metadata storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210432316.2A CN114896215A (en) | 2022-04-22 | 2022-04-22 | Metadata storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114896215A true CN114896215A (en) | 2022-08-12 |
Family
ID=82717024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210432316.2A Pending CN114896215A (en) | 2022-04-22 | 2022-04-22 | Metadata storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114896215A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048396A (en) * | 2022-12-30 | 2023-05-02 | 蜂巢科技(南通)有限公司 | Data storage device and storage control method based on log structured merging tree |
-
2022
- 2022-04-22 CN CN202210432316.2A patent/CN114896215A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048396A (en) * | 2022-12-30 | 2023-05-02 | 蜂巢科技(南通)有限公司 | Data storage device and storage control method based on log structured merging tree |
| CN116048396B (en) * | 2022-12-30 | 2024-03-08 | 蜂巢科技(南通)有限公司 | Data storage device and storage control method based on log structured merging tree |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| US8756199B2 (en) | File level hierarchical storage management system, method, and apparatus | |
| CN107045422B (en) | Distributed storage method and device | |
| CN101686209B (en) | Method and device for storing message in message retransmission system | |
| US11314689B2 (en) | Method, apparatus, and computer program product for indexing a file | |
| CN110018998B (en) | File management method and system, electronic equipment and storage medium | |
| US20190220190A1 (en) | Method and device for managing hash table, and computer program product | |
| US20090254594A1 (en) | Techniques to enhance database performance | |
| CN109213432B (en) | Storage device for writing data using log structured merge tree and method thereof | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| WO2023040399A1 (en) | Service persistence method and apparatus | |
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| CN109885577A (en) | Data processing method, device, terminal and storage medium | |
| CN107704573A (en) | A kind of intelligent buffer method coupled with business | |
| CN114706836B (en) | Data life cycle management method based on airborne embedded database | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN115878027A (en) | Storage object processing method and device, terminal and storage medium | |
| CN114896215A (en) | Metadata storage method and device | |
| CN108829345B (en) | Data processing method of log file and terminal equipment | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN112000971B (en) | File permission recording method, system and related device | |
| CN110413689B (en) | Multi-node data synchronization method and device for memory database | |
| CN111427920A (en) | Data acquisition method, device, system, computer equipment and storage medium | |
| CN111752941A (en) | Data storage method, data access method, data storage device, data access device, server and storage medium | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |