CN114443573A - Metadata retrieval method and device, electronic equipment and medium - Google Patents
Metadata retrieval method and device, electronic equipment and medium Download PDFInfo
- Publication number
- CN114443573A CN114443573A CN202210050441.7A CN202210050441A CN114443573A CN 114443573 A CN114443573 A CN 114443573A CN 202210050441 A CN202210050441 A CN 202210050441A CN 114443573 A CN114443573 A CN 114443573A
- Authority
- CN
- China
- Prior art keywords
- metadata
- search engine
- retrieved
- retrieval
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000015654 memory Effects 0.000 claims description 27
- 230000004048 modification Effects 0.000 claims description 17
- 238000012986 modification Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 12
- 230000008859 change Effects 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 6
- 230000006870 function Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- 230000005291 magnetic effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a metadata retrieval method, a metadata retrieval device, electronic equipment and a metadata retrieval medium, and relates to the technical field of cloud. The method comprises the following steps: deploying a metadata search engine; obtaining metadata to be retrieved, and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment; and acquiring retrieval conditions, and retrieving the metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions. The method and the device can realize the metadata retrieval function in the distributed file storage system so as to facilitate the management of mass data by a user.
Description
Technical Field
The present application relates to the field of cloud technologies, and in particular, to a metadata retrieval method, apparatus, electronic device, and medium.
Background
With the continuous development of information technology and the continuous growth of global data, people pay more and more attention to the management of mass data. Conventional storage systems face increasing challenges in the management of large amounts of data: the method can not quickly find out files meeting the conditions from mass data, does not support various retrieval conditions (such as file names, sizes, creation dates and the like), and does not support protocols (such as NFS/CIFS) written by unstructured data.
In response to the management demand for mass data, more and more storage systems attempt to provide simple and easy-to-use management functions, of which metadata retrieval is an important basic management function. Distributed storage systems are also widely used, can store a large amount of data and metadata, etc., and can read data or metadata based on various data manipulation commands. However, currently, in a distributed file storage system, such as cephfs, a metadata retrieval function is not supported temporarily.
Disclosure of Invention
In order to solve at least one of the problems mentioned in the background art, the present application provides a metadata retrieval method, apparatus, electronic device and medium, which can implement a metadata retrieval function in a distributed file storage system, so as to facilitate a user to manage mass data.
The embodiment of the application provides the following specific technical scheme:
in a first aspect, a metadata retrieval method is provided, which is applied to a metadata server, and includes:
deploying a metadata search engine;
obtaining metadata to be retrieved, and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment;
and acquiring retrieval conditions, and retrieving the metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
Further, the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device includes:
executing a metadata reporting step every first preset time interval;
the metadata reporting step comprises:
acquiring a metadata change event, and writing the metadata change event into a modification log;
and changing the metadata according to the modification log to obtain changed metadata, and reporting the changed metadata to the metadata search engine.
Further, the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device further includes:
and responding to the received retrieval request, and immediately executing one or more times of metadata reporting steps to update the metadata in the metadata search engine.
Further, the storage device comprises at least one of a cluster and a local disk;
the metadata to be retrieved comprises the changed metadata and the unchanged metadata;
the cluster includes a ceph cluster, and the local disk includes an object storage device.
Further, the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device further includes:
and in response to receiving a retrieval request, writing the metadata stored in the cluster and/or the local disk into the metadata search engine through an output instruction.
Further, the metadata search engine comprises a distributed full-text search engine elastic search.
Further, the retrieval condition includes at least one of a metadata type and a metadata attribute name;
the metadata type includes at least one of system metadata and custom metadata;
the metadata attribute name includes at least one of:
the metadata file extension attribute comprises a metadata name, an account to which the metadata belongs, a metadata path, a metadata size, metadata last modification time, metadata creation time, a user identification number to which the metadata belongs, a user group identification number to which the metadata belongs, and a metadata file extension attribute.
In a second aspect, a metadata retrieval apparatus is provided, the apparatus comprising:
the search engine management module is used for deploying a metadata search engine;
the metadata writing module is used for acquiring metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment;
and the metadata retrieval module is used for acquiring retrieval conditions and retrieving metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the metadata retrieval method when executing the computer program.
In a fourth aspect, a computer-readable storage medium is provided that stores computer-executable instructions for performing the metadata retrieval method.
The embodiment of the application has the following beneficial effects:
according to the metadata retrieval method, the metadata retrieval device, the electronic equipment and the media, a metadata search engine can be deployed in a distributed file storage system, metadata to be retrieved is written into the metadata search engine on the premise that service is not affected, and the metadata to be retrieved is reported by active triggering, so that the real-time performance of the metadata to be retrieved is guaranteed; and then, metadata retrieval is carried out in the metadata search engine through a plurality of retrieval conditions to meet the requirements of different users, so that the metadata retrieval function is realized in the distributed file storage system, and the users can conveniently manage mass data.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a general flowchart illustrating a metadata retrieval method provided by an embodiment of the present application;
FIG. 2 illustrates a detailed flow diagram of a metadata retrieval method according to one embodiment of the present application;
fig. 3 is a schematic structural diagram of a metadata retrieval apparatus provided in an embodiment of the present application;
FIG. 4 illustrates an exemplary system that can be used to implement the various embodiments described in this application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "S1", "S2", etc. are used for descriptive purposes only, are not intended to be used in a specific sense to refer to an order or sequence, and are not intended to limit the present application, but are merely used for convenience in describing the methods of the present application and are not to be construed as indicating the order of the steps. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Example one
The application provides a metadata retrieval method, which is applied to a metadata server, and referring to fig. 1, the method comprises the following steps:
and S1, deploying a metadata search engine.
And S2, acquiring the metadata to be retrieved, and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device.
And S3, acquiring retrieval conditions, and retrieving metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
This is further illustrated below in conjunction with fig. 2:
specifically, the distributed file storage system includes a client and a metadata server. Among other things, the client may support some unstructured data writing protocols (e.g., NFS, CIFS, etc.), the client writes the metadata, and the metadata server caches and synchronizes the distributed metadata. The distributed file storage system can comprise a cephfs system, and the Metadata server refers to a Metadata server cluster, which is called MDS for short. Aiming at the management requirement of mass data, the distributed file storage system is required to provide some simple and easy-to-use management functions, including basic management functions, and metadata retrieval belongs to one of the basic management functions. However, to implement the metadata retrieval function, a powerful search engine needs to be deployed.
In some embodiments, the metadata search engine may comprise a distributed full-text search engine elastic search.
Specifically, the Elasticsearch is a distributed, highly-extended, and highly-real-time search and data analysis engine. The Elasticsearch is also a Lucene-based search server. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. It can conveniently make a large amount of data have the capability of searching, analyzing and exploring. The horizontal flexibility of the elastic search is fully utilized, so that the data becomes more valuable in a production environment. The implementation principle of the Elasticsearch is mainly divided into the following steps: firstly, a user writes metadata into a database of an elastic search, then a word controller divides words of corresponding sentences, weights and word division results are stored into data, when the user searches the metadata, the results are ranked and scored according to the weights, and then returned results are presented to the user.
In particular, the elastic search can also be used to search various documents. It can provide extensible searching, have near real-time searching, and support multi-tenancy. Meanwhile, special configuration is not needed when multi-tenants are processed, and Solr needs more advanced settings. The Elasticsearch is distributed, which means that the index can be divided into slices (sharding), which is similar to the block mechanism of HDFS, and improves processing efficiency by dividing and multiplying. The sharding mechanism provides better distributivity, each node hosts one or more shards and acts as a coordinator to delegate operations to the correct shards. Each node forms a peer-to-peer network structure, and when some nodes fail, other nodes can be automatically allocated to work instead of the nodes. The Elasticsearch also has a high availability characteristic, and can provide a replication (replica) mechanism, and one fragment can set multiple replicas, so that in the case of a certain server being down, a cluster can still operate as usual, and can recover the replicas lost due to the server being down onto other available nodes, which is also similar to the replication mechanism of the HDFS (default is 3 replicas in the HDFS).
The following description will be further made by taking a distributed file storage system as cephfs and a metadata search engine as an elastic search (es for short) as an example:
in some embodiments, S2 includes:
and S21, executing the metadata reporting step every a first preset time interval.
The metadata reporting step comprises the following steps:
acquiring a metadata change event, and writing the metadata change event into a modification log;
and changing the metadata according to the modification log to obtain changed metadata, and reporting the changed metadata to a metadata search engine.
Specifically, the first preset time refers to a time required for the metadata reporting interval, and may be a preset standard value, or may be a custom time according to an actual requirement of a user, for example, 1 minute, 3 minutes, half an hour, and the like.
Specifically, the MDS employs a mechanism of writing a log (journal) first, and the metadata is landed by reading a write operation in the log. In order not to affect the service performance of the cluster, when the metadata (journal trim) is modified according to the log (journal), the changed metadata can be actively reported to the es. Thus, although the service performance of the cluster is not affected, the following problems may occur:
1. the method has certain influence on the real-time performance of the search, and only after the metadata (journel trim) is modified (second level), the change can be embodied in the search result;
2. if the metadata to be retrieved already exists in the cluster before the metadata retrieval function is started, and the corresponding files cannot be retrieved without any metadata change operation.
To address these issues, in some embodiments, S2 further includes:
s22, responding to the received retrieval request, immediately executing one or more times of metadata reporting steps to update the metadata in the metadata search engine.
Specifically, for the problem of modifying the metadata (journal trim), before the metadata retrieval is performed, the journal trim may be triggered one or more times, so as to ensure that the retrieved metadata is all the latest version, i.e. has real-time property.
In some embodiments, the storage device comprises at least one of a cluster and a local disk; the metadata to be retrieved comprises changed metadata and unchanged metadata; the cluster includes a ceph cluster and the local disk includes an object storage device.
Based on this, in some embodiments, S2 further includes:
and S23, in response to receiving the retrieval request, writing the metadata stored in the cluster and/or the local disk into a metadata search engine through an output instruction.
Specifically, for existing metadata, the existing metadata (unchanged metadata) may be subjected to trigger ls and other similar instruction operations through a developed interface, and the metadata stored in the cluster and/or the local disk is written into the metadata search engine. The ls command is one of the most commonly used commands in Linux, and is a very old command, which can be traced to Unix system at the earliest time, and even the same command in older Multics. The ls command writes to the standard output, each Directory specified by the Directory parameter or each name specified by the File parameter, as well as other information required. If the File or Directory parameter is not specified, the ls command displays the contents of the current Directory.
Specifically, the Object Storage Device refers to an Object Storage Device, abbreviated as OSD, and its main functions are to store data, copy data, balance data, restore data, and the like. Generally, a hard disk corresponds to an OSD, and the OSD manages the hard disk storage.
Specifically, because es has a data amplification phenomenon of about 3 times, for example, the size of written metadata is about 1M, and the capacity actually occupied in es is about 3M, if es data is directly stored in a local disk, es needs to ensure high availability of data in a copy manner, so that a separate data disk needs to be provided for es, and the capacity of the separate disk is limited, and the es data occupies a relatively large amount. If the data is stored in the ceph cluster, the es does not need to be copied any more, and the ceph cluster can ensure high availability of the data. Thus, local disks are not needed to be provided for es independently, and the capacity of the ceph cluster is large. Because es consumes more memory when querying, in order to reduce the impact of es on cluster services, it is suggested that es be deployed in a container.
In some embodiments, the retrieval condition includes at least one of a metadata type and a metadata attribute name; the metadata type comprises at least one of system metadata and custom metadata; metadata attribute names include at least one of: the metadata file extension attribute comprises a metadata name, an account to which the metadata belongs, a metadata path, a metadata size, metadata last modification time, metadata creation time, a user identification number to which the metadata belongs, a user group identification number to which the metadata belongs, and a metadata file extension attribute.
Specifically, metadata retrieval is mainly realized by the following ways: the MDS provides an inquiry interface for the outside to ensure the safety of data; the es opens an external interface for the user to directly inquire through the es. The MDS belongs to one of the assemblies of the cluster, and therefore data safety can be guaranteed. But frequent query operations may occupy resources of the MDS, thereby affecting normal business performance. Therefore, an external interface needs to be opened through the es, and because the es is a peripheral component, the es is directly opened to the outside, the safety of data cannot be guaranteed, but the influence on the service can be reduced.
Specifically, the search condition and the specific description may refer to the following table one:
watch 1
In this embodiment, a metadata search engine can be deployed in a distributed file storage system, metadata to be retrieved is written into the metadata search engine on the premise that a service is not affected, and the metadata to be retrieved is reported by actively triggering the metadata, so that the real-time performance of the metadata to be retrieved is ensured; and then, metadata retrieval is carried out in the metadata search engine through a plurality of retrieval conditions to meet the requirements of different users, so that the metadata retrieval function is realized in the distributed file storage system, and the users can conveniently manage mass data.
Example two
Corresponding to the above embodiment, the present application further provides a metadata retrieval apparatus, and with reference to fig. 3, the apparatus includes: the device comprises a search engine management module, a metadata writing module and a metadata retrieval module.
The search engine management module is used for deploying a metadata search engine; the metadata writing module is used for acquiring metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment; and the metadata retrieval module is used for acquiring retrieval conditions and retrieving metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
Further, the metadata writing module is further configured to perform a metadata reporting step every first preset time interval. Wherein, the metadata reporting step comprises: acquiring a metadata change event, and writing the metadata change event into a modification log; and changing the metadata according to the modification log to obtain changed metadata, and reporting the changed metadata to the metadata search engine.
Further, the metadata writing module is further configured to immediately execute the metadata reporting step one or more times in response to receiving the retrieval request, so as to update the metadata in the metadata search engine.
Further, the apparatus further includes a metadata storage module, configured to store the metadata to be retrieved in a cluster and/or a local disk. The metadata to be retrieved comprises the changed metadata and unchanged metadata; the cluster includes a ceph cluster, and the local disk includes an object storage device.
Further, the metadata writing module is further configured to write, in response to receiving a retrieval request, metadata stored in the cluster and/or the local disk into the metadata search engine through an output instruction.
Further, the metadata search engine comprises a distributed full-text search engine elastic search.
Further, the retrieval condition in the metadata retrieval module includes at least one of a metadata type and a metadata attribute name. Wherein the metadata type includes at least one of system metadata and custom metadata; the metadata attribute name comprises at least one of:
the metadata file extension attribute comprises a metadata name, an account to which the metadata belongs, a metadata path, a metadata size, metadata last modification time, metadata creation time, a user identification number to which the metadata belongs, a user group identification number to which the metadata belongs, and a metadata file extension attribute.
The metadata search engine, the search criteria and other relevant limitations have been described in detail in the above embodiments of the method, and therefore are not described herein again.
The modules in the metadata retrieval device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
EXAMPLE III
Corresponding to the foregoing embodiments, the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the metadata retrieval method may be implemented.
As shown in fig. 4, in some embodiments, the system can be the above-described electronic device for the metadata retrieval method in any of the above-described embodiments. In some embodiments, a system may include one or more computer-readable media (e.g., system memory or NVM/storage) having instructions and one or more processors (e.g., processor (s)) coupled with the one or more computer-readable media and configured to execute the instructions to implement modules to perform actions described herein.
For one embodiment, the system control module may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) and/or any suitable device or component in communication with the system control module.
The system control module may include a memory controller module to provide an interface to the system memory. The memory controller module may be a hardware module, a software module, and/or a firmware module.
System memory may be used, for example, to load and store data and/or instructions for the system. For one embodiment, the system memory may comprise any suitable volatile memory, such as suitable DRAM. In some embodiments, the system memory may include a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
For one embodiment, the system control module may include one or more input/output (I/O) controllers to provide an interface to the NVM/storage and communication interface(s).
For example, the NVM/storage may be used to store data and/or instructions. The NVM/storage may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drive(s) (HDD (s)), one or more Compact Disc (CD) drive(s), and/or one or more Digital Versatile Disc (DVD) drive (s)).
The NVM/storage may include storage resources that are physically part of the device on which the system is installed, or it may be accessible by the device and not necessarily part of the device. For example, the NVM/storage may be accessible over a network via the communication interface(s).
The communication interface(s) may provide an interface for the system to communicate over one or more networks and/or with any other suitable device. The system may wirelessly communicate with one or more components of the wireless network according to any of one or more wireless network standards and/or protocols.
For one embodiment, at least one of the processor(s) may be packaged together with logic for one or more controllers (e.g., memory controller modules) of the system control module. For one embodiment, at least one of the processor(s) may be packaged together with logic for one or more controllers of the system control module to form a System In Package (SiP). For one embodiment, at least one of the processor(s) may be integrated on the same die with logic for one or more controllers of the system control module. For one embodiment, at least one of the processor(s) may be integrated on the same die with logic of one or more controllers of a system control module to form a system on a chip (SoC).
In various embodiments, the system may be, but is not limited to being: a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, the system may have more or fewer components and/or different architectures. For example, in some embodiments, a system includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including touch screen displays), a non-volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and speakers.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software programs of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including associated data structures) of the present application may be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Those skilled in the art will appreciate that the form in which the computer program instructions reside on a computer-readable medium includes, but is not limited to, source files, executable files, installation package files, and the like, and that the manner in which the computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Computer-readable media herein can be any available computer-readable storage media or communication media that can be accessed by a computer.
Communication media includes media whereby communication signals, including, for example, computer readable instructions, data structures, program modules, or other data, are transmitted from one system to another. Communication media may include conductive transmission media such as cables and wires (e.g., fiber optics, coaxial, etc.) and wireless (non-conductive transmission) media capable of propagating energy waves such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied in a modulated data signal, for example, in a wireless medium such as a carrier wave or similar mechanism such as is embodied as part of spread spectrum techniques. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The modulation may be analog, digital or hybrid modulation techniques.
An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform a method and/or a solution according to the aforementioned embodiments of the present application.
Example four
Corresponding to the above embodiments, the present application also provides a computer-readable storage medium storing computer-executable instructions for performing a metadata retrieval method.
In the present embodiment, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media include, but are not limited to, volatile memory such as random access memory (RAM, DRAM, SRAM); and non-volatile memory such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disk, tape, CD, DVD); or other now known media or later developed that can store computer-readable information/data for use by a computer system.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the true scope of the embodiments of the present application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A metadata retrieval method is applied to a metadata server, and is characterized by comprising the following steps:
deploying a metadata search engine;
obtaining metadata to be retrieved, and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment;
and acquiring retrieval conditions, and retrieving the metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
2. The metadata retrieval method of claim 1, wherein the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and a storage device comprises:
executing a metadata reporting step every first preset time interval;
the metadata reporting step comprises:
acquiring a metadata change event, and writing the metadata change event into a modification log;
and changing the metadata according to the modification log to obtain changed metadata, and reporting the changed metadata to the metadata search engine.
3. The metadata retrieval method according to claim 2, wherein the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device further comprises:
and responding to the received retrieval request, immediately executing one or more times of metadata reporting steps so as to update the metadata in the metadata search engine.
4. The metadata retrieval method of claim 2, wherein the storage device comprises at least one of a cluster and a local disk;
the metadata to be retrieved comprises the changed metadata and the unchanged metadata;
the cluster includes a ceph cluster, and the local disk includes an object storage device.
5. The metadata retrieval method according to claim 4, wherein the obtaining metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage device further comprises:
and in response to receiving a retrieval request, writing the metadata stored in the cluster and/or the local disk into the metadata search engine through an output instruction.
6. The metadata retrieval method of claim 1, wherein the metadata search engine comprises a distributed full text search engine elasti csearch.
7. The metadata retrieval method according to claim 1, wherein the retrieval condition includes at least one of a metadata type and a metadata attribute name;
the metadata type includes at least one of system metadata and custom metadata;
the metadata attribute name includes at least one of:
the metadata file extension attribute comprises a metadata name, an account to which the metadata belongs, a metadata path, a metadata size, metadata last modification time, metadata creation time, a user identification number to which the metadata belongs, a user group identification number to which the metadata belongs, and a metadata file extension attribute.
8. A metadata retrieval apparatus, characterized in that the apparatus comprises:
the search engine management module is used for deploying a metadata search engine;
the metadata writing module is used for acquiring metadata to be retrieved and synchronously writing the metadata to be retrieved into the metadata search engine and the storage equipment;
and the metadata retrieval module is used for acquiring retrieval conditions and retrieving metadata in the metadata search engine according to the metadata to be retrieved and the retrieval conditions.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the metadata retrieval method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing computer-executable instructions for performing the metadata retrieval method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210050441.7A CN114443573A (en) | 2022-01-17 | 2022-01-17 | Metadata retrieval method and device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210050441.7A CN114443573A (en) | 2022-01-17 | 2022-01-17 | Metadata retrieval method and device, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114443573A true CN114443573A (en) | 2022-05-06 |
Family
ID=81367371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210050441.7A Withdrawn CN114443573A (en) | 2022-01-17 | 2022-01-17 | Metadata retrieval method and device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443573A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113898A1 (en) * | 2022-11-30 | 2024-06-06 | 苏州元脑智能科技有限公司 | Metadata reporting method and apparatus, and device and storage medium |
-
2022
- 2022-01-17 CN CN202210050441.7A patent/CN114443573A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024113898A1 (en) * | 2022-11-30 | 2024-06-06 | 苏州元脑智能科技有限公司 | Metadata reporting method and apparatus, and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831779B2 (en) | Seamless data migration across databases | |
| US10303649B2 (en) | Storage media abstraction for uniform data storage | |
| US9311252B2 (en) | Hierarchical storage for LSM-based NoSQL stores | |
| US20200257450A1 (en) | Data hierarchical storage and hierarchical query method and apparatus | |
| US9697258B2 (en) | Supporting enhanced content searches in an online content-management system | |
| US11093472B2 (en) | Using an LSM tree file structure for the on-disk format of an object storage platform | |
| US9384201B2 (en) | Method of managing data of file system using database management system | |
| US20170068469A1 (en) | Remote Shared Virtual Disk Snapshot Creation | |
| US20210011884A1 (en) | Storage organization system with associated storage utilization values | |
| CN111324665B (en) | Log playback method and device | |
| CN109804359A (en) | For the system and method by write back data to storage equipment | |
| US20190034445A1 (en) | Cognitive file and object management for distributed storage environments | |
| US11687533B2 (en) | Centralized storage for search servers | |
| WO2023232120A1 (en) | Data processing method, electronic device, and storage medium | |
| US11500873B2 (en) | Methods and systems for searching directory access groups | |
| CN115729951B (en) | Data query method, system, device and computer readable storage medium | |
| US11544229B1 (en) | Enhanced tracking of data flows | |
| WO2019057000A1 (en) | Log writing method, apparatus and system | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN114443573A (en) | Metadata retrieval method and device, electronic equipment and medium | |
| CN113918535A (en) | Data reading method, device, equipment and storage medium | |
| CN114942908B (en) | Index system, data processing method, electronic device, and medium | |
| US11989159B2 (en) | Hybrid snapshot of a global namespace | |
| US20230409235A1 (en) | File system improvements for zoned storage device operations | |
| US11726978B2 (en) | Computer program for providing efficient change data capture in a database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220506 |