WO2024104073A1

WO2024104073A1 - Metadata access method and device, and storage medium

Info

Publication number: WO2024104073A1
Application number: PCT/CN2023/126791
Authority: WO
Inventors: 王淏舟; 杨峻峰; 冯雷
Original assignee: 杭州拓数派科技发展有限公司
Priority date: 2022-11-14
Filing date: 2023-10-26
Publication date: 2024-05-23
Also published as: CN115470008A; CN115470008B

Abstract

A metadata access method and device, and a storage medium. The method comprises: a data extractor extracting target metadata from a metadata service and caching same to a cloud disk, and a loader loading cached data of the target metadata into a first memory of the loader from the cloud disk, wherein the target metadata is metadata having the data updating frequency lower than a first preset value and the data access frequency higher than a second preset value in the metadata service; in response to a metadata access request of a master node, loading the cached data of the target metadata in the first memory into a second memory of a slave node corresponding to the master node; and the slave node reading the second memory to access the target metadata.

Description

A metadata access method, device and storage medium

Related Applications

This application claims priority to the Chinese patent application filed on November 14, 2022, with application number 202211418015.0, and invention name “A metadata access method, device and storage medium”, the entire contents of which are incorporated by reference into this application.

Technical Field

The present application relates to the field of data processing, and in particular to a metadata access method, device and storage medium.

Background technique

The traditional metadata access method is that the user sends the access instruction to the distribution node, the distribution node connects to a master node, and sends the access instruction to the master node. After receiving the instruction, the master node pulls metadata from the metadata service to parse the instruction. The master node starts the computing node and sends the instruction to the computing node. After the computing node starts, it receives the instruction, and the computing node pulls metadata from the metadata service and processes the instruction. The computing node returns the processing result and exits and is destroyed to release computing resources. In the metadata access method in traditional technology, all nodes (distribution nodes, master nodes, and slave nodes) need to access the metadata service. The data volume is large, which occupies a large amount of network bandwidth, increases network overhead and cost, greatly increases the load of the metadata service, and leads to poor database performance, and requires more resources to be allocated to the metadata service. The high load of the metadata service limits the maximum number of nodes, which means that it limits the performance of the entire database cluster.

Currently, no effective solution has been proposed to solve the problem in traditional technologies that all nodes need to access the metadata service, resulting in high metadata service load.

Summary of the invention

According to various embodiments of the present application, a metadata access method, device, and storage medium are provided.

In a first aspect, a metadata access method is provided in this embodiment, and the method includes:

The target metadata is extracted from the metadata service by a data extractor and cached to a cloud disk, and the loader loads the cached data of the target metadata from the cloud disk to the first memory of the loader; the target metadata is metadata in the metadata service whose data update frequency is lower than a first preset value and whose data access frequency is higher than a second preset value;

In response to a metadata access request from the master node, the cache data of the target metadata in the first memory is loaded into a second memory of a slave node corresponding to the master node; the slave node reads the second memory to access the target metadata.

In some embodiments, before loading the cache data of the target metadata into the first memory, the method includes:

Obtaining target metadata from the metadata service;

Generate cache data of the target metadata according to the acquired target metadata.

In some embodiments, generating cache data of the target metadata according to the acquired target metadata includes:

Classifying the target metadata according to the data type of the target metadata to obtain type information;

Performing feature extraction on the target metadata;

Encoding the extracted features and the classified target metadata to obtain encoded data;

The cache data is generated according to the encoded data and the type information.

The version information of the target metadata is obtained, and the cache data is generated according to the version information.

In some embodiments, the method further comprises:

In response to a data update instruction sent by the metadata service, acquiring the execution status of the task of the slave node; the data update instruction is used to instruct to update the cache data of the target metadata stored in the first memory;

According to the execution state, cache data of the target metadata corresponding to the execution state is selected from the first memory and loaded into the second memory of the slave node.

In some embodiments, selecting, according to the execution state, from the first memory, cache data of the target metadata corresponding to the execution state and loading it into the second memory of the slave node comprises:

When the execution state is that the slave node executes the task, cache data of the target metadata before the update is selected from the first memory and loaded into the second memory of the slave node;

When the execution state is that the slave node is idle, the updated cache data of the target metadata is selected from the first memory and loaded into the second memory of the slave node.

In some embodiments, the method further comprises:

In response to an access request from a slave node, the address of the first memory is connected to the slave node to be accessed.

In some of the embodiments, the slave node is a stateless computing node.

In a second aspect, a metadata access device is provided in this embodiment, and the device includes:

A first loading module is used to extract target metadata from a metadata service through a data extractor and cache the target metadata to a cloud disk, and the loader loads the cached data of the target metadata from the cloud disk to a first memory of the loader; the target metadata is metadata in the metadata service whose data update frequency is lower than a first preset value and whose data access frequency is higher than a second preset value;

The access module is used to respond to the metadata access request of the master node and to The cached data of the target metadata is loaded into the second memory of the slave node corresponding to the master node; the slave node reads the second memory to access the target metadata.

According to a third aspect, a computer-readable storage medium is provided in this embodiment, on which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata access method described in the first aspect are implemented.

Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

FIG. 1 is a hardware structure block diagram of a terminal for executing a metadata access method according to one or more embodiments of the present application.

FIG. 2 is a flowchart of a metadata access method according to one or more embodiments of the present application.

FIG. 3 is a flowchart of a metadata cache generation method in one or more embodiments of the present application.

FIG. 4 is a flowchart of a metadata cache access method in one or more embodiments of the present application.

FIG5 is a flowchart of a computing node updating method in one or more embodiments of the present application.

FIG. 6 is a flowchart of a metadata cache dynamic update method in one or more embodiments of the present application.

FIG. 7 is a structural block diagram of a metadata access device according to one or more embodiments of the present application.

Detailed ways

In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And/or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" may mean: A exists alone, There are three situations: A and B exist at the same time, and B exists alone. Generally, the character "/" indicates that the objects related to each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of objects.

For ease of understanding, the description of concepts related to the present application is provided for reference by way of example. It can be understood that the description of related concepts is also part of the embodiments of the present application, as follows:

1. Elastic Distributed Computing

Computing nodes based on distributed clusters can be dynamically generated and destroyed on demand. Distributed computing node resources do not need to be generated in advance.

2. Stateless Compute Nodes

Stateless computing nodes do not store any cluster information or data, and their creation and destruction will not have any impact on the distributed cluster. All slave nodes involved in this application are stateless computing nodes, and their states are all stored in the metadata service.

3. Metadata

A type of data used to describe/execute user data/queries/operations in a database. The metadata involved in this application is stored independently. Metadata is key data in the database. Once damaged, the database will stop serving and cannot be recovered.

4. Distributed Database

The distributed database involved in this application is a distributed data with storage and computing separation in a master-segment node architecture. Among them, the master node is responsible for receiving user instructions (queries) and parsing, and the slave node is a stateless computing node in the eMPP (elastic Massive Parallel Processing) architecture, which is responsible for processing user instructions, reading and processing data, and returning the results to the master node. The general order of magnitude is one master node and thousands of slave nodes.

5. Metadata Storage/Access

The metadata storage and access involved in this application are all stored and accessed by the same node, that is, metadata is stored uniformly as key data to ensure its security. All nodes (including all master and slave nodes) need to access metadata. The metadata service system refers to a database-like service system that can provide unified metadata services for distributed databases, including storage, query, modification, and insertion.

6. User command (query)

When users add, delete, check, and modify stored data, the database master node will parse and translate the user instructions into machine language after receiving the user instructions, and then pass them to the computing node for processing. In the entire process (parsing, translation, processing), metadata is required for processing.

The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, FIG1 is a hardware structure block diagram of a terminal that executes a metadata access method of an embodiment of the present application. As shown in FIG. 1 , the terminal may include one or more (only one is shown in FIG. 1 ) processors 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 and an input/output device 108 for communication functions. It will be appreciated by those skilled in the art that the structure shown in FIG. 1 is for illustration only and does not limit the structure of the terminal. For example, the terminal may also include more or fewer components than those shown in FIG. 1 , or may have a different configuration than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a metadata access method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The transmission device 106 is used to receive or send data via a network. The above-mentioned network includes a wireless network provided by the communication provider of the terminal. In one embodiment, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

In this embodiment, a metadata access method is provided. FIG. 2 is a flowchart of a metadata access method in an embodiment of the present application. As shown in FIG. 2 , the process includes the following steps:

Step S210, extract the target metadata from the metadata service through the data extractor and cache it to the cloud disk, and the loader loads the cached data of the target metadata from the cloud disk into the first memory of the loader; the target metadata is the metadata in the metadata service whose data update frequency is lower than the first preset value and whose data access frequency is higher than the second preset value.

Specifically, this method can be applied to the eMPP architecture, and further to the distributed storage-computing separation database based on eMPP. The loader loads the cached data of the target metadata into the first memory, where the target metadata is metadata in the metadata service with a data update frequency lower than a first preset value and a data access frequency higher than a second preset value. The first memory here is the local memory of the loader, and the loader can load the cached data of the target metadata from the cloud disk to the local memory of the loader. More specifically, according to the activity level of the metadata, the metadata can be divided into hot data and cold data, where cold data can be defined as metadata with a low update frequency and a high access frequency, and hot data can be defined as metadata with a high update frequency and a low access frequency. For cold data, it will be frequently accessed by computing nodes, but its data update frequency is low. The cold data can be extracted from the metadata service as the target metadata to generate a cached data image, and the computing nodes can use the cached data from the cache. The corresponding target metadata can be obtained from the stored data image without accessing the metadata service, which greatly saves network bandwidth and reduces the complexity of the metadata service.

Exemplarily, the loader here may be a loader provided by the eMPP architecture.

Step S220, in response to the metadata access request of the master node, the cache data of the target metadata in the first memory is loaded into the second memory of the slave node corresponding to the master node; the slave node reads the second memory to access the target metadata.

Specifically, the loader responds to the metadata access request of the master node, and loads the cached data of the target metadata in the first memory into the second memory of the slave node corresponding to the master node; the slave node reads the second memory to access the target metadata, and the second memory here can be the local memory of the slave node. The slave node here is a stateless computing node, and the slave node's access to the cached data of the target metadata can only be notified to the loader by the master node. The loader responds to the metadata access request of the master node, and the metadata access request carries the information of the slave node that needs to access the target metadata, and loads the cached data of the target metadata in the local memory of the loader into the local memory of the corresponding slave node, and the slave node reads the target metadata from its local memory to access the target metadata.

In this embodiment, after the target metadata in the metadata is cached to the cloud disk, when the slave node needs to access the target metadata, the loader first loads the target metadata into the local memory of the loader. When the slave node needs to access the corresponding target metadata, the target metadata is loaded into the local memory of the slave node, thereby reducing the access pressure of the metadata service and solving the problem in the prior art that all nodes need to access the metadata service, resulting in high metadata service load.

In some of the embodiments, before loading the cache data of the target metadata into the first memory, the process includes: acquiring the target metadata in the metadata service, and generating the cache data of the target metadata according to the acquired target metadata.

Specifically, the target metadata is extracted from the metadata service, and cache data of the target metadata is generated according to the acquired target metadata.

In some of the embodiments, generating cache data of the target metadata based on the acquired target metadata includes: classifying the target metadata according to the data type of the target metadata to obtain type information; performing feature extraction on the classified target metadata; encoding the extracted features and the classified target metadata to obtain encoded data; and generating cache data based on the encoded data and the type information.

Specifically, feature extraction is performed on the classified target metadata to extract feature values, and the extracted feature values and the classified target metadata are encoded together to obtain encoded data. According to the encoded data and type information, cache data is generated, and the cache data includes the type information of the encoded data and the target metadata. A corresponding cache data image is generated and saved in the cloud disk. Exemplarily, the encoding here can be binary encoding, so that the encoded data obtained after encoding conforms to the data structure in the local memory of the loader and the local memory of the slave node. By classifying the target metadata and extracting feature values, the query speed of the target metadata in the cloud disk is accelerated.

In some of the embodiments, generating cache data of the target metadata according to the acquired target metadata includes: acquiring version information of the target metadata, and generating cache data according to the version information.

Specifically, the version information of the target metadata is obtained, the version information is generated for the cached data, the cached data with the version information is used to generate a corresponding cached data image, and the image is saved in the cloud disk.

In some of the embodiments, the metadata access method also includes a cache dynamic update process, which includes: in response to a data update instruction sent by the metadata service, obtaining the execution status of the task of the slave node; the data update instruction is used to indicate the update of the cache data of the target metadata stored in the first memory; according to the execution status, selecting the cache data of the target metadata corresponding to the execution status from the first memory and loading it into the second memory of the slave node.

Specifically, the loader responds to the data update instruction sent by the metadata service, obtains the execution status of the task of the slave node, and when the execution status is that the slave node is executing the task, selects the cache data of the target metadata before the update from the first memory and loads it into the second memory of the slave node; when the execution status is that the slave node is idle, selects the cache data of the target metadata after the update from the first memory and loads it into the second memory of the slave node. After all the slave nodes have selected the cache data of the updated target metadata from the first memory and loaded it into the second memory of the slave node, that is, after all the slave nodes have completed the cache connection update, the slave node deletes the old memory cache and the system deletes the mirror file of the old cache data.

The embodiments of the present application are described and illustrated below through one or more embodiments.

In one or more embodiments, a metadata cache generation method is provided. As shown in FIG3 , the method includes the following steps:

Step S310: The data extractor extracts the required metadata from the metadata service.

Specifically, the required metadata here is the target metadata, that is, cold data. The data extractor here can be a module that implements the data extraction function in the eMPP architecture.

Step S320: the cache data generator classifies the metadata according to the attributes of the data inside the metadata.

Specifically, the metadata is classified according to the attributes of the data within the single metadata. The cache data generator here can be a module that implements the cache data generation function in the eMPP architecture.

Step S330: The cache data generator pre-calculates metadata.

Specifically, the pre-calculation here includes scanning the extracted classified metadata and calculating the feature value according to the feature class defined by the system. The feature value here can be calculated by a hash algorithm, and the feature value here can be used to characterize the type of metadata.

Step S340: the cache data generator encodes the extracted metadata.

Specifically, the cache data generator performs binary encoding on the classified metadata and calculated feature values to ensure that they conform to the in-memory data structure. The data extractor passes the extracted data to the cache data generator for encoding to improve loading and query speeds.

Step S350: perform version verification on the metadata to generate version information.

Step S360: Pack the processed metadata to generate cache data.

In step S370, the cache data generator adds version information to the packaged cache data, generates a corresponding cache data image, and stores it in the cloud disk.

Specifically, the cache data generator adds version information to the packaged cache data, generates a cache data image from the cache data with added version information, and saves the generated cache data image in the cloud disk. Subsequently, the master node or metadata service triggers metadata access, and the loader reads the cache data image of the metadata from the cloud disk and puts it into the loader's local hard disk. The computing node reads data from the loader's local hard disk to the computing node's local hard disk. The loader's local hard disk and the computing node's local hard disk are in the same server. The computing node here is a stateless computing node.

In this embodiment, data extraction extracts the required metadata from the metadata service and caches it to the cloud disk. When the computing node needs to access the target metadata, the loader first loads the target metadata into the local memory of the loader. When the computing node needs to access the corresponding target metadata, the target metadata is then loaded into the local memory of the computing node, thereby reducing the access pressure of the metadata service and solving the problem that all nodes in the prior art need to access the metadata service, resulting in high metadata service load; the target metadata is classified, pre-calculated and encoded, which speeds up the subsequent query speed for cached data. The metadata is classified into cold and hot data, making offline caching possible. The offline cache is made into a data mirror and can be directly mounted through the operating system without the need for special hardware devices. The metadata is pre-calculated to improve the query speed of cached data. The query keywords of the metadata are hashed with corresponding multi-keywords, and the metadata is stored using a dedicated data structure. The dedicated data structure is designed for cached data in the memory, classified and stored, and the generated data structure after encoding. The metadata is specially encoded to improve the security and loading speed of the metadata. The full binary encoding can be directly loaded into the memory as a whole block. Data verification is added to the dedicated data structure to ensure the correctness of the data. That is, the metadata cache generation method provided by this implementation reduces the load of the metadata service, reduces the network transmission bandwidth required for metadata, reduces the latency of metadata queries, improves the overall performance of the database cluster, and increases the number of physical nodes that can be supported by the database cluster.

In one or more embodiments, a metadata cache access method is provided, as shown in FIG4 , the method includes the following steps:

Step S410, mounting the cached data image to the local environment.

Specifically, the system mounts the cache data image to the local environment, where the system can be an operating system of the eMPP architecture. The offline cache is made into a data image and can be directly mounted through the operating system without the need for special hardware devices.

Step S420: The loader reads the cached data image of the metadata.

Specifically, the loader loads the cached data image in the cloud disk into the loader's memory, that is, the loader The cached data image in the disk is loaded into the local hard disk of the loader. The loader first verifies the metadata version. After the version is correct, the loader verifies the metadata. The loader reads the binary file in the cloud disk and saves it in the loader's memory. The loader reads the metadata through the I/O link of the cloud disk, which does not occupy the network bandwidth and reduces the data access pressure of the system.

Step S430 , the loader connects the cache data in its memory to the local memory of the computing node.

Step S440: The computing node obtains the corresponding metadata from its local memory.

Specifically, when a computing node needs metadata, if the metadata has been loaded into the local memory of the computing node, the computing node can directly read the corresponding memory to obtain the corresponding metadata.

In this embodiment, when a computing node needs to access target metadata, the loader first loads the target metadata into the local memory of the loader. When the computing node needs to access the corresponding target metadata, the target metadata is then loaded into the local memory of the computing node, thereby reducing the access pressure of the metadata service and solving the problem of high metadata service load caused by all nodes needing to access the metadata service in the prior art. The cache is independently controlled by the loader, and only one memory copy is required for each physical environment, which greatly saves memory space (the number of computing nodes in each physical environment is >100).

In one or more embodiments, a computing node updating method is provided. As shown in FIG5 , the method includes the following steps:

Step S510, when the loader receives an instruction to add a new computing node, the loader connects to the memory address of the loader that caches the data.

Step S520, the loader notifies the computing node and connects the memory address of the loader that caches the data to the local memory of the new computing node.

Specifically, after the memory address of the cache data loader is connected to the local memory of the new computing node, the new computing node can work normally. When the computing node is destroyed, recycled or abnormally exited, its de-cached memory connection will be automatically disconnected by the operating system without additional processing.

In this embodiment, the creation, destruction and recycling of computing nodes do not require special processing of the cache and do not occupy any computing resources; the exit of the computing node in an uncontrollable state does not affect the cache itself and does not require special processing of the cache.

In one or more embodiments, a metadata cache dynamic update method is provided. As shown in FIG6 , the method includes the following steps:

Step S610: When the metadata is updated, the metadata service notifies the cache data generator, and the cache data generator generates a cache data image of the new metadata and notifies the loader.

Specifically, when the metadata service detects that there is a new version of metadata, the metadata service notifies the cache data generator, the cache data generator generates a cache data image of the new metadata, and notifies the loader.

Step S620: The loader re-reads the cache data image of the new metadata and generates a new metadata memory cache.

Specifically, the loader re-reads the cache data image of the new metadata from the cloud disk and generates a loader memory cache of the new metadata.

Step S630: The loader checks all computing nodes and updates them according to the status of the computing nodes.

Specifically, if the computing node still has tasks being executed, it will wait. When the computing node completes the current task, the loader notifies the computing node to disconnect the current metadata memory cache and reconnect to the new memory cache; if the computing node currently has no tasks or is about to execute a new task, it will notify the computing node to disconnect the current metadata memory cache and reconnect it to the new metadata memory cache. This allows dynamic updates of the metadata cache.

Step S640: After all computing nodes complete the cache connection update, the computing nodes delete their old memory caches and corresponding image files, and the loaders delete their old memory caches and corresponding image files.

In this embodiment, when the metadata is updated, the loader determines whether to connect to the loader memory cache of the new metadata according to the status of the computing node, thereby realizing dynamic update of the metadata cache, and the execution of the tasks of the computing node is not affected during the update process. The cache data image is dynamically updated with the metadata version. The cluster does not need to be shut down during the dynamic update, and the currently executing tasks are not affected. The cache data image is dynamically rolled over and switched, and the memory and disk space occupied by the old cache data image will be recovered in time.

In this embodiment, a metadata access device is also provided, which is used to implement the above embodiments and implementation methods, and the descriptions that have been made will not be repeated. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and/or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

FIG. 7 is a structural block diagram of a metadata access device according to an embodiment of the present application. As shown in FIG. 7 , the device includes:

The first loading module 710 is used to extract the target metadata from the metadata service through the data extractor and cache it to the cloud disk, and the loader loads the cached data of the target metadata from the cloud disk into the first memory of the loader; the target metadata is metadata in the metadata service whose data update frequency is lower than the first preset value and whose data access frequency is higher than the second preset value;

The access module 720 is used to load the cache data of the target metadata in the first memory into the second memory of the slave node corresponding to the master node in response to the metadata access request of the master node; the slave node reads the second memory to access the target metadata.

It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

Optionally, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

S1, loading cache data of target metadata into a first memory; the target metadata is metadata in a metadata service whose data update frequency is lower than a first preset value and whose data access frequency is higher than a second preset value;

S2, in response to the metadata access request of the master node, load the cache data of the target metadata in the first memory into the second memory of the slave node corresponding to the master node; the slave node reads the second memory to access the target metadata.

It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

In addition, in combination with a metadata access method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, the steps of any metadata access method in the above embodiment are implemented.

It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

A metadata access method, characterized in that the method comprises:

The target metadata is extracted from the metadata service by a data extractor and cached to a cloud disk, and the loader loads the cached data of the target metadata from the cloud disk to the first memory of the loader; the target metadata is metadata in the metadata service whose data update frequency is lower than a first preset value and whose data access frequency is higher than a second preset value;

In response to a metadata access request from the master node, the cache data of the target metadata in the first memory is loaded into a second memory of a slave node corresponding to the master node; the slave node reads the second memory to access the target metadata.
The metadata access method according to claim 1, wherein before loading the cache data of the target metadata into the first memory, the method further comprises:

Obtaining target metadata from the metadata service;

Generate cache data of the target metadata according to the acquired target metadata.
According to the metadata access method of claim 2, wherein generating cache data of the target metadata according to the acquired target metadata comprises:

Classifying the target metadata according to the data type of the target metadata to obtain type information;

Performing feature extraction on the target metadata;

Encoding the extracted features and the classified target metadata to obtain encoded data;

The cache data is generated according to the encoded data and the type information.
According to the metadata access method of claim 2, wherein generating cache data of the target metadata according to the acquired target metadata comprises:

The version information of the target metadata is obtained, and the cache data is generated according to the version information.
The metadata access method according to claim 1, wherein the method further comprises:

In response to a data update instruction sent by the metadata service, acquiring the execution status of the task of the slave node; the data update instruction is used to instruct to update the cache data of the target metadata stored in the first memory;

According to the execution state, cache data of the target metadata corresponding to the execution state is selected from the first memory and loaded into the second memory of the slave node.
According to the metadata access method of claim 5, wherein, according to the execution state, selecting cache data of the target metadata corresponding to the execution state from the first memory and loading it into the second memory of the slave node comprises:

When the execution state is that the slave node executes the task, cache data of the target metadata before the update is selected from the first memory and loaded into the second memory of the slave node;

When the execution state is that the slave node is idle, the updated cache data of the target metadata is selected from the first memory and loaded into the second memory of the slave node.
The metadata access method according to claim 1, wherein the method further comprises:

In response to an access request from a slave node, the address of the first memory is connected to the slave node to be accessed.
The metadata access method according to any one of claims 1 to 7, wherein the slave node is a stateless computing node.
A metadata access device, characterized in that the device comprises:

A first loading module is used to extract target metadata from a metadata service through a data extractor and cache the target metadata to a cloud disk, and the loader loads the cached data of the target metadata from the cloud disk to a first memory of the loader; the target metadata is metadata in the metadata service whose data update frequency is lower than a first preset value and whose data access frequency is higher than a second preset value;

An access module is used to load the cache data of the target metadata in the first memory into the second memory of the slave node corresponding to the master node in response to the metadata access request of the master node; the slave node reads the second memory to access the target metadata.
A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the metadata access method described in any one of claims 1 to 8 are implemented.