CN118277336A

CN118277336A - Metadata management method, metadata management device, and computer storage medium

Info

Publication number: CN118277336A
Application number: CN202410276203.7A
Authority: CN
Inventors: 徐涛; 施玉龙; 江文龙; 周明伟
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Filing date: 2024-03-11
Publication date: 2024-07-02

Abstract

The application provides a metadata management method, a metadata management device and a computer storage medium. The metadata management method is applied to a metadata node, and comprises the following steps: marking metadata information of the sensitive data and/or the high-level data as a deleting state in an image file in response to a deleting instruction of the sensitive data and/or the high-level data; acquiring a confirmation instruction of an administrator; when the confirmation instruction is a confirmation deleting instruction, deleting the metadata information marked as the deleting state in the image file; and when the confirmation instruction is a confirmation recovery instruction, recovering the metadata information marked as the deleting state in the image file to the storage state. By the metadata management method, the deleted data recovery is realized, and after the deleted data is ensured to be deleted, the metadata in the image file is completely deleted or recovered after the confirmation of a manager.

Description

Metadata management method, metadata management device, and computer storage medium

Technical Field

The present application relates to the field of distributed storage technologies, and in particular, to a metadata management method, a metadata management device, and a computer storage medium.

Background

In the cloud era, the data scale of a storage system is rapidly increased, metadata is introduced for more effectively managing and organizing the data in the storage system, the metadata comprises important functions such as data positioning, data control azimuth, data indexing and searching parts, space management and the like, and the distributed system can be ensured to provide more efficient and reliable data storage and access services. However, if the metadata is abnormal, no recovery means is provided, so that the data is read, lost, which is absolutely not allowed in the storage, and the data storage is disordered.

Disclosure of Invention

In order to solve the technical problems, the application provides a metadata management method, a metadata management device and a computer storage medium.

In order to solve the above technical problems, the present application provides a metadata management method, which is applied to a metadata node, and the metadata management method includes:

marking metadata information of the sensitive data and/or the high-level data as a deleting state in an image file in response to a deleting instruction of the sensitive data and/or the high-level data;

Acquiring a confirmation instruction of an administrator;

when the confirmation instruction is a confirmation deleting instruction, deleting the metadata information marked as the deleting state in the image file;

And when the confirmation instruction is a confirmation recovery instruction, recovering the metadata information marked as the deleting state in the image file to the storage state.

Wherein, the metadata management method further comprises:

Responding to the deleting instruction of the sensitive data and/or the high-level data, and acquiring a current metadata log;

analyzing the current metadata log, and judging whether the change condition of metadata exists or not;

If yes, triggering an alarm signal.

Wherein the marking the metadata information of the sensitive data and/or the high-level data as a deleted state in the image file comprises:

judging whether the metadata information exists in the current image file or not;

if yes, marking the metadata information as a deleting state in the current image file;

If not, the metadata information is stored as a new image file, and the metadata information is marked as a deleting state in the new image file.

Wherein, the metadata management method further comprises:

responding to the metadata management instruction, and comparing the changed metadata with preset metadata rules;

And deleting or repairing the changed metadata which does not accord with the preset metadata rule.

After the changed metadata is compared with a preset metadata rule, the metadata management method comprises the following steps:

comparing the changed metadata conforming to the preset metadata rule with metadata of other related data sources;

and deleting or repairing the changed metadata which is different from the metadata of the other related data sources in data format.

In order to solve the technical problem, the application also provides another metadata management method, which is applied to a metadata management system, wherein the metadata management system comprises a main metadata node and a standby metadata node; the metadata management method comprises the following steps:

the main metadata node records a first mark sequence number of the latest generation time point of the image file through a first metadata log;

The backup metadata node records a second mark sequence number of the latest generation time point of the image file through a second metadata log;

The master metadata node checks whether the first marking sequence number is the same as the second marking sequence number;

If not, the master metadata node sends a synchronous log record to the backup metadata node according to the first mark sequence number and the second mark sequence number;

And the backup metadata node synchronizes the latest image file of the main metadata node according to the synchronization log record.

The master metadata node sends a synchronization log record to the backup metadata node according to the first mark sequence number and the second mark sequence number, and the method comprises the following steps:

in response to the standby metadata node being abnormally restarted, the master metadata node writes the updated third marker sequence number into a first metadata log;

And after the standby metadata node is restarted, the main metadata node sends the synchronous log record to the standby metadata node according to the second mark sequence number and the third mark sequence number.

Wherein after the backup metadata node synchronizes the latest image file of the master metadata node according to the synchronization log record, the metadata management method further includes:

the master metadata node obtains the current resource utilization rate;

when the current resource utilization rate is larger than a resource utilization rate threshold value, the main metadata node and the standby metadata node perform metadata log synchronization according to a preset period;

When the current resource utilization rate is larger than a resource utilization rate threshold value, the main metadata node and the standby metadata node perform metadata log synchronization in real time

In order to solve the above technical problems, the present application further provides a metadata management device, where the metadata management device includes a memory and a processor coupled to the memory; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the metadata management method as described above.

In order to solve the above technical problem, the present application further provides a computer storage medium for storing program data, which when executed by a computer, is configured to implement the metadata management method described above.

Compared with the prior art, the application has the beneficial effects that: the metadata node responds to a deleting instruction of the sensitive data and/or the high-level data, and marks the sensitive data and/or the metadata information of the high-level data as a deleting state in the image file; acquiring a confirmation instruction of an administrator; when the confirmation instruction is a confirmation deleting instruction, deleting the metadata information marked as the deleting state in the image file; and when the confirmation instruction is a confirmation recovery instruction, recovering the metadata information marked as the deleting state in the image file to the storage state. By the metadata management method, the deleted data recovery is realized, and after the deleted data is ensured to be deleted, the metadata in the image file is completely deleted or recovered after the confirmation of a manager.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Wherein:

FIG. 1 is a flowchart illustrating an embodiment of a metadata management method according to the present application;

FIG. 2 is a schematic diagram of the overall flow of metadata deletion provided by the present application;

FIG. 3 is a flowchart illustrating another embodiment of a metadata management method according to the present application;

FIG. 4 is a schematic diagram of the overall flow of metadata verification provided by the present application;

FIG. 5 is a flowchart illustrating a metadata management method according to another embodiment of the present application;

FIG. 6 is a schematic diagram of a distributed storage system according to the present application;

FIG. 7 is a schematic diagram of a recovery process of a metadata image file according to the present application;

FIG. 8 is a schematic diagram of a metadata node backup recovery process provided by the present application;

FIG. 9 is a schematic diagram of an embodiment of a metadata management apparatus according to the present application;

Fig. 10 is a schematic structural diagram of an embodiment of a computer storage medium provided by the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The technical field of the application provides a method for realizing consistency of distributed main and standby metadata for a distributed storage cluster system, and the method can prevent the situation that the main and standby nodes are inconsistent in metadata under the abnormal conditions of power failure and the like of the metadata cluster. The application has sufficient application scenes in security scenes with higher data requirements because of the occurrence of data loss caused by inconsistent data.

Specifically, the metadata log subscription technology related to the application has the following properties:

1) The data of the main metadata and the standby metadata of the distributed file system keep consistent, and the consistency of the data can be restored under abnormal conditions.

2) The management and monitoring of the data can be used to monitor and manage changes in the data, such as writing of data, deleting actions.

3) For the safety and compliance of data, the access and operation of the data can be monitored, the compliance with regulations and safety strategies can be ensured, and the potential safety threat and leakage can be timely detected.

4) Automated tasks and workflows, through metadata log subscription techniques, may be used to trigger execution of tasks and workflows.

5) Metadata changes of the data sources can be captured through metadata log subscription technology, and whether the data sources change or not can be timely found, so that consistency of metadata is guaranteed.

Referring to fig. 1 and fig. 2, fig. 1 is a flow chart of an embodiment of a metadata management method according to the present application, and fig. 2 is an overall flow chart of metadata deletion according to the present application.

The metadata management method is applied to the metadata management device, wherein the metadata management device can be a server, terminal equipment or a system formed by mutually matching the server and the terminal equipment. Accordingly, each part, for example, each unit, sub-unit, module, and sub-module, included in the metadata management apparatus may be all provided in the server, may be all provided in the terminal device, or may be provided in the server and the terminal device, respectively.

Further, the server may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules, for example, software or software modules for providing a distributed server, or may be implemented as a single software or software module, which is not specifically limited herein.

Specifically, the metadata management method of the present application is applied to metadata nodes in a metadata management system, including but not limited to: a primary data node and a backup data node, etc.

As shown in fig. 1, the specific steps are as follows:

Step S11: responsive to a deletion instruction of the sensitive data, and/or the high-level data, the sensitive data, and/or metadata information of the high-level data is marked as a deleted state in the image file.

In the embodiment of the application, the metadata node can monitor and record the access and use behaviors of the data by analyzing editlog logs, namely metadata logs, through the metadata log subscription technology, and can automatically limit or prohibit the access of sensitive data. When deleting or modifying data marked as sensitive or high-level, a corresponding alarm can be triggered.

As shown in fig. 2, when sensitive data or high-level data in a data node is deleted, the metadata node acquires that a corresponding editlog log has access record or modifies record through a metadata log subscription technology, and a corresponding image file, that is, an image file has change. At this time, the metadata node triggers an alarm signal, and sends the alarm information to an administrator through mail, short message, and the like, so that the data is prevented from being deleted by mistake.

Further, in order to prevent the metadata node from being deleted by mistake, after the metadata node is updated in the editlog log, the corresponding metadata information in the image file may be marked as a deleted state, that is, a delete state.

Specifically, if the deleted data is data that has been recorded in the image file at this time, the corresponding metadata information in the image file may be directly marked in the delete state. If the deleted data are the data which are not recorded in the image file at the moment, storing metadata information corresponding to the data in a memory in the form of the image file; at this time, the occupied memory is not large, and therefore, the actual performance is not affected.

Step S12: and acquiring a confirmation instruction of the administrator.

In the embodiment of the application, when updating the image file, the metadata node outputs the metadata information marked as the delete state according to the image file in the memory to wait for the secondary confirmation of the administrator.

Step S13: and deleting the metadata information marked as the deleting state in the image file when the confirmation instruction is a confirmation deleting instruction.

In the embodiment of the present application, if the confirmation instruction input by the administrator is a confirmation deletion instruction, the metadata node deletes all delete state metadata information in the image when the image file is updated next time.

Step S14: and when the confirmation instruction is a confirmation recovery instruction, recovering the metadata information marked as the deleted state in the image file to the storage state.

In the embodiment of the application, if the confirmation instruction input by the administrator is a confirmation deleting instruction, when the image file is updated next time, the metadata node restores all delete state metadata information in the image to a storage state, and restores the deleted sensitive data and the deleted high-level data.

In the embodiment of the application, a metadata node responds to a deletion instruction of sensitive data and/or high-level data, and marks the sensitive data and/or metadata information of the high-level data as a deletion state in an image file; acquiring a confirmation instruction of an administrator; when the confirmation instruction is a confirmation deleting instruction, deleting the metadata information marked as the deleting state in the image file; and when the confirmation instruction is a confirmation recovery instruction, recovering the metadata information marked as the deleting state in the image file to the storage state. By the metadata management method, the deleted data recovery is realized, and after the deleted data is ensured to be deleted, the metadata in the image file is completely deleted or recovered after the confirmation of a manager.

Further, please continue to refer to fig. 3 and fig. 4, fig. 3 is a flowchart illustrating another embodiment of the metadata management method according to the present application, and fig. 4 is an overall flowchart illustrating metadata verification according to the present application.

As shown in fig. 3, the specific steps are as follows:

step S21: and responding to the metadata management instruction, and comparing the changed metadata with preset metadata rules.

In the embodiment of the application, the metadata node records the related operation information of the metadata in editlog logs through a metadata log subscription technology. When the deleting operation or the updating operation of the metadata occurs, the metadata of the operation is compared with a preset metadata rule, and whether the changed metadata meets the specification is verified. Specifically, the preset metadata rule according to the present application is a data format specification about metadata, such as a data length, a data sequence, a data text format, a data compression format, etc., which is imported by an administrator.

Step S22: and deleting or repairing the changed metadata which does not accord with the preset metadata rule.

In the embodiment of the application, when the metadata node finds that the change of the metadata violates the preset metadata rule, the metadata node can repair or delete the metadata according to editlog logs of the metadata. The repairing refers to compliance adjustment of the changed metadata according to preset metadata rules.

Step S23: and comparing the changed metadata conforming to the preset metadata rule with metadata of other related data sources.

In the embodiment of the application, the metadata node further checks whether the metadata conforming to the preset metadata rule is consistent with the metadata of other related data sources.

Specifically, after checking that metadata accords with a preset metadata rule, the metadata node also needs to ensure that the changed metadata is consistent with metadata of other related data sources, so that metadata management between the related data sources is consistent. Wherein, the coincidence at least includes the coincidence of the data formats of the metadata.

Step S24: and deleting or repairing the changed metadata which is different from the metadata of other related data sources in data format.

In the embodiment of the application, when the metadata node finds that the change of the metadata is inconsistent with the metadata of other related data sources, the metadata node can repair or delete the metadata according to editlog logs of the metadata. Wherein, repairing refers to compliance adjustment of changed metadata according to metadata formats of other related data sources.

The application further provides a metadata management method based on the metadata management system based on the single-node metadata management method, and particularly referring to fig. 5 to 7. Fig. 5 is a flow chart of a further embodiment of the metadata management method provided by the present application, fig. 6 is a schematic structural diagram of a distributed storage system provided by the present application, and fig. 7 is a schematic recovery flow chart of a metadata image file provided by the present application.

As shown in fig. 6, the distributed storage system according to the present application is composed of a primary metadata node, a backup metadata node, and a plurality of data child nodes. Wherein Mds in fig. 6 is a main metadata node, mds is a standby metadata node, and datanodes 1 to DataNodeN are data child nodes.

The metadata nodes are responsible for managing metadata correlation of the data nodes, and a log subscription technology is introduced in metadata management. Specifically, the metadata node will make the metadata content of the current metadata node into an image file, where the image file stores the namespaces, directories and all metadata information related to the current file system. The image file represents the state of the metadata in the cluster at the current time point, and the metadata can be ensured to be restored under the condition that the metadata is abnormal through the image file.

As shown in fig. 5, the specific steps are as follows:

Step S31: the master metadata node records a first mark sequence number of the latest generation time point of the image file through a first metadata log.

Step S32: the backup metadata node records a second mark sequence number of the latest generation time point of the image file through a second metadata log.

In the embodiment of the application, the main metadata node and the standby metadata node record various operations on metadata through editlog logs, including deletion and addition, and the log also records XID of the last generation time point of the image file, namely the linear marking serial number when the last image file is generated.

Step S33: the master metadata node checks whether the first tag sequence number is the same as the second tag sequence number.

In the embodiment of the present application, as shown in fig. 7, the metadata node of the present application determines, according to a protocol, whether metadata of the primary and secondary metadata nodes are consistent through log subscription.

Specifically, the master node and the slave node of the metadata determine whether the image files are consistent through XID, if the two image files are inconsistent, the image files are synchronized in full, so that the image files of the master node and the slave node are synchronized, and step S34 is performed.

Step S34: and the master metadata node sends the synchronous log record to the standby metadata node according to the first mark sequence number and the second mark sequence number.

In the embodiment of the present application, the master metadata node obtains all editlog of the master metadata node editlog after XID according to the last XID record of the master backup node, i.e. obtains the synchronization log record between the first mark sequence number and the second mark sequence number. The master metadata node reads back the records and sends them to the backup metadata node, which will check the data, append the check value to the data tail, and transmit token encryption to the backup metadata node.

Step S35: and the backup metadata node records the latest image file of the synchronous master metadata node according to the synchronous log.

In the embodiment of the application, the backup metadata node firstly decrypts the token, reads the data after the token is matched with the token, compares the check values, requests to be sent again if the check values are inconsistent, and if the check values are consistent with the check values, the backup metadata node recovers the metadata through the logs and synchronizes the image file.

Further, the embodiment of the application also relates to a special restarting scenario of the backup metadata node, specifically referring to fig. 8, fig. 8 is a schematic flow chart of the backup recovery of the metadata node provided by the application.

As shown in fig. 8, in the case where the XID of the spare metadata node does not agree with the XID of the main metadata node, the main metadata node first reads editlog behind the spare metadata node XID back to the spare metadata node. And the backup metadata node receives the log content, performs metadata operation and supplements corresponding data information.

When the backup metadata node recovers, the main metadata node puts the newly written log into the memory at the moment, and the main metadata node synchronously writes the log. If the standby metadata node is restarted again, the master metadata node will immediately write the logs in all the memories into the real-time editlog, and continue writing in editlog until the standby metadata node startup service is normal.

In addition, the metadata image file recovery flow in fig. 7 also involves primary and backup metadata node synchronization, where the metadata node automatically selects real-time synchronization or cycle synchronization according to the traffic.

Specifically, when the data of the metadata master node and the backup metadata master node are consistent, an adaptive synchronization strategy is adopted. Specifically, a threshold value of the resource utilization rate is set according to the load condition of the service, when the resource utilization rate is greater than the threshold value, the current service is considered to be in a busy state, and then a periodic data synchronization strategy is used, that is, by setting the period of the timing, the synchronization is performed on editlog periodically. When the resource usage is not busy, then real-time synchronization of primary and backup data and editlog is performed.

When switching to the periodic synchronization strategy, firstly recording the real-time synchronous XID, if the real-time synchronization is not completed, recording after waiting for completion, and synchronizing the last XID after the XID to the master node after the subsequent synchronization, thereby finally realizing the purpose of ensuring the data consistency of the master node and the slave node of the metadata.

The application aims to ensure consistency of metadata. Through cooperation of the images and editlog, recovery of the metadata can be guaranteed under the condition that the metadata of the nodes are abnormal.

According to the method and the device, through the images and editlog, when the primary and the secondary data are inconsistent due to various abnormal reasons, the primary and the secondary metadata can be quickly restored to be consistent, and the fault tolerance of the metadata cluster is improved.

When the application writes, reads and deletes the data, editlog records the corresponding operation, and can be used for monitoring and managing the change of the data.

The application can recover the data to be consistent rapidly without affecting the performance of the cluster by adaptively recovering the metadata of the standby node, and finally realizes the consistency of the data.

The application can ensure that the data accords with regulations and security policies by monitoring and managing the access and change of the data, and ensure that the data accords with the reservation standard after the change, and can timely detect potential security threats and leakage.

The application can send alarm in time when the sensitive data is accessed, and when the sensitive data is operated, the metadata is deleted, but the metadata is not deleted in the image, the image is set to delete state, and the sensitive data can be deleted after being confirmed by a manager and can be recovered, thereby ensuring the safety of the data.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

In order to implement the metadata management method, the present application further provides a metadata management device, and referring to fig. 9 specifically, fig. 9 is a schematic structural diagram of an embodiment of the metadata management device provided by the present application.

The metadata management apparatus 400 of the present embodiment includes a processor 41, a memory 42, an input-output device 43, and a bus 44.

The processor 41, the memory 42, and the input/output device 43 are respectively connected to the bus 44, and the memory 42 stores program data, and the processor 41 is configured to execute the program data to implement the metadata management method described in the above embodiment.

In an embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit ). The processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 may also be a general purpose processor, a digital signal processor (DSP, digital Signal Process), an Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA, field Programmable GATE ARRAY) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The general purpose processor may be a microprocessor or the processor 41 may be any conventional processor or the like.

The present application further provides a computer storage medium, please continue to refer to fig. 10, fig. 10 is a schematic structural diagram of an embodiment of the computer storage medium provided by the present application, in which a computer program 61 is stored in the computer storage medium 600, and the computer program 61 is used to implement the metadata management method of the above embodiment when being executed by a processor.

Embodiments of the present application may be stored in a computer readable storage medium when implemented in the form of software functional units and sold or used as a stand alone product. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing description is only of embodiments of the present application, and is not intended to limit the scope of the application, and all equivalent structures or equivalent processes using the descriptions and the drawings of the present application or directly or indirectly applied to other related technical fields are included in the scope of the present application.

Claims

1.A metadata management method, wherein the metadata management method is applied to a metadata node, the metadata management method comprising:

Acquiring a confirmation instruction of an administrator;

2. The method for metadata management as recited in claim 1, wherein,

The metadata management method further includes:

If yes, triggering an alarm signal.

3. The method for metadata management as recited in claim 1, wherein,

The marking the metadata information of the sensitive data and/or the high-level data as a deleted state in the image file comprises the following steps:

4. The method for metadata management as recited in claim 1, wherein,

The metadata management method further includes:

5. The method for metadata management as recited in claim 4, wherein,

After comparing the changed metadata with a preset metadata rule, the metadata management method comprises the following steps:

6. A metadata management method, wherein the metadata management method is applied to a metadata management system, and the metadata management system comprises a main metadata node and a standby metadata node; the metadata management method comprises the following steps:

7. The method for metadata management as recited in claim 6, wherein,

8. The method for metadata management as recited in claim 6, wherein,

After the backup metadata node synchronizes the latest image file of the master metadata node according to the synchronization log record, the metadata management method further includes:

the master metadata node obtains the current resource utilization rate;

And when the current resource utilization rate is larger than a resource utilization rate threshold value, the main metadata node and the standby metadata node perform metadata log synchronization in real time.

9. A metadata management apparatus, wherein the metadata management apparatus comprises a memory and a processor coupled to the memory;

Wherein the memory is for storing program data and the processor is for executing the program data to implement the metadata management method of any one of claims 1 to 8.

10. A computer storage medium for storing program data which, when executed by a computer, is adapted to carry out the metadata management method of any one of claims 1 to 8.