WO2022042514A1

WO2022042514A1 - Metadata synchronization method and apparatus

Info

Publication number: WO2022042514A1
Application number: PCT/CN2021/114190
Authority: WO
Inventors: 德语心; 彭勇; 王英; 李浩然; 冯阳
Original assignee: 华为技术有限公司
Priority date: 2020-08-31
Filing date: 2021-08-24
Publication date: 2022-03-03
Also published as: CN114116641A

Abstract

A metadata synchronization method and apparatus, which belong to the technical field of data storage. Said method comprises: determining that a triggering event for synchronizing metadata occurs (201); acquiring at least one operation log to be processed from stored operation logs, each operation log recording file identifiers corresponding to files operated by a file system, parent directory identifiers corresponding to the files and metadata of the files generated after the file system executes operations on the files, and the metadata comprising storage address information of the files in the file system (202); and sending, to an object system, file identifiers, parent directory identifiers and metadata which are corresponding to files corresponding to the at least one operation log to be processed (203), so that the object system stores the received file identifiers, parent directory identifiers and metadata of the files correspondingly (204). Said method can save storage resources of an object system.

Description

Method and apparatus for metadata synchronization

technical field

The present application relates to the technical field of data storage, and in particular, to a method and device for synchronization of metadata.

Background technique

There are generally two types of data storage systems: file system and object system. File systems and object systems can be oriented to different users. In addition, the file system and the object system need to provide the same file service to the user, that is, a certain file can be read through the file system or through the object system.

Currently, in order for a certain file to be read through the file system or through the object system, it is usually necessary for the technician to migrate the file in the file system to the object system every time period. After each migration, the same files can be maintained in the file system and object system.

In the process of realizing this application, the inventor found that the related art has at least the following problems:

The contents of the files stored in the file system and the object system are the same, but both the storage space of the file system and the storage space of the object system are occupied, which is a waste of storage resources.

SUMMARY OF THE INVENTION

In order to solve the problem that the object system needs to store all files in the related art, which leads to the waste of storage resources, the embodiments of the present application provide a method and apparatus for synchronizing metadata. The technical solution is as follows:

In a first aspect, a metadata synchronization method is provided, the method is applied to a file system, and the method includes:

It is determined that a trigger event for metadata synchronization occurs, and at least one pending operation log is obtained from the stored operation log, wherein the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and the The metadata of the file generated after the file system performs an operation on the file, the metadata includes the storage address information of the file in the file system, and the file identifier and parent directory corresponding to each file corresponding to at least one operation log to be processed are sent to the target system. ID and metadata, so that the object system stores the file ID, parent directory ID, and metadata of each file received correspondingly.

In the solution shown in the embodiment of the present application, the file system may store an operation log, and the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and after the file system performs an operation on the file Metadata for the generated file. When a trigger event for metadata synchronization occurs, the file system can send the file identifier, parent directory identifier, and metadata recorded in the pending operation log to the object system, and the object system can compare the file identifier, parent directory identifier, metadata, and metadata of the file to the object system. Data is stored accordingly. The metadata includes the storage address information of the file in the file system. In this way, the object system does not need to store the file itself. When the user has a file acquisition requirement, he only needs to acquire the file according to the storage address information of the stored file. , which effectively saves the storage resources of the object system.

In a possible implementation, after obtaining at least one operation log to be processed in the stored operation log, the method further includes:

Delete the stored pending operation log.

In the solution shown in the embodiment of the present application, after the operation log to be processed is obtained, it can be deleted from the disk, so as to save storage space and avoid repeated acquisition of an operation log.

In a possible implementation manner, the determining that a trigger event for metadata synchronization occurs includes:

Make sure the file system is restarted;

The obtaining at least one pending operation log that satisfies the processing condition in the stored operation log includes:

Get all operation logs stored as pending operation logs.

In a possible implementation manner, the operation log further records the operation time of the file system on the file;

The determining that a trigger event for metadata synchronization occurs includes:

receiving a metadata synchronization request sent by the terminal, wherein the metadata synchronization request carries the synchronization time;

The obtaining at least one pending operation log in the stored operation log includes:

In the stored operation logs, all operation logs whose recorded operation time is before the synchronization time are acquired as pending operation logs.

In the solution shown in the embodiment of the present application, the metadata in the object system can be synchronized to a specified time according to user requirements.

It is determined that the preset metadata synchronization period is reached;

Get all operation logs stored as pending operation logs.

In the solution shown in the embodiment of the present application, the operation logs to be processed can be obtained periodically, so that the operation logs will not accumulate too much, and the metadata in the file system and the object system will not differ too much.

In a possible implementation, the method further includes:

After the first file is operated, a corresponding first operation log is generated, the parent directory identifier of the first file recorded in the first operation log is obtained, and the first bloom filter corresponding to the current storage period is obtained, recording the parent directory identifier of the first file in the first bloom filter, and storing the first file in the first operation log storage area corresponding to the first bloom filter;

Receive a metadata synchronization request for the second file sent by the target system, where the metadata synchronization request carries the parent directory identifier of the second file

Determine the second bloom filter that records the parent directory identifier of the second file, and obtain the operation log stored in the second operation log storage area corresponding to the second bloom filter and other storage areas in the second bloom filter. The operation log stored before the operation log stored in the operation log storage area is used as the pending operation log.

In the solution shown in the embodiment of the present application, the file system may use a Bloom filter to record the parent directory identifier in the stored operation log. In this way, when the target system has a synchronization requirement, the metadata in all the stored operation logs may not be synchronized, but only the metadata in part of the operation logs can be synchronized, which can effectively save bandwidth.

In a possible implementation manner, the acquiring operation logs stored in the second operation log storage area corresponding to the second bloom filter and the operation logs stored in the second operation log storage area in other storage areas After the operation log stored before the operation log is used as the pending operation log, the method further includes:

All the values of the arrays in the Bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed are set to 0.

In the solution shown in the embodiment of the present application, after the operation log is obtained, all the values of the array in the bloom filter corresponding to the corresponding operation log storage area can be set to 0, so that the bloom filter can be used cyclically in the future .

In a possible implementation, the method further includes:

receiving a read request for a third file sent by the target system, wherein the read request carries storage address information of the third file;

Obtain the third file according to the storage address information of the third file, and send the third file to the target system.

In the solution shown in the embodiment of the present application, the object system may query the metadata of the target file to determine the storage address of the target file, and then obtain the target file from the file system.

In a second aspect, a metadata synchronization method is provided, the method is applied to an object system, and the method includes:

receiving the file identifier, parent directory identifier and metadata corresponding to the file sent by the file system, wherein the metadata includes storage address information of the file in the file system;

The received file identifier, parent directory identifier, and metadata of the file are stored correspondingly.

In a possible implementation, the method further includes:

receiving the first read request of the target file sent by the terminal, where the first read request carries the file identifier and the parent directory identifier of the target file;

obtain metadata of the target file;

sending a second read request for the target file to the file system, where the second read request carries the storage address information of the target file;

receiving the target file returned by the file system;

Return the target file to the terminal.

In a third aspect, an apparatus for metadata storage and synchronization is provided, the apparatus is applied to a file system, and the apparatus includes:

A determination module, used to determine the occurrence of a trigger event for metadata synchronization;

The acquisition module is used to acquire at least one pending operation log in the stored operation log, wherein the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file, the metadata includes storage address information of the file in the file system;

A sending module, configured to send the file identifier, parent directory identifier and metadata corresponding to each file corresponding to the at least one operation log to be processed to the object system, so that the object system can identify the received file identifier of each file , parent directory ID, and metadata are stored accordingly.

In a possible implementation, the device also includes a deletion module for:

Delete the stored operation log to be processed.

In a possible implementation manner, the determining module is used for:

determine that the file system is restarted;

The acquisition module is used for:

Get all operation logs stored as pending operation logs.

The determining module is used for:

receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries a synchronization time;

The acquisition module is used for:

In a possible implementation manner, the determining module is used for:

It is determined that the preset metadata synchronization period is reached;

The acquisition module is used for:

Get all operation logs stored as pending operation logs.

In a possible implementation manner, the apparatus further includes a generating module for:

After the first file is operated, a corresponding first operation log is generated;

Obtain the parent directory identifier of the first file recorded in the first operation log;

Obtain the first bloom filter corresponding to the current storage period, and record the parent directory identifier of the first file in the first bloom filter;

storing the first file in the first operation log storage area corresponding to the first bloom filter;

The determining module is used for:

receiving a metadata synchronization request for the second file sent by the object system, wherein the metadata synchronization request carries the parent directory identifier of the second file;

The acquisition module is used for:

determining a second bloom filter that records the parent directory identifier of the second file;

Obtain the operation log stored in the second operation log storage area corresponding to the second bloom filter and the operation log stored before the operation log stored in the second operation log storage area in other storage areas, as pending processing Operation log.

In a possible implementation, the apparatus further includes:

The setting module is configured to set all the values in the Bloom filter corresponding to the storage area corresponding to the operation log to be processed to 0.

In a possible implementation manner, the apparatus further includes a receiving module for:

obtaining the third file according to the storage address information of the third file;

The third file is sent to the subject system.

In a fourth aspect, an apparatus for metadata synchronization is provided, the apparatus is applied to an object system, and the apparatus includes a receiving module for:

The storage module is used for correspondingly storing the received file identifier, parent directory identifier and metadata of the file.

In a possible implementation manner, the apparatus further includes a reading module for:

obtain metadata of the target file;

Send the second read request to the target file to the file system, and carry the storage address information of the target file in the second read request;

receiving the target file returned by the file system;

Return the target file to the terminal.

In a fifth aspect, a file system is provided, the file system includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to achieve the above-mentioned first The method of metadata synchronization described in the aspect.

In a sixth aspect, an object system, the object system includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the second aspect above. method for metadata synchronization described above.

In a seventh aspect, a computer-readable storage medium is provided, comprising instructions, when the computer-readable storage medium runs on a file system, the file system is caused to perform the method for synchronizing metadata in the first aspect.

An eighth aspect, a computer-readable storage medium, characterized by comprising instructions, when the computer-readable storage medium runs on an object system, causing the file system to perform the metadata synchronization described in the second aspect above Methods.

The beneficial effects brought by the technical solutions provided in the embodiments of the present application are:

In the embodiment of the present application, the file system may store an operation log, and the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and the metadata of the file generated after the file system performs an operation on the file. data. When a trigger event for metadata synchronization occurs, the file system can send the file identifier, parent directory identifier, and metadata of the file recorded in the operation log to be processed to the object system, and the object system can verify the file identifier and parent directory identifier of the file. , metadata corresponding to storage. The metadata includes the storage address information of the file in the file system. In this way, the object system does not need to store the file itself. When the user has a file acquisition requirement, he only needs to acquire the file according to the storage address information of the stored file. , which effectively saves the storage resources of the object system.

Description of drawings

FIG. 1 is an implementation scenario diagram provided by an embodiment of the present application;

2 is a flowchart of a method for synchronizing metadata provided by an embodiment of the present application;

3 is a flowchart of a method for reading a file provided by an embodiment of the present application;

4 is a flowchart of a method for prefix query provided by an embodiment of the present application;

5 is a schematic structural diagram of an apparatus for metadata synchronization provided by an embodiment of the present application;

6 is a schematic structural diagram of an apparatus for metadata synchronization provided by an embodiment of the present application;

7 is a schematic structural diagram of a file system provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an object system provided by an embodiment of the present application.

detailed description

The embodiment of the present application provides a method for synchronizing metadata, and the method can be jointly implemented by a file system and an object system. Referring to the implementation scenario diagram shown in FIG. 1 , in this embodiment of the present application, the file system will generate a corresponding operation log when performing an operation on a file, and store the operation log. That is, in the embodiment of the present application, the file system may store operation logs in addition to files and metadata. The metadata of the file after the operation is completed can be recorded in the operation log. In addition, through various trigger events, the file system is triggered to write the metadata in the operation log into the object system, so that the metadata in the object system and the file system are consistent.

The following describes the processing flow of the metadata synchronization method provided by the embodiment of the present application with reference to FIG. 1 :

Step 201: The file system determines a trigger event for metadata synchronization to occur.

In the implementation, the user can operate the specified file in the file system through the terminal. After the file system performs operations on the files, the corresponding operation logs can be generated and stored. For example, the generated operation log can be stored on disk. The operation log may record the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and the metadata of the file generated after the file system performs an operation on the file. Metadata can record file attribute information, such as creation time, modification time, owner, read and write permissions, and storage address information. The storage address information indicates the storage location of the file in the file system.

In addition, the operation type and the like may also be recorded in the operation log. The operation type may include create, delete, modify, and so on.

In order to keep the metadata of the files stored in the object system consistent with the metadata of the files stored in the file system, you can set a trigger event for metadata synchronization. Data is synchronized to the target system. Specifically, there may be various trigger events for metadata synchronization. Several possible trigger events are listed below.

1. A file system restart or failure recovery is detected.

2. The preset metadata synchronization period is reached.

3. Receive a metadata synchronization request sent by the terminal.

In this case, the user can directly send a metadata synchronization request to the file system through the terminal, so as to trigger the file system to perform metadata synchronization processing.

4. Receive the metadata synchronization request sent by the target system.

In this case, when a user queries a specified file through the object system, the object system may first send a metadata synchronization request to the file system, so as to trigger the file system to perform metadata synchronization processing.

Step 202: Acquire at least one pending operation log from the stored operation log.

In implementation, whenever a trigger event for metadata synchronization is sent, all stored operation logs may be used as pending operation logs.

In a possible implementation manner, when the operation log is stored, the operation log may be recorded in combination with a Bloom Filter. Therefore, when acquiring the log to be processed, it is not necessary to acquire all the stored operation logs. Correspondingly, the specific processing can be as follows:

In order to facilitate understanding, the following is a brief description of the use of BloomFilter:

BloomFilter is essentially a data structure consisting of a very long array and multiple hash functions. When an element needs to be recorded in the BloomFilter, each hash function in the BloomFilter can be used to calculate a hash value of the element. Then, the corresponding position in the BloomFilter array can be disposed of by 1. When you need to query whether an element is recorded in the BloomFilter, you must first use each hash function in the BloomFilter to calculate a hash value of the element, and then query the BloomFilter array with the hash value. Whether the corresponding position is 1, if it is all 1, it means that the BloomFilter may have recorded the element, if it is not all 1, it means that the BloomFilter must not have recorded the element.

The file system can store multiple BloomFilters in memory, and BloomFilters are used to record the parent directory identifier recorded in the operation log. Here, the parent directory identifier of the file can be considered as the element in the above-mentioned brief description of the usage method of BloomFilter.

Each BloomFilter corresponds to an operation log store on disk. Each BloomFilter corresponds to an operation log write cycle. Each operation log writing period can be tens of seconds or several minutes, and the operation log writing period can be set according to actual requirements.

It should be noted that these BloomFilters can be used cyclically, that is, after the operation log writing cycle corresponding to the last BloomFilter ends, it is possible to switch to the first BloomFilter to continue to use, so as to cycle.

The following describes the process of writing the operation log into the operation log storage area, taking N BloomFilters stored as an example:

When the file system is started, the operation log writing cycle starts to time, and the operation log generated in the first operation log writing cycle obtains the parent directory identifier recorded in it, and records it in the first BloomFilter, the specific record The method is as described in the above-mentioned brief description of the usage method of BloomFilter, and will not be repeated here. At the same time, the operation log is written into the operation log storage area corresponding to the first BloomFilter. It should also be noted here that, in each operation log storage area, it needs to be stored in the order in which it is written.

For the operation log generated in the Nth operation log writing cycle, obtain the parent directory ID recorded in it, and record it in the Nth BloomFilter. At the same time, the operation log is written into the operation log storage area corresponding to the Nth BloomFilter.

Next, for the operation log generated in the N+1th operation log writing cycle, obtain the parent directory ID recorded in it, generate its hash value, and record it in the first BloomFilter. At the same time, the operation log is written into the operation log storage area corresponding to the first BloomFilter.

Corresponding to several trigger events in the above step 101, the specific implementation of obtaining the pending operation log in the stored operation log according to each trigger event is described here:

Corresponding to the first trigger event above:

When obtaining pending operation logs, you can obtain all stored operation logs as pending operation logs.

Corresponding to the above-mentioned second trigger event:

In this case, when obtaining the pending operation log, the following implementations can be implemented:

Implementation 1. Obtain all stored operation logs as pending operation logs.

Implementation 2: Obtain the first preset number of operation logs written in the operation log writing cycle as the pending operation logs.

In the second implementation, a first storage threshold and a second storage threshold may also be set. The first storage threshold is greater than the second storage threshold. When the preset metadata synchronization period is reached, if it is detected that the operation log stored in the disk is greater than For the first storage threshold, the above-mentioned preset number may be increased, for example, 1 is added to the above-mentioned preset number. When the preset metadata synchronization period is reached, if it is detected that the operation log stored in the disk is smaller than the second storage threshold, the number of metadata synchronization periods may be adjusted to the above-mentioned preset number.

Corresponds to the third trigger event above:

The metadata synchronization request sent by the terminal may carry the synchronization time, and the synchronization time indicates that the user needs to synchronize the metadata in the target system to this time. In addition, the operation time of the file may be recorded in the operation log, and the operation time of the file is the time when the file system performs an operation on the file. Correspondingly, when acquiring the pending operation logs, the file system may use all the operation logs whose recorded operation time is before the above synchronization time as the pending operation logs.

Alternatively, the operation time of the file is not recorded in the operation log, but the file system can record the writing time of the operation log when storing the operation log. The writing time is the time when the file system writes the operation log to the disk. All operation logs whose writing time is before the above synchronization time are regarded as pending operation logs.

Corresponding to the above fourth trigger event:

After receiving a user's read request for a file, the object system can send a metadata synchronization request for the file to the file system, and the metadata synchronization request carries the parent directory identifier of the file. The file system obtains the parent directory identifier carried in the metadata synchronization request, generates a hash value of the parent directory identifier, and determines which BloomFilter records the parent directory identifier according to the hash value. The specific determination method is as described in the above-mentioned brief description of the usage method of BloomFilter, which is not repeated here.

If only one BloomFilter is determined, the operation log in the operation log storage area corresponding to the BloomFilter and the operation logs of other operation log storage areas whose writing time is before the operation log in the operation log storage area are regarded as pending processing. Operation log.

If multiple BloomFilters are determined at the same time, the corresponding operation logs are written to the target operation log in the operation log storage area corresponding to the BloomFilter with the last cycle, and other operation log storage areas whose writing time is before these target operation logs The operation log in , as the pending operation log.

In this way, the amount of metadata to be synchronized at one time can be reduced, and the problem of excessive bandwidth occupation caused by synchronizing multiple pieces of data at the same time can be avoided as much as possible.

Step 203: The file system sends the file identifier, parent directory identifier, and metadata corresponding to each file corresponding to at least one operation log to be processed to the target system.

In implementation, after the operation log to be processed is acquired, the file identifier, parent directory identifier and metadata recorded therein can be extracted, and sent to the target system for synchronous storage. At the same time, the file system can also delete the pending operation log from the disk, and set each value in the corresponding BloomFilter array to 0.

In addition, in order to save bandwidth, for the metadata in the operation log with the same parent directory ID and file ID, only the last generated metadata can be taken and sent to the target system for synchronous storage.

Step 204: The object system stores correspondingly the file identifier, parent directory identifier and metadata of each file received.

In implementation, the object system may store correspondingly the file identifier, parent directory identifier and metadata of each file received. That is, the "parent directory identifier + file identifier" is used as an index, and the corresponding metadata is used as storage data.

Here, in the case where the file system sends the metadata in all pending operation logs to the object system, the object system can perform overwrite storage, that is, for the metadata with the same parent directory ID and file ID, use the metadata received later. The metadata overwrites the previously received metadata.

In a possible implementation manner, the user can read the target file through the object system, and the corresponding processing can be as follows:

S301. The terminal sends a first read request for the target file to the target system, where the first read request carries the parent directory identifier of the target file and the file identifier of the target file.

S302, the object system may send a metadata synchronization request to the target file to the file system, where the metadata synchronization request carries the parent directory identifier of the target file.

S303, the file system acquires the pending operation log according to the parent directory identifier of the target file.

Here, the file system may obtain the pending operation log according to the method for obtaining the pending operation log corresponding to the fourth trigger event in the foregoing step 102 .

S304, the file system sends the parent directory identifier, file identifier and metadata recorded in the operation log to be processed to the target system.

S305. The object system stores the received parent directory identifier, file identifier and metadata correspondingly, and obtains the metadata of the target file according to the parent directory identifier and file identifier of the target file.

S306. The object system sends a second read request for the target file to the file system, where the second read request carries the storage address information of the target file.

S307, the file system acquires the target file according to the storage address information of the target file.

S308, the file system sends the target file to the target system.

S309, the target system returns the target file to the terminal.

In another possible implementation manner, the user can query the object system for which files satisfy a certain prefix. Correspondingly, the processing can be as follows:

S401. The terminal sends a prefix query request to the target system, wherein the prefix query request carries the target prefix.

S402. The object system sends a metadata synchronization request to the file system.

S403. The file system acquires all stored operation logs as pending operation logs.

S404, the file system sends the parent directory identifier, file identifier and metadata recorded in the operation log to be processed to the object system.

S405, the object system stores the received parent directory identifier, file identifier and metadata correspondingly.

S406, the target system finds an index satisfying the target prefix in the recorded indexes.

Among them, the index is composed of the parent directory ID of the file and the file ID.

S407. The target system returns the index satisfying the target prefix to the terminal in lexicographical order.

Based on the same technical concept, an embodiment of the present application also provides an apparatus for storing and synchronizing metadata. The apparatus is applied to a file system, and the apparatus includes:

The determination module 510 is used to determine the occurrence of a trigger event for metadata synchronization, which can specifically implement the determination function of the file system in the above step 201, as well as other implicit steps;

The obtaining module 520 is configured to obtain at least one pending operation log in the stored operation log, wherein the operation log records the file identifier corresponding to the file operated by the file system and the parent directory identifier corresponding to the file And the metadata of the file generated after the file system performs an operation on the file, the metadata includes the storage address information of the file in the file system, and specifically the file system in the above step 202 can be implemented , and other implicit steps;

The sending module 530 is configured to send the file identifier, parent directory identifier and metadata corresponding to each file corresponding to the at least one operation log to be processed to the target system, so that the target system can respond to the received files of each file The identifier, the parent directory identifier, and the metadata are stored correspondingly, and specifically, the sending function of the file system in the above step 203 and other implicit steps can be implemented.

In a possible implementation, the device also includes a deletion module for:

Delete the stored operation log to be processed.

In a possible implementation manner, the determining module 510 is configured to:

determine that the file system is restarted;

The obtaining module 520 is used for:

Get all operation logs stored as pending operation logs.

The determining module 510 is used for:

The obtaining module 520 is used for:

It is determined that the preset metadata synchronization period is reached;

The obtaining module 520 is used for:

Get all operation logs stored as pending operation logs.

The determining module 510 is used for:

The obtaining module 520 is used for:

In a possible implementation, the apparatus further includes:

The setting module is used to set all the values in the Bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to 0.

The third file is sent to the subject system.

It should be noted that: when the metadata synchronization device provided in the above embodiments synchronizes metadata, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions may be allocated to different functional modules as required. Complete, that is, dividing the internal structure of the file system into different functional modules to complete all or part of the functions described above. In addition, the device for synchronizing metadata provided in the above embodiments and the method embodiments for synchronizing metadata belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

Based on the same technical concept, an embodiment of the present application also provides a metadata synchronization device, the device is applied to an object system, and the device includes a receiving module 610 for:

Receive the file identifier, parent directory identifier, and metadata corresponding to the file sent by the file system, wherein the metadata includes the storage address information of the file in the file system. Specifically, the receiving of the object system in the above step 204 can be realized. functions, and other implied steps;

The storage module 620 is configured to store the received file identifier, parent directory identifier, and metadata correspondingly. Specifically, the storage function of the object system in the above step 204 and other implicit steps can be implemented.

obtain metadata of the target file;

receiving the target file returned by the file system;

Return the target file to the terminal.

It should be noted that: when the metadata synchronization device provided in the above embodiments synchronizes metadata, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions may be allocated to different functional modules as required. To complete, that is, to divide the internal structure of the object system into different functional modules to complete all or part of the functions described above. In addition, the device for synchronizing metadata provided in the above embodiments and the method embodiments for synchronizing metadata belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

7 is a schematic structural diagram of a file system provided by an embodiment of the present application. The file system 700 may vary greatly due to different configurations or performance, and may include one or more central processing units (CPU) 701 and one or more memories 702, wherein, at least one instruction is stored in the memory 702, and the at least one instruction is loaded and executed by the processor 701 to implement the above metadata synchronization method.

FIG. 8 is a schematic structural diagram of an object system provided by an embodiment of the present application. The object system 800 may vary greatly due to different configurations or performance, and may include one or more central processing units (CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 801 to implement the above metadata synchronization method.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof, and when implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions, and when the computer program instructions are loaded and executed on a device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that the device can access, or a data storage device such as a server, data center, or the like that includes an integration of one or more available media. The usable medium may be a magnetic medium (such as a floppy disk, a hard disk, and a magnetic tape, etc.), an optical medium (such as a digital video disk (Digital Video Disk, DVD), etc.), or a semiconductor medium (such as a solid-state disk, etc.).

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, etc.

The above is only an embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. Inside.

Claims

A method for metadata synchronization, wherein the method is applied to a file system, and the method includes:

determine the occurrence of a trigger event for metadata synchronization;

Obtain at least one operation log to be processed from the stored operation log, wherein the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and the file system in the file system. metadata of the file generated after performing an operation on the file, the metadata includes storage address information of the file in the file system;

Send the file identifier, parent directory identifier, and metadata corresponding to each file corresponding to the at least one operation log to be processed to the object system, so that the object system can compare the received file identifier, parent directory identifier, Metadata corresponds to storage.
method according to claim 1, is characterized in that, described in the operation log of storage after obtaining at least one to-be-processed operation log, described method also comprises:

Delete the stored operation log to be processed.
The method according to claim 2, wherein the determining that a trigger event for metadata synchronization occurs comprises:

determine that the file system is restarted;

The obtaining at least one pending operation log that satisfies the processing condition in the stored operation log includes:

Get all operation logs stored as pending operation logs.
The method according to claim 2, wherein the operation log further records the operation time of the file system on the file;

The determining that a trigger event for metadata synchronization occurs includes:

receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries a synchronization time;

The obtaining at least one pending operation log in the stored operation log includes:

In the stored operation logs, all operation logs whose recorded operation time is before the synchronization time are acquired as pending operation logs.
The method according to claim 2, wherein the determining that a trigger event for metadata synchronization occurs comprises:

It is determined that the preset metadata synchronization period is reached;

The obtaining at least one pending operation log in the stored operation log includes:

Get all operation logs stored as pending operation logs.
The method according to claim 2, wherein the method further comprises:

After the first file is operated, a corresponding first operation log is generated;

Obtain the parent directory identifier of the first file recorded in the first operation log;

Obtain the first bloom filter corresponding to the current storage period, record the parent directory identification of the first file in the first bloom filter;

storing the first file in the first operation log storage area corresponding to the first bloom filter;

The determining that a trigger event for metadata synchronization occurs includes:

receiving a metadata synchronization request for the second file sent by the object system, wherein the metadata synchronization request carries the parent directory identifier of the second file;

The obtaining at least one pending operation log in the stored operation log includes:

determining a second bloom filter that records the parent directory identifier of the second file;

Obtain the operation log stored in the second operation log storage area corresponding to the second bloom filter and the operation log stored before the operation log stored in the second operation log storage area in other storage areas, as pending processing Operation log.
The method according to claim 6, wherein the acquiring the operation log stored in the second operation log storage area corresponding to the second bloom filter and other storage areas are stored in the second operation log After the operation log stored before the operation log stored in the area is used as the operation log to be processed, the method further includes:

All values in the Bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed are set to 0.
The method according to claim 7, wherein the method further comprises:

receiving a read request for a third file sent by the target system, wherein the read request carries storage address information of the third file;

obtaining the third file according to the storage address information of the third file;

The third file is sent to the subject system.
A method for metadata synchronization, wherein the method is applied to an object system, and the method includes:

receiving the file identifier, parent directory identifier and metadata corresponding to the file sent by the file system, wherein the metadata includes storage address information of the file in the file system;

The received file identifier, parent directory identifier, and metadata of the file are stored correspondingly.
The method according to claim 9, wherein the method further comprises:

receiving the first read request of the target file sent by the terminal, where the first read request carries the file identifier and the parent directory identifier of the target file;

obtain metadata of the target file;

sending a second read request for the target file to the file system, where the second read request carries the storage address information of the target file;

receiving the target file returned by the file system;

Return the target file to the terminal.
An apparatus for storing and synchronizing metadata, wherein the apparatus is applied to a file system, and the apparatus includes:

A determination module, used to determine the occurrence of a trigger event for metadata synchronization;

The acquisition module is used to acquire at least one pending operation log in the stored operation log, wherein the operation log records the file identifier corresponding to the file operated by the file system, the parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file, the metadata includes storage address information of the file in the file system;

A sending module, configured to send the file identifier, parent directory identifier and metadata corresponding to each file corresponding to the at least one operation log to be processed to the object system, so that the object system can identify the file identifiers of the files received by the object system , parent directory ID, and metadata are stored accordingly.
device according to claim 11, is characterized in that, described device also comprises deletion module, is used for:

Delete the stored operation log to be processed.
The device according to claim 12, wherein the determining module is configured to:

determine that the file system is restarted;

The acquisition module is used for:

Get all operation logs stored as pending operation logs.
The device according to claim 12, wherein the operation log further records the operation time of the file system on the file;

The determining module is used for:

receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries a synchronization time;

The acquisition module is used for:

In the stored operation logs, all operation logs whose recorded operation time is before the synchronization time are acquired as pending operation logs.
The device according to claim 12, wherein the determining module is configured to:

It is determined that the preset metadata synchronization period is reached;

The acquisition module is used for:

Get all operation logs stored as pending operation logs.
The apparatus according to claim 12, wherein the apparatus further comprises a generating module for:

After the first file is operated, a corresponding first operation log is generated;

Obtain the parent directory identifier of the first file recorded in the first operation log;

Obtain the first bloom filter corresponding to the current storage period, and record the parent directory identifier of the first file in the first bloom filter;

storing the first file in the first operation log storage area corresponding to the first bloom filter;

The determining module is used for:

receiving a metadata synchronization request for the second file sent by the object system, wherein the metadata synchronization request carries the parent directory identifier of the second file;

The acquisition module is used for:

determining a second bloom filter that records the parent directory identifier of the second file;

Obtain the operation log stored in the second operation log storage area corresponding to the second bloom filter and the operation log stored before the operation log stored in the second operation log storage area in other storage areas, as pending processing Operation log.
The apparatus of claim 16, wherein the apparatus further comprises:

The setting module is configured to set all the values in the Bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to 0.
The device according to claim 17, wherein the device further comprises a receiving module for:

receiving a read request for a third file sent by the target system, wherein the read request carries storage address information of the third file;

obtaining the third file according to the storage address information of the third file;

The third file is sent to the subject system.
An apparatus for metadata synchronization, characterized in that the apparatus is applied to an object system, and the apparatus includes a receiving module for:

receiving the file identifier, parent directory identifier and metadata corresponding to the file sent by the file system, wherein the metadata includes storage address information of the file in the file system;

The storage module is used for correspondingly storing the received file identifier, parent directory identifier and metadata of the file.
The device according to claim 19, wherein the device further comprises a reading module for:

receiving a first fetch request for reading a target file sent by a terminal, where the first reading request carries a file identifier and a parent directory identifier of the target file;

obtain metadata of the target file;

sending a second read request for the target file to the file system, where the second read request carries the storage address information of the target file;

receiving the target file returned by the file system;

Return the target file to the terminal.
A file system, characterized in that, the file system includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement claims 1 to 8 The method for metadata synchronization according to any claim.
An object system, characterized in that, the object system includes a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement claims 9 to 10 Any of the described methods for metadata synchronization.
A computer-readable storage medium, comprising instructions, when the computer-readable storage medium runs on a file system, causing the file system to execute the file system described in any one of claims 1 to 8 Metadata synchronization method.
A computer-readable storage medium, comprising instructions, when the computer-readable storage medium runs on an object system, causing the file system to execute the file system described in any one of claims 9 to 10 Metadata synchronization method.