CN114116641A - Method and device for synchronizing metadata - Google Patents
Method and device for synchronizing metadata Download PDFInfo
- Publication number
- CN114116641A CN114116641A CN202010901025.4A CN202010901025A CN114116641A CN 114116641 A CN114116641 A CN 114116641A CN 202010901025 A CN202010901025 A CN 202010901025A CN 114116641 A CN114116641 A CN 114116641A
- Authority
- CN
- China
- Prior art keywords
- file
- metadata
- operation log
- stored
- identifier
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
Abstract
The embodiment of the application discloses a method and a device for metadata synchronization, and belongs to the technical field of data storage. The method comprises the following steps: determining a trigger event for synchronizing metadata, and acquiring at least one to-be-processed operation log from a stored operation log, wherein the operation log records a file identifier corresponding to a file operated by a file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file, and the metadata includes storage address information of the file in the file system. By the method and the device, storage resources of the object system can be saved.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method and an apparatus for metadata synchronization.
Background
Data storage systems typically have both file systems and object systems. The file system and the object system may be oriented to different users. And the file system and the object system need to provide the same file service for the user, namely, a certain file can be read through the file system or can be read through the object system.
At present, in order to make a certain file read through either a file system or an object system, a technician is usually required to migrate the file in the file system to the object system once every certain period of time. After each migration, the same file can be maintained in the file system and the object system.
In the course of implementing the present application, the inventors found that the related art has at least the following problems:
the contents of the files stored in the file system and the object system are the same, but the storage space of the file system and the storage space of the object system are occupied, which wastes storage resources.
Disclosure of Invention
In order to solve the problem of storage resource waste caused by the fact that an object system needs to store all files in the related art, the embodiment of the application provides a method and a device for metadata synchronization. The technical scheme is as follows:
in a first aspect, a method for metadata synchronization is provided, where the method is applied to a file system, and the method includes:
determining a trigger event for synchronizing metadata, and acquiring at least one to-be-processed operation log from a stored operation log, wherein the operation log records a file identifier corresponding to a file operated by a file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs operation on the file, the metadata includes storage address information of the file in the file system, and the file identifier, the parent directory identifier and the metadata corresponding to each file corresponding to the at least one to-be-processed operation log are sent to an object system, so that the object system correspondingly stores the received file identifier, the parent directory identifier and the metadata of each file.
In the solution shown in the embodiment of the present application, the file system may store an operation log, where the operation log records a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file. When a trigger event for synchronizing metadata occurs, the file system may send the file identifier, the parent directory identifier, and the metadata of the file recorded in the operation log to be processed to the object system, and the object system stores the file identifier, the parent directory identifier, and the metadata of the file correspondingly. The metadata comprises the storage address information of the file in the file system, so that the object system does not need to store the file, and when a user has a file acquisition requirement, the user only needs to acquire the file according to the storage address information of the stored file, thereby effectively saving the storage resource of the object system.
In one possible implementation manner, after obtaining at least one to-be-processed operation log from the stored operation logs, the method further includes:
and deleting the stored to-be-processed operation log.
In the scheme shown in the embodiment of the application, after the operation log to be processed is obtained, the operation log can be deleted from the disk, so that the storage space is saved, and meanwhile, one operation log is prevented from being repeatedly obtained.
In one possible implementation, the determining that the trigger event for metadata synchronization occurs includes:
determining that the file system restarts;
the acquiring, from the stored operation logs, at least one to-be-processed operation log that satisfies a processing condition includes:
and acquiring all the stored operation logs as to-be-processed operation logs.
In a possible implementation manner, the operation log further records an operation time of the file system on the file;
the determining that a trigger event for metadata synchronization occurs includes:
receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries synchronization time;
the obtaining at least one to-be-processed operation log from the stored operation logs includes:
and acquiring all operation logs of which the recorded operation time is before the synchronization time from the stored operation logs as to-be-processed operation logs.
In the scheme shown in the embodiment of the application, the metadata in the object system can be synchronized to the specified time according to the user requirement.
In one possible implementation, the determining that the trigger event for metadata synchronization occurs includes:
determining that a preset metadata synchronization period is reached;
the obtaining at least one to-be-processed operation log from the stored operation logs includes:
and acquiring all the stored operation logs as to-be-processed operation logs.
In the scheme shown in the embodiment of the application, the operation logs to be processed can be obtained according to the period, so that the operation logs are not accumulated too much, and simultaneously, the metadata in the file system and the metadata in the object system are not different too much.
In one possible implementation, the method further includes:
after a first file is operated, generating a corresponding first operation log, acquiring a parent directory identifier of the first file recorded in the first operation log, acquiring a first bloom filter corresponding to a current storage period, recording the parent directory identifier of the first file in the first bloom filter, and storing the first file to a first operation log storage area corresponding to the first bloom filter;
the determining that a trigger event for metadata synchronization occurs includes:
receiving a metadata synchronization request for a second file sent by an object system, wherein the metadata synchronization request carries a parent directory identifier of the second file
The obtaining at least one to-be-processed operation log from the stored operation logs includes:
and determining a second bloom filter recorded with a parent directory identifier of a second file, and acquiring an operation log stored in a second operation log storage area corresponding to the second bloom filter and an operation log stored in other storage areas before the operation log stored in the second operation log storage area as an operation log to be processed.
In the solution shown in the embodiment of the present application, the file system may record the parent directory identifier in the stored operation log by using a bloom filter. Therefore, when the object system has a synchronization requirement, all metadata in the stored operation logs can be not synchronized, and only the metadata in part of the operation logs can be synchronized, so that the bandwidth can be effectively saved.
In a possible implementation manner, after the obtaining of the operation log stored in the second operation log storage area corresponding to the second bloom filter and the operation log stored in another storage area before the operation log stored in the second operation log storage area is taken as the operation log to be processed, the method further includes:
and setting all the values of the arrays in the bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to be 0.
In the solution shown in the embodiment of the present application, after the operation log is obtained, all values of the array in the bloom filter corresponding to the corresponding operation log storage area may be set to 0, so that the bloom filter may be recycled in the following process.
In one possible implementation, the method further includes:
receiving a reading request for a third file sent by the object system, wherein the reading request carries storage address information of the third file;
and acquiring the third file according to the storage address information of the third file, and sending the third file to an object system.
In the scheme shown in the embodiment of the application, the object system may determine the storage address of the target file by querying the metadata of the target file, and further acquire the target file from the file system.
In a second aspect, a method for metadata synchronization is provided, where the method is applied to an object system, and the method includes:
receiving a file identifier, a parent directory identifier and metadata corresponding to a file sent by a file system, wherein the metadata comprises storage address information of the file in the file system;
and correspondingly storing the received file identification, the parent directory identification and the metadata of the file.
In one possible implementation, the method further includes:
receiving a first reading request of a target file sent by a terminal, wherein the first reading request carries a file identifier and a parent directory identifier of the target file;
acquiring metadata of the target file;
sending a second reading request for the target file to the file system, wherein the second reading request carries storage address information of the target file;
receiving the target file returned by the file system;
and returning the target file to the terminal.
In a third aspect, an apparatus for metadata storage synchronization is provided, where the apparatus is applied to a file system, and the apparatus includes:
a determining module for determining that a trigger event for metadata synchronization occurs;
an obtaining module, configured to obtain at least one to-be-processed operation log in a stored operation log, where a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file are recorded in the operation log, and the metadata includes storage address information of the file in the file system;
and the sending module is used for sending the file identifier, the father directory identifier and the metadata corresponding to each file corresponding to the at least one to-be-processed operation log to an object system so that the object system correspondingly stores the received file identifier, the father directory identifier and the metadata of each file.
In a possible implementation manner, the apparatus further includes a deletion module configured to:
and deleting the stored to-be-processed operation log.
In one possible implementation manner, the determining module is configured to:
determining that the file system reboots;
the obtaining module is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
In a possible implementation manner, the operation log further records an operation time of the file system on the file;
the determining module is configured to:
receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries synchronization time;
the obtaining module is configured to:
and acquiring all operation logs of which the recorded operation time is before the synchronization time from the stored operation logs as to-be-processed operation logs.
In one possible implementation manner, the determining module is configured to:
determining that a preset metadata synchronization period is reached;
the obtaining module is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
In one possible implementation manner, the apparatus further includes a generation module configured to:
after the first file is operated, generating a corresponding first operation log;
acquiring a parent directory identifier of the first file recorded in the first operation log;
acquiring a first bloom filter corresponding to a current storage period, and recording a parent directory identifier of the first file in the first bloom filter;
storing the first file to a first operation log storage area corresponding to the first bloom filter;
the determining module is configured to:
receiving a metadata synchronization request for a second file sent by the object system, wherein the metadata synchronization request carries a parent directory identifier of the second file;
the obtaining module is configured to:
determining a second bloom filter recorded with a parent directory identifier of the second file;
and acquiring the operation log stored in a second operation log storage area corresponding to the second bloom filter and the operation log stored in other storage areas before the operation log stored in the second operation log storage area as the operation log to be processed.
In one possible implementation, the apparatus further includes:
and the setting module is used for setting all values in the bloom filter corresponding to the storage area corresponding to the operation log to be processed to 0.
In a possible implementation manner, the apparatus further includes a receiving module, configured to:
receiving a reading request for a third file sent by the object system, wherein the reading request carries storage address information of the third file;
acquiring the third file according to the storage address information of the third file;
sending the third file to the object system.
In a fourth aspect, an apparatus for metadata synchronization is provided, where the apparatus is applied to an object system, and the apparatus includes a receiving module configured to:
receiving a file identifier, a parent directory identifier and metadata corresponding to a file sent by a file system, wherein the metadata comprises storage address information of the file in the file system;
and the storage module is used for correspondingly storing the received file identification, the father directory identification and the metadata of the file.
In a possible implementation manner, the apparatus further includes a reading module, configured to:
receiving a first reading request of a target file sent by a terminal, wherein the first reading request carries a file identifier and a parent directory identifier of the target file;
acquiring metadata of the target file;
sending a second reading request for the target file to the file system, wherein the second reading request carries storage address information of the target file;
receiving the target file returned by the file system;
and returning the target file to the terminal.
In a fifth aspect, there is provided a file system comprising a processor and a memory, the memory having stored therein at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method of metadata synchronization as described in the first aspect above.
In a sixth aspect, an object system comprises a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for metadata synchronization according to the second aspect.
In a seventh aspect, there is provided a computer-readable storage medium comprising instructions that, when run on a file system, cause the file system to perform the method for metadata synchronization of the first aspect.
In an eighth aspect, a computer-readable storage medium comprises instructions that, when executed on a subject system, cause the file system to perform the method for metadata synchronization of the second aspect.
The technical scheme provided by the embodiment of the application has the following beneficial effects:
in this embodiment, the file system may store an operation log, where the operation log records a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file. When a trigger event for synchronizing metadata occurs, the file system can send the file identifier, the parent directory identifier and the metadata of the file recorded in the operation log to be processed to the object system, and the object system correspondingly stores the file identifier, the parent directory identifier and the metadata of the file. The metadata comprises the storage address information of the file in the file system, so that the object system does not need to store the file, and when a user has a file acquisition requirement, the user only needs to acquire the file according to the storage address information of the stored file, thereby effectively saving the storage resource of the object system.
Drawings
Fig. 1 is a diagram of an implementation scenario provided by an embodiment of the present application;
FIG. 2 is a flow chart of a method for metadata synchronization according to an embodiment of the present application;
FIG. 3 is a flowchart of a method for reading a file according to an embodiment of the present application;
fig. 4 is a flowchart of a prefix query method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of an apparatus for metadata synchronization according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an apparatus for metadata synchronization according to an embodiment of the present application;
FIG. 7 is a schematic structural diagram of a file system according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an object system according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a metadata synchronization method, which can be realized by a file system and an object system together. Referring to the implementation scenario diagram shown in fig. 1, in the embodiment of the present application, when a file system performs an operation on a file, a corresponding operation log is generated, and the operation log is stored. That is, in the embodiment of the present application, the file system may store an operation log in addition to the file and the metadata. The metadata of the file after the operation is completed can be recorded in the operation log. In addition, through various trigger events, the file system is triggered to write the metadata in the operation log into the object system, so that the metadata in the object system and the file system are kept consistent.
The following describes a processing flow of the method for metadata synchronization provided in the embodiment of the present application with reference to fig. 1:
step 201, the file system determines that a trigger event of metadata synchronization occurs.
In implementation, a user can realize the operation of specifying a file in the file system through a terminal. After the file system executes the operation on the file, a corresponding operation log can be generated and stored. For example, the generated operation log may be stored in a disk. The operation log may record a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file. The metadata may record attribute information of the file, such as creation time, modification time, owner, read-write authority, storage address information, and the like. Wherein the storage address information indicates a storage location of the file in the file system.
Further, the operation log may also record an operation type and the like. The operation type may include create, delete, modify, etc.
In order to keep the metadata of the file stored in the object system consistent with the metadata of the file stored in the file system, a trigger event of metadata synchronization may be set, and an operation log is acquired to synchronize the metadata to the object system whenever the trigger event of metadata synchronization occurs. Specifically, the triggering event of metadata synchronization may be various. Several possible triggering events are listed below.
First, a file system reboot or failover is detected.
And secondly, reaching a preset metadata synchronization period.
And thirdly, receiving a metadata synchronization request sent by the terminal.
In this case, the user may send a metadata synchronization request directly to the file system through the terminal to trigger the file system to perform the metadata synchronization process.
And fourthly, receiving a metadata synchronization request sent by the object system.
In this case, when the user queries the specified file through the object system, the object system may first send a metadata synchronization request to the file system to trigger the file system to perform the metadata synchronization process.
Step 202, at least one to-be-processed operation log is obtained from the stored operation logs.
In implementation, every time a trigger event for metadata synchronization is sent, all the stored oplogs may be taken as pending oplogs.
In one possible implementation, the operation log may be recorded in conjunction with a bloom filter (BloomFilter) when being stored. Thus, when the log to be processed is acquired, all the stored operation logs may not be acquired. Accordingly, the specific processing may be as follows:
for ease of understanding, the following brief description of the method of use of the BloomFilter:
the BloomFilter is essentially a data structure consisting of a very long array and a plurality of hash functions. When an element needs to be recorded in the BloomFilter, each hash function in the BloomFilter can be used to calculate a hash value of the element. Then, it is sufficient to treat the corresponding position in the BloomFilter array by 1. When whether a certain element is required to be recorded in the BloomFilter is required to be inquired, each hash function in the BloomFilter is also required to be used firstly, a hash value of the element is calculated respectively, then, whether the position corresponding to the hash value in the array of the BloomFilter is 1 or not is inquired, if all the positions are 1, the BloomFilter is possible to record the element, and if not all the positions are 1, the BloomFilter is required to not record the element.
The file system may store a plurality of bloomfilters in the memory, and the bloomfilters are used to record parent directory identifiers recorded in the operation log. Here, the parent directory identifier of the file can be regarded as an element in the above-mentioned brief description of the method of using BloomFilter.
Each BloomFilter corresponds to an operation log storage area in the disk. Each BloomFilter corresponds to an operation log write cycle. Each oplog write period may be several tens of seconds, or several minutes, and the oplog write period may be set according to actual requirements.
It should be noted that the bloomfilters can be used circularly, that is, after the write period of the operation log corresponding to the last BloomFilter is ended, the first BloomFilter can be switched to continue to be used, so that the circulation is performed.
The following describes a process of writing an operation log into an operation log storage area, taking storing N bloomfilters as an example:
when the file system is started, the operation log writing period starts to be timed, the operation log generated in the first operation log writing period obtains the parent directory identifier recorded therein, and records the parent directory identifier in the first BloomFilter. And simultaneously, writing the operation log into an operation log storage area corresponding to the first BloomFilter. It should be further noted here that, in each operation log storage area, storage is required in the order in which it is written.
And acquiring a parent directory identifier recorded in the operation log generated in the Nth operation log writing period, and recording the parent directory identifier in the Nth BloomFilter. And simultaneously, writing the operation log into an operation log storage area corresponding to the Nth BloomFilter.
Then, in the operation log generated in the (N + 1) th operation log writing cycle, the parent directory identifier recorded therein is obtained, the hash value of the parent directory identifier is generated, and the hash value is recorded in the first BloomFilter. And simultaneously, writing the operation log into an operation log storage area corresponding to the first BloomFilter.
Corresponding to several trigger events in the above step 101, a specific implementation of obtaining the to-be-processed operation log from the stored operation logs according to each trigger event is described here:
corresponding to the first triggering event:
when the to-be-processed operation log is acquired, all the stored operation logs may be acquired as the to-be-processed operation log.
Corresponding to the second triggering event:
in this case, when acquiring the operation log to be processed, the following implementation may be made:
and firstly, acquiring all stored operation logs as to-be-processed operation logs.
And secondly, obtaining the operation logs written in the writing period of the first preset number of operation logs as the operation logs to be processed.
In the second implementation, a first storage threshold and a second storage threshold may also be set, where the first storage threshold is greater than the second storage threshold, and when the preset metadata synchronization period is reached, if it is detected that the operation log stored in the disk is greater than the first storage threshold, the preset number may be increased, for example, 1 is added to the preset number. When the preset metadata synchronization period is reached, if the operation log stored in the disk is detected to be smaller than the second storage threshold, the number of the metadata synchronization periods can be adjusted to the preset number.
Corresponding to the third triggering event described above:
the metadata synchronization request sent by the terminal may carry synchronization time, where the synchronization time indicates that the user needs to synchronize the metadata in the object system to the time. In addition, the operation log may record an operation time of the file, that is, a time when the file system performs an operation on the file. Correspondingly, when the file system acquires the operation log to be processed, all the operation logs with the recorded operation time before the synchronization time can be used as the operation logs to be processed.
Or, the operation time of the file is not recorded in the operation log, and the file system may record the write time of the operation log when storing the operation log, where the write time is the time when the file system writes the operation log to the disk, and the file system may use all the operation logs whose write time is before the synchronization time as the operation logs to be processed.
Corresponding to the fourth triggering event described above:
after receiving a read request of a user for a certain file, the object system may send a metadata synchronization request for the file to the file system, where the metadata synchronization request carries a parent directory identifier of the file. And the file system acquires the parent directory identifier carried in the metadata synchronization request, generates a hash value of the parent directory identifier, and determines which BloomFilter records the parent directory identifier according to the hash value. The specific determination method is described in the above brief description of the usage method of the BloomFilter, and is not described herein again.
And if only one BloomFilter is determined, taking the operation log in the operation log storage area corresponding to the BloomFilter and the operation logs written in other operation log storage areas before the operation log in the operation log storage area as the operation logs to be processed.
And if a plurality of BloomFilter are determined simultaneously, writing the corresponding operation logs into target operation logs in an operation log storage area corresponding to the BloomFilter with the most back period and operation logs in other operation log storage areas with the writing time before the target operation logs as operation logs to be processed.
Therefore, the number of the metadata synchronized at one time can be reduced, and the problem of overlarge bandwidth occupation caused by the simultaneous synchronization of excessive metadata is avoided as much as possible.
Step 203, the file system sends the file identifier, the parent directory identifier and the metadata corresponding to each file corresponding to at least one to-be-processed operation log to the object system.
In implementation, after the operation log to be processed is obtained, the file identifier, the parent directory identifier and the metadata recorded in the operation log can be extracted and sent to the object system for synchronous storage. Meanwhile, the file system can also delete the operation log to be processed in the disk, and set each value in the corresponding array in the BloomFilter to 0.
In addition, in order to save bandwidth, for the metadata in the operation log with the same recorded parent directory identifier and file identifier, only the metadata generated last may be taken and sent to the object system for synchronous storage.
And step 204, the object system correspondingly stores the file identification, the parent directory identification and the metadata of each received file.
In implementation, the object system may store the file identifier, the parent directory identifier, and the metadata of each received file correspondingly. Namely, the parent directory identifier + file identifier is used as an index, and the corresponding metadata is used as storage data.
Here, in the case where the file system sends all the metadata in the to-be-processed operation log to the object system, the object system may perform the overwriting storage, that is, for the metadata whose parent directory identifier and file identifier are the same, the metadata received later is used to overwrite the metadata received earlier.
In a possible implementation manner, the user may read the target file through the object system, and accordingly, the processing may be as follows:
s301, the terminal sends a first reading request for the target file to the object system, wherein the first reading request carries a parent directory identifier of the target file and a file identifier of the target file.
S302, the object system can send a metadata synchronization request for the target file to the file system, wherein the metadata synchronization request carries a parent directory identifier of the target file.
S303, the file system acquires the operation log to be processed according to the parent directory identifier of the target file.
Here, the file system may obtain the to-be-processed operation log according to the method of obtaining the to-be-processed operation log corresponding to the fourth trigger event in step 102.
S304, the file system sends the record in the operation log to be processed, namely the father directory identifier, the file identifier and the metadata to the object system.
S305, the object system correspondingly stores the received father directory identification, file identification and metadata, and acquires the metadata of the target file according to the father directory identification and the file identification of the target file.
S306, the object system sends a second reading request for the target file to the file system, wherein the second reading request carries the storage address information of the target file.
S307, the file system acquires the target file according to the storage address information of the target file.
S308, the file system sends the target file to the object system.
S309, the object system returns the target file to the terminal.
In another possible implementation manner, the user may query which files satisfy a certain prefix through the object system, and accordingly, the processing may be as follows:
s401, the terminal sends a prefix query request to the object system, wherein the prefix query request carries a target prefix.
S402, the object system sends a metadata synchronization request to the file system.
And S403, the file system acquires all the stored operation logs as to-be-processed operation logs.
S404, the file system sends the father directory identifier, the file identifier and the metadata recorded in the operation log to be processed to the object system.
S405, the object system correspondingly stores the received parent directory identification, file identification and metadata.
S406, the object system finds the index meeting the target prefix in the recorded indexes.
The index is composed of a parent directory identifier and a file identifier of the file.
S407, the object system returns the index meeting the target prefix to the terminal in a lexicographic order.
In this embodiment, the file system may store an operation log, where the operation log records a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file. When a trigger event for synchronizing metadata occurs, the file system can send the file identifier, the parent directory identifier and the metadata of the file recorded in the operation log to be processed to the object system, and the object system correspondingly stores the file identifier, the parent directory identifier and the metadata of the file. The metadata comprises the storage address information of the file in the file system, so that the object system does not need to store the file, and when a user has a file acquisition requirement, the user only needs to acquire the file according to the storage address information of the stored file, thereby effectively saving the storage resource of the object system.
Based on the same technical concept, an embodiment of the present application further provides an apparatus for synchronizing metadata storage, where the apparatus is applied to a file system, and the apparatus includes:
a determining module 510, configured to determine that a trigger event for metadata synchronization occurs, and specifically may implement the determining function of the file system in step 201, and other implicit steps;
an obtaining module 520, configured to obtain at least one to-be-processed operation log from stored operation logs, where a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file are recorded in the operation logs, and the metadata includes storage address information of the file in the file system, and specifically, the obtaining function of the file system in step 202 may be implemented, and other implicit steps may be implemented;
the sending module 530 is configured to send the file identifier, the parent directory identifier, and the metadata corresponding to each file corresponding to the at least one to-be-processed operation log to the object system, so that the object system stores the received file identifier, the parent directory identifier, and the metadata correspondingly, which may specifically implement the sending function of the file system in step 203 and other implicit steps.
In a possible implementation manner, the apparatus further includes a deletion module configured to:
and deleting the stored to-be-processed operation log.
In a possible implementation manner, the determining module 510 is configured to:
determining that the file system reboots;
the obtaining module 520 is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
In a possible implementation manner, the operation log further records an operation time of the file system on the file;
the determining module 510 is configured to:
receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries synchronization time;
the obtaining module 520 is configured to:
and acquiring all operation logs of which the recorded operation time is before the synchronization time from the stored operation logs as to-be-processed operation logs.
In a possible implementation manner, the determining module 510 is configured to:
determining that a preset metadata synchronization period is reached;
the obtaining module 520 is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
In one possible implementation manner, the apparatus further includes a generation module configured to:
after the first file is operated, generating a corresponding first operation log;
acquiring a parent directory identifier of the first file recorded in the first operation log;
acquiring a first bloom filter corresponding to a current storage period, and recording a parent directory identifier of the first file in the first bloom filter;
storing the first file to a first operation log storage area corresponding to the first bloom filter;
the determining module 510 is configured to:
receiving a metadata synchronization request for a second file sent by the object system, wherein the metadata synchronization request carries a parent directory identifier of the second file;
the obtaining module 520 is configured to:
determining a second bloom filter recorded with a parent directory identifier of the second file;
and acquiring the operation log stored in a second operation log storage area corresponding to the second bloom filter and the operation log stored in other storage areas before the operation log stored in the second operation log storage area as the operation log to be processed.
In one possible implementation, the apparatus further includes:
and the setting module is used for setting all values in the bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to 0.
In a possible implementation manner, the apparatus further includes a receiving module, configured to:
receiving a reading request for a third file sent by the object system, wherein the reading request carries storage address information of the third file;
acquiring the third file according to the storage address information of the third file;
sending the third file to the object system.
It should be noted that: in the metadata synchronization apparatus provided in the foregoing embodiment, when synchronizing metadata, only the division of the functional modules is illustrated, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the file system is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the metadata synchronization apparatus and the metadata synchronization method provided in the foregoing embodiments belong to the same concept, and specific implementation processes thereof are described in detail in the method embodiments and are not described herein again.
Based on the same technical concept, an embodiment of the present application further provides an apparatus for metadata synchronization, where the apparatus is applied to an object system, and the apparatus includes a receiving module 610 configured to:
receiving a file identifier, a parent directory identifier and metadata corresponding to a file sent by a file system, wherein the metadata includes storage address information of the file in the file system, and specifically, the receiving function of the object system in the step 204 can be realized, and other implicit steps can be realized;
the storage module 620 is configured to correspondingly store the file identifier, the parent directory identifier, and the metadata of the received file, and may specifically implement the storage function of the object system in step 204, and other implicit steps.
In a possible implementation manner, the apparatus further includes a reading module, configured to:
receiving a first reading request of a target file sent by a terminal, wherein the first reading request carries a file identifier and a parent directory identifier of the target file;
acquiring metadata of the target file;
sending a second reading request for the target file to the file system, wherein the second reading request carries storage address information of the target file;
receiving the target file returned by the file system;
and returning the target file to the terminal.
It should be noted that: in the metadata synchronization apparatus provided in the above embodiment, when synchronizing metadata, only the division of the above functional modules is exemplified, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the object system is divided into different functional modules to complete all or part of the above described functions. In addition, the metadata synchronization apparatus and the metadata synchronization method provided in the foregoing embodiments belong to the same concept, and specific implementation processes thereof are described in detail in the method embodiments and are not described herein again.
Fig. 7 is a schematic structural diagram of a file system 700 according to an embodiment of the present disclosure, where the file system 700 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 701 and one or more memories 702, where the memory 702 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 701 to implement the above-mentioned method for metadata synchronization.
Fig. 8 is a schematic structural diagram of an object system 800 according to an embodiment of the present disclosure, where the object system 800 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 801 and one or more memories 802, where the memory 802 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 801 to implement the above-mentioned method for metadata synchronization.
In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware or any combination thereof, and when the implementation is realized by software, all or part of the implementation may be realized in the form of a computer program product. The computer program product comprises one or more computer program instructions which, when loaded and executed on a device, cause a process or function according to an embodiment of the application to be performed, in whole or in part. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optics, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by the device or a data storage device, such as a server, a data center, etc., that is integrated into one or more available media. The usable medium may be a magnetic medium (such as a floppy Disk, a hard Disk, a magnetic tape, etc.), an optical medium (such as a Digital Video Disk (DVD), etc.), or a semiconductor medium (such as a solid state Disk, etc.).
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only one embodiment of the present application and should not be taken as limiting the present application, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (24)
1. A method for metadata synchronization, wherein the method is applied to a file system, and the method comprises:
determining that a trigger event for metadata synchronization occurs;
acquiring at least one to-be-processed operation log from a stored operation log, wherein a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file are recorded in the operation log, and the metadata includes storage address information of the file in the file system;
and sending the file identifier, the father directory identifier and the metadata corresponding to each file corresponding to the at least one to-be-processed operation log to an object system, so that the object system correspondingly stores the received file identifier, the father directory identifier and the metadata of each file.
2. The method of claim 1, wherein after obtaining at least one pending oplog from the stored oplogs, the method further comprises:
and deleting the stored to-be-processed operation log.
3. The method of claim 2, wherein determining that a trigger event for metadata synchronization has occurred comprises:
determining that the file system reboots;
the acquiring, from the stored operation logs, at least one to-be-processed operation log that satisfies a processing condition includes:
and acquiring all the stored operation logs as to-be-processed operation logs.
4. The method according to claim 2, wherein the operation log further records an operation time of the file system on the file;
the determining that a trigger event for metadata synchronization occurs includes:
receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries synchronization time;
the obtaining at least one to-be-processed operation log from the stored operation logs includes:
and acquiring all operation logs of which the recorded operation time is before the synchronization time from the stored operation logs as to-be-processed operation logs.
5. The method of claim 2, wherein determining that a trigger event for metadata synchronization has occurred comprises:
determining that a preset metadata synchronization period is reached;
the obtaining at least one to-be-processed operation log from the stored operation logs includes:
and acquiring all the stored operation logs as to-be-processed operation logs.
6. The method of claim 2, further comprising:
after the first file is operated, generating a corresponding first operation log;
acquiring a parent directory identifier of the first file recorded in the first operation log;
acquiring a first bloom filter corresponding to a current storage period, and recording a parent directory identifier of the first file in the first bloom filter;
storing the first file to a first operation log storage area corresponding to the first bloom filter;
the determining that a trigger event for metadata synchronization occurs includes:
receiving a metadata synchronization request for a second file sent by the object system, wherein the metadata synchronization request carries a parent directory identifier of the second file;
the obtaining at least one to-be-processed operation log from the stored operation logs includes:
determining a second bloom filter recorded with a parent directory identifier of the second file;
and acquiring the operation log stored in a second operation log storage area corresponding to the second bloom filter and the operation log stored in other storage areas before the operation log stored in the second operation log storage area as the operation log to be processed.
7. The method according to claim 6, wherein after the obtaining of the operation log stored in the second operation log storage area corresponding to the second bloom filter and the operation log stored in the other storage area before the operation log stored in the second operation log storage area as the operation logs to be processed, the method further comprises:
and setting all values in the bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to be 0.
8. The method of claim 7, further comprising:
receiving a reading request for a third file sent by the object system, wherein the reading request carries storage address information of the third file;
acquiring the third file according to the storage address information of the third file;
sending the third file to the object system.
9. A method for metadata synchronization, wherein the method is applied to an object system, and the method comprises:
receiving a file identifier, a parent directory identifier and metadata corresponding to a file sent by a file system, wherein the metadata comprises storage address information of the file in the file system;
and correspondingly storing the received file identification, the parent directory identification and the metadata of the file.
10. The method of claim 9, further comprising:
receiving a first reading request of a target file sent by a terminal, wherein the first reading request carries a file identifier and a parent directory identifier of the target file;
acquiring metadata of the target file;
sending a second reading request for the target file to the file system, wherein the second reading request carries storage address information of the target file;
receiving the target file returned by the file system;
and returning the target file to the terminal.
11. An apparatus for metadata storage synchronization, the apparatus being applied to a file system, the apparatus comprising:
a determining module for determining that a trigger event for metadata synchronization occurs;
an obtaining module, configured to obtain at least one to-be-processed operation log in a stored operation log, where a file identifier corresponding to a file operated by the file system, a parent directory identifier corresponding to the file, and metadata of the file generated after the file system performs an operation on the file are recorded in the operation log, and the metadata includes storage address information of the file in the file system;
and the sending module is used for sending the file identifier, the father directory identifier and the metadata corresponding to each file corresponding to the at least one to-be-processed operation log to an object system so that the object system correspondingly stores the received file identifier, the father directory identifier and the metadata of each file.
12. The apparatus of claim 11, further comprising a deletion module to:
and deleting the stored to-be-processed operation log.
13. The apparatus of claim 12, wherein the determining module is configured to:
determining that the file system reboots;
the obtaining module is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
14. The apparatus according to claim 12, wherein the operation log further records an operation time of the file system on the file;
the determining module is configured to:
receiving a metadata synchronization request sent by a terminal, wherein the metadata synchronization request carries synchronization time;
the obtaining module is configured to:
and acquiring all operation logs of which the recorded operation time is before the synchronization time from the stored operation logs as to-be-processed operation logs.
15. The apparatus of claim 12, wherein the determining module is configured to:
determining that a preset metadata synchronization period is reached;
the obtaining module is configured to:
and acquiring all the stored operation logs as to-be-processed operation logs.
16. The apparatus of claim 12, further comprising a generation module configured to:
after the first file is operated, generating a corresponding first operation log;
acquiring a parent directory identifier of the first file recorded in the first operation log;
acquiring a first bloom filter corresponding to a current storage period, and recording a parent directory identifier of the first file in the first bloom filter;
storing the first file to a first operation log storage area corresponding to the first bloom filter;
the determining module is configured to:
receiving a metadata synchronization request for a second file sent by the object system, wherein the metadata synchronization request carries a parent directory identifier of the second file;
the obtaining module is configured to:
determining a second bloom filter recorded with a parent directory identifier of the second file;
and acquiring the operation log stored in a second operation log storage area corresponding to the second bloom filter and the operation log stored in other storage areas before the operation log stored in the second operation log storage area as the operation log to be processed.
17. The apparatus of claim 16, further comprising:
and the setting module is used for setting all values in the bloom filter corresponding to the operation log storage area corresponding to the operation log to be processed to 0.
18. The apparatus of claim 17, further comprising a receiving module configured to:
receiving a reading request for a third file sent by the object system, wherein the reading request carries storage address information of the third file;
acquiring the third file according to the storage address information of the third file;
sending the third file to the object system.
19. An apparatus for metadata synchronization, the apparatus being applied to an object system, the apparatus comprising a receiving module configured to:
receiving a file identifier, a parent directory identifier and metadata corresponding to a file sent by a file system, wherein the metadata comprises storage address information of the file in the file system;
and the storage module is used for correspondingly storing the received file identification, the father directory identification and the metadata of the file.
20. The apparatus of claim 19, further comprising a reading module to:
receiving a first reading request of reading a target file sent by a terminal, wherein the first reading request carries a file identifier and a parent directory identifier of the target file;
acquiring metadata of the target file;
sending a second reading request for the target file to the file system, wherein the second reading request carries storage address information of the target file;
receiving the target file returned by the file system;
and returning the target file to the terminal.
21. A file system comprising a processor and a memory, the memory having stored therein at least one instruction that is loaded and executed by the processor to implement the method of metadata synchronization according to any of claims 1 to 8.
22. An object system comprising a processor and a memory, the memory having stored therein at least one instruction that is loaded and executed by the processor to implement the method of metadata synchronization according to any of claims 9 to 10.
23. A computer-readable storage medium comprising instructions that, when run on a file system, cause the file system to perform the method of metadata synchronization of any of claims 1 to 8.
24. A computer-readable storage medium comprising instructions that, when run on a subject system, cause the file system to perform the method for metadata synchronization of any of claims 9 to 10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010901025.4A CN114116641A (en) | 2020-08-31 | 2020-08-31 | Method and device for synchronizing metadata |
| PCT/CN2021/114190 WO2022042514A1 (en) | 2020-08-31 | 2021-08-24 | Metadata synchronization method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010901025.4A CN114116641A (en) | 2020-08-31 | 2020-08-31 | Method and device for synchronizing metadata |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114116641A true CN114116641A (en) | 2022-03-01 |
Family
ID=80352671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010901025.4A Pending CN114116641A (en) | 2020-08-31 | 2020-08-31 | Method and device for synchronizing metadata |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114116641A (en) |
| WO (1) | WO2022042514A1 (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153804B (en) * | 2017-11-17 | 2021-03-16 | 极道科技（北京）有限公司 | Metadata log updating method for symmetric distributed file system |
| US11347696B2 (en) * | 2019-02-19 | 2022-05-31 | Oracle International Corporation | System for transition from a hierarchical file system to an object store |
| CN110781152B (en) * | 2019-10-09 | 2022-04-22 | 济南浪潮数据技术有限公司 | Multi-protocol data sharing architecture |
| CN110807000B (en) * | 2019-10-25 | 2022-06-10 | 北京达佳互联信息技术有限公司 | File repair method and device, electronic equipment and storage medium |
-
2020
- 2020-08-31 CN CN202010901025.4A patent/CN114116641A/en active Pending
-
2021
- 2021-08-24 WO PCT/CN2021/114190 patent/WO2022042514A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022042514A1 (en) | 2022-03-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10564880B2 (en) | Data deduplication method and apparatus | |
| CN111176549B (en) | Data storage method and device based on cloud storage and storage medium | |
| CN105095300A (en) | Method and system for database backup | |
| CN115599747B (en) | Metadata synchronization method, system and equipment of distributed storage system | |
| CN109885577B (en) | Data processing method, device, terminal and storage medium | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN104156361A (en) | Method and system for achieving data synchronization | |
| US20220318095A1 (en) | Using a storage log to generate an incremental backup | |
| CN110597762A (en) | File processing method, device, equipment and storage medium | |
| CN110457305B (en) | Data deduplication method, device, equipment and medium | |
| CN114528255A (en) | Metadata management method, electronic device and computer program product | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| CN109753381B (en) | Continuous data protection method based on object storage | |
| CN113239012B (en) | Database migration method and device, electronic equipment and storage medium | |
| CN111309799A (en) | Method, device and system for realizing data merging and storage medium | |
| US9870385B2 (en) | Computer system, data management method, and computer | |
| CN114116641A (en) | Method and device for synchronizing metadata | |
| CN115840731A (en) | File processing method, computing device and computer storage medium | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| CN115421856A (en) | Data recovery method and device | |
| CN111078643B (en) | Method and device for deleting files in batch and electronic equipment | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN114089912A (en) | Data processing method and device based on message middleware and storage medium | |
| CN113625952A (en) | Object storage method, device, equipment and storage medium | |
| CN113190507B (en) | Index information synchronization method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |