WO2023178899A1 - Data management method and apparatus of file system, electronic device, and storage medium - Google Patents

Abstract

A data management method and apparatus for a file system, an electronic device, and a storage medium. A file system comprises file data, and the file data is divided into a plurality of data slices. The file system further comprises a source file item and a snapshot item. The source file item comprises a first bitmap and a second bitmap. The snapshot item comprises a third bitmap. The first bitmap is used for recording the writing states of a plurality of data slices. The second bitmap is used for recording the snapshot generation states of a plurality of data slices. The third bitmap is used for recording the migration states of a plurality of data slices. The method comprises: in response to receiving a data write request, determining, according to a first bitmap and a second bitmap, whether to perform data migration (S10); and in response to determining to perform data migration, executing a data migration operation and updating corresponding bits in the first bitmap, the second bitmap and a third bitmap (S20). In the described method, the data part of a snapshot is separated from an object storage device, reducing the dependence of a file system snapshot on the object storage device, thus the overall function of the snapshot is convenient to manage and maintain.

Description

Data management method and device, electronic equipment and storage medium of file system

This application claims priority from Chinese Patent Application No. 202210274271.0 submitted on March 21, 2022. The disclosure of the above Chinese patent application is hereby cited in its entirety as part of this application.

Technical field

Embodiments of the present disclosure relate to a data management method and data management device for a file system, electronic equipment, and storage media.

Background technique

In the storage field, with the increasing volume of data, the importance of data backup has gradually become more and more prominent. For a storage system, storing data and ensuring data security are its most basic functions and requirements. Snapshot is a data backup method. Because it can be completed instantly and achieve zero-window backup of data, snapshots have become a common and basic data backup technology among many data backup technologies. Snapshots have become the cornerstone of backup in the storage industry.

A snapshot refers to a fully available copy of a specified data collection, which includes an image of the corresponding data at a certain point in time (the point in time when the copy starts). The main function of snapshots is to enable data backup and recovery. When an application failure or file corruption occurs on the storage device, rapid data recovery can be performed to restore the data to a certain available point in time, thereby meeting the enterprise's requirements for business continuity and data reliability.

Contents of the invention

At least one embodiment of the present disclosure provides a data management method for a file system, wherein the file system includes file data divided into multiple data slices; the file system further includes source file items and Snapshot item, the source file item includes a first bitmap and a second bitmap, the snapshot item includes a third bitmap; multiple bits of the first bitmap correspond to the multiple data pieces one-to-one , the first bitmap is used to record the writing status of the multiple data slices; the multiple bits of the second bitmap correspond to the multiple data slices one-to-one, and the second bitmap uses In recording the snapshot generation status of the plurality of data pieces; a plurality of bits of the third bitmap correspond to the plurality of data pieces, and the third bitmap is used to record the plurality of data pieces. migration status; the method includes: in response to receiving a data write request, determining whether to perform data migration according to the first bitmap and the second bitmap; in response to determining to perform data migration, performing a data migration operation and updating Corresponding bits in the first bitmap, the second bitmap, and the third bitmap.

For example, in a method provided by an embodiment of the present disclosure, determining whether to perform data migration based on the first bitmap and the second bitmap includes: determining the data slice to be operated on based on the data write request; reading The bits in the first bitmap corresponding to the data piece are read, and the bits in the second bitmap corresponding to the data piece are read; in response to the bits in the first bitmap corresponding to the data piece. If the bit corresponding to the slice is a valid value and the bit corresponding to the data slice in the second bitmap is a valid value, it is determined to perform data migration.

For example, in a method provided by an embodiment of the present disclosure, determining whether to perform data migration based on the first bitmap and the second bitmap further includes: responding to the difference between the first bitmap and the data slice. If the corresponding bit is an invalid value or the bit corresponding to the data slice in the second bitmap is an invalid value, it is determined that data migration is not performed.

For example, in a method provided by an embodiment of the present disclosure, performing the data migration operation and updating corresponding bits in the first bitmap, the second bitmap, and the third bitmap includes: reading Get the data slice corresponding to the data migration operation, construct the read data slice into a snapshot type data object and store it; update the corresponding bits in the first image to a valid value, and convert the second The corresponding bits in the bitmap are updated to invalid values, and the corresponding bits in the third bitmap are updated to valid values.

For example, in a method provided by an embodiment of the present disclosure, reading the data slice corresponding to the data migration operation, constructing the read data slice into a data object of the snapshot type and storing it includes: obtaining the file path The latest snapshot version number; read the data slice corresponding to the data migration operation, and store the read data slice as a data object; add the latest snapshot version number and identification information indicating the snapshot type to the data In the information field of the object; associate the snapshot item corresponding to the latest snapshot version number with the data object.

For example, in a method provided by an embodiment of the present disclosure, the corresponding bits in the first bitmap are updated to valid values, the corresponding bits in the second bitmap are updated to invalid values, and the corresponding bits in the second bitmap are updated to invalid values. Updating the corresponding bits in the third bitmap to a valid value includes: updating the corresponding bits in the first bitmap of the source file item to a valid value, and updating the second bitmap of the source file item to a valid value. The corresponding bits in the bitmap are updated to invalid values; the latest snapshot version number of the file path is obtained; and the corresponding bits in the third bitmap of the snapshot item corresponding to the latest snapshot version number are updated to valid values.

For example, the method provided by an embodiment of the present disclosure further includes: in response to receiving the data write request, determining whether the data write request is an initial write request; in response to the data write request being the initial write request, The content of the first bitmap is synchronized to the second bitmap, and all bit positions of the third bitmap are set to invalid values.

For example, in the method provided by an embodiment of the present disclosure, the snapshot item further includes a fourth bitmap, and a plurality of bits of the fourth bitmap correspond to the plurality of data slices in a one-to-one manner. The bitmap is used to record the writing status of the corresponding source file item when the snapshot item is created; the method also includes: in response to the data write request being the initial write request, converting the content of the first bitmap to Sync to the fourth bitmap.

For example, a method provided by an embodiment of the present disclosure further includes: in response to receiving a snapshot read request, determining the data location according to the third bitmap of each version of the snapshot item and reading the data.

For example, in a method provided by an embodiment of the present disclosure, determining the data location and reading the data according to the third bitmap of the snapshot item of each version includes: determining the current snapshot item according to the snapshot read request, wherein the current snapshot item The snapshot item is a snapshot item of the version that needs to be read; the corresponding bit in the third bitmap of the current snapshot item is obtained; in response to the corresponding bit in the third bitmap of the current snapshot item being a valid value, Determine the data location according to the current snapshot item and read the data; in response to the corresponding bit in the third bitmap of the current snapshot item being an invalid value, obtain the third bitmap of the snapshot item with a larger version number in sequence. Corresponding bits, until the obtained bits are valid values, determine the data location and read the data according to the snapshot item of the version corresponding to the valid value; respond to the corresponding bits in the third bitmap of all versions of the snapshot items All are invalid values. Determine the data location and read the data based on the source file item.

For example, in a method provided by an embodiment of the present disclosure, the file system further includes a clone item, and the clone item is a copy of the snapshot item; the method further includes: in response to receiving a clone write request, based on The snapshot item associated with the clone item determines the data location, copies the required data to the storage location corresponding to the clone item, and performs a write operation.

For example, a method provided by an embodiment of the present disclosure further includes: in response to receiving a clone read request, determining the data location based on the snapshot item associated with the clone item and reading the required data.

For example, in the method provided by an embodiment of the present disclosure, after performing the data migration operation and updating the corresponding bits in the first bitmap, the second bitmap, and the third bitmap, the The method further includes: performing a write operation on the source file item.

For example, in a method provided by an embodiment of the present disclosure, after determining whether to perform data migration based on the first bitmap and the second bitmap, the method further includes: in response to determining not to perform data migration, The source file item performs a write operation.

At least one embodiment of the present disclosure also provides a data management device for a file system, wherein the file system includes file data, and the file data is divided into multiple data slices; the file system also includes source file items and a snapshot item, the source file item includes a first bitmap and a second bitmap, the snapshot item includes a third bitmap; the multiple bits of the first image are identical to the multiple data pieces. Correspondingly, the first bitmap is used to record the writing status of the multiple data slices; the multiple bits of the second bitmap correspond to the multiple data slices one-to-one, and the second bitmap used to record the snapshot generation status of the plurality of data pieces; a plurality of bits of the third bitmap correspond to the plurality of data pieces, and the third bitmap is used to record the plurality of data The migration status of the slice; the device includes: a judgment unit configured to respond to receiving a data write request and determine whether to perform data migration according to the first bitmap and the second bitmap; a processing unit configured to respond to It is determined to perform data migration, perform a data migration operation and update corresponding bits in the first bitmap, the second bitmap, and the third bitmap.

At least one embodiment of the present disclosure also provides an electronic device, including the data management apparatus for a file system provided by any embodiment of the present disclosure.

At least one embodiment of the present disclosure also provides an electronic device, including: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured To be executed by the processor, the one or more computer program modules include a method for implementing the data management method for a file system provided by any embodiment of the present disclosure.

At least one embodiment of the present disclosure also provides a storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the methods provided by any embodiment of the present disclosure can be implemented. File system data management methods.

Description of the drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure and do not limit the present disclosure. .

Figure 1 is a schematic diagram of the file structure of a file system;

Figure 2 is a schematic diagram of a bitmap in a file system provided by some embodiments of the present disclosure;

Figure 3 is a schematic flowchart of a data management method for a file system provided by some embodiments of the present disclosure;

Figure 4 is an exemplary flow chart of step S10 in Figure 3;

Figure 5 is an exemplary flow chart of step S20 in Figure 3;

Figure 6 is an exemplary flow chart of step S21 in Figure 5;

Figure 7 is an exemplary flow chart of step S22 in Figure 5;

Figure 8 is a schematic flowchart of another data management method for a file system provided by some embodiments of the present disclosure;

Figure 9 is a schematic flowchart of another data management method for a file system provided by some embodiments of the present disclosure;

Figure 10 is an exemplary flow chart of step S80 in Figure 9;

Figure 11A is one of the schematic diagrams of the data structure used in the data management method provided by some embodiments of the present disclosure;

Figure 11B is a second schematic diagram of the data structure used in the data management method provided by some embodiments of the present disclosure;

Figure 12A is a schematic diagram of a read operation in a data management method provided by some embodiments of the present disclosure;

Figure 12B is a schematic diagram of a write operation in a data management method provided by some embodiments of the present disclosure;

Figure 13 is a schematic block diagram of a data management device for a file system provided by some embodiments of the present disclosure;

Figure 14 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure;

Figure 15 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure;

Figure 16 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present disclosure.

Unless otherwise defined, technical terms or scientific terms used in this disclosure shall have the usual meaning understood by a person with ordinary skill in the art to which this disclosure belongs. "First", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Likewise, similar words such as "a", "an" or "the" do not indicate a quantitative limitation but rather indicate the presence of at least one. Words such as "include" or "include" mean that the elements or things appearing before the word include the elements or things listed after the word and their equivalents, without excluding other elements or things. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to express relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

Snapshots are usually used in file systems. Since file systems have complex metadata, data and other information, snapshot implementation is more complicated. For example, when generating snapshots, the cascading relationship between snapshots cannot be realized. That is, if a snapshot is taken of this directory, it will generally make it impossible to take snapshots of subdirectories. This limits the implementation of snapshots and leads to inconvenience in use. Moreover, the data part of the snapshot has a strong dependence on the Object Storage Device (OSD), which limits the optimization, reconstruction, and development of new features for OSD, and makes it inconvenient to maintain the snapshot function.

At least one embodiment of the present disclosure provides a data management method and data management device, electronic device and storage medium for a file system. This data management method separates the data part of the snapshot from the object storage device (OSD), reduces the dependence of the file system snapshot on the object storage device, facilitates management and maintenance of the overall snapshot function, and facilitates developers to make changes to the object storage device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numbers in different figures will be used to refer to the same elements that have been described.

At least one embodiment of the present disclosure provides a data management method for a file system. The file system includes file data, and the file data is divided into multiple data pieces. The file system also includes source file entries and snapshot entries. The source file item includes the first bitmap and the second bitmap, and the snapshot item includes the third bitmap. Multiple bits in the first bit map correspond to multiple data slices one-to-one, and the first bit image is used to record the writing status of multiple data slices. Multiple bits of the second bitmap correspond to multiple data slices one-to-one, and the second bitmap is used to record the snapshot generation status of the multiple data slices. Multiple bits of the third bitmap correspond to multiple data slices one-to-one, and the third bitmap is used to record the transition status of the multiple data slices. The method includes: in response to receiving a data write request, determining whether to perform data migration based on the first bitmap and the second bitmap; in response to determining to perform data migration, performing a data migration operation and updating the first bitmap and the second bitmap. , the corresponding bit in the third bitmap.

For example, file systems typically include data and metadata. Metadata is data used to describe data (data about data). It mainly describes information about data attributes and is used to support functions such as indicating storage location, historical data, resource search, file recording, etc. Metadata of a file system includes, for example, directory entries (dentries) and index nodes (inodes). Directory entries are used to describe the organizational form of files (or folders) in the file system, and index nodes are attribute information of files (or folders). In the file system, each file corresponds to an index node, which serves as the unique number of the file. The file system accesses the information of the index node through the index node number (inodeid).

Data (or data part) is the specific content of each file. Metadata can be used to obtain the attribute information of the data part, thereby determining the storage location (or storage address) of the data part, so that the data part can be read and written. . For example, in some examples, as shown in Figure 1, using the directory entries of the file system, a tree structure reflecting the organizational information of the file system can be obtained, and then using the files (or folders) dir1, dir2, dir3, file1 , the index node number of file2 obtains the information of the corresponding index node, so that the specific content of each file (or folder) can be read, that is, the corresponding data part. For example, in this example, dir1, dir2, and dir3 represent folders, and file1 and file2 represent files.

For example, in the embodiment of the present disclosure, the file system includes file data, and the file data is the aforementioned data part. For example, file data refers to data in the file system other than metadata. For example, file data is divided into multiple data pieces. The size of each data piece is, for example, 512K. Of course, the embodiments of the present disclosure are not limited to this, and the size of each data piece can also be 256K, 1M, 2M, or any other size, which can be determined according to actual needs, such as the characteristics of the file system.

In embodiments of the present disclosure, the file system also includes source file items and snapshot items. The source file item refers to the data object for the file or folder of the file system itself, and the snapshot item refers to the data object for the snapshot of the file system. For example, different snapshot items target different versions of snapshots.

In an embodiment of the present disclosure, for example, the source file item includes a first bitmap and a second bitmap, and the snapshot item includes a third bitmap. Here, the first bitmap, the second bitmap, and the third bitmap all refer to bitmaps, and "first", "second", and "third" are used to distinguish different bitmaps. Each bitmap includes a plurality of bits, and the size of the bitmap (bitmap size) is, for example, 4k, or can be any other applicable size. The embodiments of the present disclosure do not limit this. For example, in some examples, each bit of the bitmap corresponds to a data piece of size 512K, so each byte of the bitmap corresponds to a 4M data block, and the 4k bitmap (bitmap size = 4k) can correspond to 16G file data. Whenever the file written content exceeds 16G, the bitmap will dynamically apply for 4k size. For example, the size of the first bitmap, the second bitmap, and the third bitmap can be the same or different.

Multiple bits in the first bit map correspond to multiple data slices one-to-one, and the first bit image is used to record the writing status of multiple data slices. For example, each bit of the first bit map corresponds to a piece of data, and different bits correspond to different pieces of data. For each bit of the first bit map, when the bit is a valid value, it means that the corresponding data piece has been written, and when the bit is an invalid value, it means that the corresponding data piece has not been written. For example, you can use "1" to represent valid values and "0" to represent invalid values.

Multiple bits of the second bitmap correspond to multiple data slices one-to-one, and the second bitmap is used to record the snapshot generation status of the multiple data slices. For example, each bit of the second bitmap corresponds to a data piece, and different bits correspond to different data pieces. For each bit of the second bitmap, when the bit is a valid value, it means that the corresponding data piece has been snapshotted, and when the bit is an invalid value, it means that the corresponding data piece has not been snapshotted. For example, you can use "1" to represent valid values and "0" to represent invalid values.

Multiple bits of the third bitmap correspond to multiple data slices one-to-one, and the third bitmap is used to record the transition status of the multiple data slices. For example, each bit of the third bitmap corresponds to a data piece, and different bits correspond to different data pieces. For each bit of the third bitmap, when the bit is a valid value, it means that the corresponding data piece has been migrated. When the bit is an invalid value, it means that the corresponding data piece has not been migrated. For example, you can use "1" to represent valid values and "0" to represent invalid values. For example, data migration may refer to copy-on-write (COW). When a COW operation is performed, the corresponding data piece will be copied to a new storage location as the data part of the snapshot.

Figure 2 is a schematic diagram of a bitmap in a file system provided by some embodiments of the present disclosure. In some examples, as shown in Figure 2, the bitmap includes multiple bits. Each box in Figure 2 represents a bit in the bitmap, and each bit corresponds to a piece of data. For example, the bit may be 1 or 0, indicating different meanings. For example, the first bitmap, the second bitmap, and the third bitmap can all use similar bitmap structures, but the meanings of the bits in different bitmaps are different.

Figure 3 is a schematic flowchart of a data management method for a file system provided by some embodiments of the present disclosure. As shown in Figure 3, in some embodiments, the data management method includes the following operations.

Step S10: In response to receiving the data write request, determine whether to perform data migration based on the first bitmap and the second bitmap;

Step S20: In response to determining to perform data migration, perform a data migration operation and update corresponding bits in the first bitmap, the second bitmap, and the third bitmap.

For example, in step S10, the data write request may trigger a data write operation for the source file item, thereby modifying the file or folder of the file system itself. Since data writing operations will change the data content, if the corresponding files or folders have been snapshotted, data migration (such as COW) is required. Therefore, after receiving the data write request, it is necessary to determine whether to perform data migration based on the first bitmap and the second bitmap. For example, if the collection of snapshot items is a data warehouse, only repeated writing of the source file items will COW the data to the corresponding position of the last snapshot item, while the data of other previous versions of the snapshot items will not change.

FIG. 4 is an exemplary flow chart of step S10 in FIG. 3 . In some examples, as shown in Figure 4, the above step S10 includes the following operations.

Step S11: Determine the data piece to be operated according to the data write request;

Step S12: Read the bits corresponding to the data slice in the first bitmap, and read the bits corresponding to the data slice in the second bitmap;

Step S13: In response to the bit in the first bitmap corresponding to the data slice being a valid value and the bit in the second bitmap corresponding to the data slice being a valid value, it is determined to perform data migration.

Step S14: In response to the bit in the first bitmap corresponding to the data slice being an invalid value or the bit in the second bitmap corresponding to the data slice being an invalid value, determine not to perform data migration.

For example, in step S11, the data write request will carry information to indicate which data slice or data slices the write request is for the write operation. Therefore, the information carried by the data write request can be parsed to determine the data slice to be operated on.

For example, in step S12, since the data piece that needs to be operated has been determined, based on the correspondence between the data piece and the first bitmap and the second bitmap, the bits in the first bitmap corresponding to the data piece can be determined. , and determine the bits corresponding to the data piece in the second bitmap, thereby reading the bits corresponding to the data piece in the first bitmap and the second bitmap respectively.

For example, in step S13, if the bit corresponding to the data piece in the first bitmap is a valid value (for example, 1) and the bit corresponding to the data piece in the second bitmap is a valid value (for example, 1) ), it is determined that data migration is required. For example, if the bit corresponding to the data piece in the first bit of the map is a valid value, it means that the data piece has been written, so file data exists. If the bit corresponding to the data slice in the second bitmap is a valid value, it means that a snapshot has been generated for the data slice. Therefore, data migration (such as COW) is required before the write operation is performed, so that after the write operation is performed, the data slice can still be processed according to the Snapshot obtains the original file data.

For example, in step S14, if the bit corresponding to the data piece in the first bitmap is an invalid value (for example, 0) or the bit corresponding to the data piece in the second bitmap is an invalid value (for example, 0) ), it is determined not to perform data migration. For example, if the bit corresponding to the data slice in the first bit of the map is an invalid value, it means that the data slice has not been written, so of course there is no need to perform data migration. If the bit corresponding to the data slice in the second bitmap is an invalid value, it means that the data slice has not been snapshotted. Therefore, even if a write operation is to be performed on the data slice, there is no need to perform data migration.

Returning to FIG. 3 , for example, in step S20 , if it is determined to perform data migration, the data migration operation is performed and the corresponding bits in the first bitmap, the second bitmap, and the third bitmap are updated.

FIG. 5 is an exemplary flowchart of step S20 in FIG. 3 . In some examples, as shown in Figure 5, the above step S20 includes the following operations.

Step S21: Read the data slice corresponding to the data migration operation, construct the read data slice into a snapshot type data object and store it;

Step S22: Update the corresponding bits in the first bitmap to valid values, update the corresponding bits in the second bitmap to invalid values, and update the corresponding bits in the third bitmap to valid values.

For example, in step S21, the data migration operation can be similar to a normal COW, that is, the corresponding data piece is copied to a new storage location, and the snapshot of the corresponding version can be located in the new storage location, thereby facilitating Write the data piece in the original storage location. It should be noted that in the embodiment of the present disclosure, when storing a data piece to a new storage location, it needs to be constructed as a snapshot type data object. In the embodiment of the present disclosure, the data part of the snapshot is logically abstracted, so that it can be implemented uniformly according to the file system data interface. The data part in the file system can be divided into a data part corresponding to the source file item and a data part corresponding to the snapshot item. These are data objects, which are called file type data objects and snapshot type data objects respectively. For example, it can be distinguished by the object name. For example, for a snapshot type data object, a snap field can be added to the identifier (oid) of the object.

FIG. 6 is an exemplary flowchart of step S21 in FIG. 5 . In some examples, as shown in Figure 6, the above step S21 includes the following operations.

Step S211: Obtain the latest snapshot version number of the file path;

Step S212: Read the data slice corresponding to the data migration operation, and store the read data slice as a data object;

Step S213: Add the latest snapshot version number and identification information indicating the snapshot type to the information field of the data object;

Step S214: Associate the snapshot item corresponding to the latest snapshot version number with the data object.

For example, in step S211, the latest snapshot version number of the file path is first obtained. For example, the file path refers to the path to the data piece to be operated, and the latest snapshot version number of the file path records the latest snapshot version of the data piece to be operated. For example, in some examples, the latest snapshot version number may be 3, which means that the latest snapshot version of the data piece is the version 3 snapshot; in other examples, the latest snapshot version number may be 5, which means that the latest snapshot version number may be 5. The latest snapshot version of the data sheet is the version 5 snapshot. The above description of the latest snapshot version number is only illustrative, not restrictive, and does not constitute a limitation on the embodiments of the present disclosure.

For example, in step S212, the data slice corresponding to the data migration operation is read, and the read data slice is stored as a data object. For example, the read data piece can be constructed as a snapshot type data object. For detailed description of the snapshot type data object, please refer to the previous description and will not be repeated here.

For example, in step S213, the latest snapshot version number and identification information indicating the snapshot type are added to the information field of the data object. For example, information describing the data object can be recorded in the information field of the data object. Therefore, the latest snapshot version number can be recorded in the information field, and identification information (such as a snap field) indicating the snapshot type can also be recorded in the information field. Therefore, when reading a data object of the snapshot type, it can be learned that the data object is the data part of the snapshot item, and the snapshot version corresponding to the data object can be learned.

For example, in step S214, the snapshot item corresponding to the latest snapshot version number is associated with the data object. For example, an appropriate addressing method can be used to establish a relationship between the data object and the corresponding snapshot item, so that the storage location of the data object can be located according to the corresponding snapshot item to facilitate reading of the data object. For example, the snapshot item corresponding to the data object is the snapshot item corresponding to the latest snapshot version number. For example, in some examples, assuming that the latest snapshot version number is 5, the data object is associated with the snapshot item of version 5. Therefore, the storage location of the data object can be located based on the snapshot item of version 5, so that Read this data object.

Returning to Figure 5, for example, in step S22, the corresponding bits in the first bitmap are updated to valid values, the corresponding bits in the second bitmap are updated to invalid values, and the corresponding bits in the third bitmap are updated. Bits are updated to valid values.

FIG. 7 is an exemplary flowchart of step S22 in FIG. 5 . In some examples, as shown in Figure 7, the above step S22 includes the following operations.

Step S221: Update the corresponding bits in the first bitmap of the source file item to a valid value, and update the corresponding bits in the second bitmap of the source file item to an invalid value;

Step S222: Obtain the latest snapshot version number of the file path;

Step S223: Update the corresponding bit in the third bitmap of the snapshot item corresponding to the latest snapshot version number to a valid value.

For example, in step S221, the corresponding bit in the first bit of the source file item is updated to a valid value (for example, updated to 1), thereby indicating that the data piece corresponding to the bit is written. Since the data slice corresponding to this bit has been migrated before the write operation, and the data slice after the write operation has not been snapshotted, the corresponding bit in the second bitmap of the source file item is updated to Invalid value (for example, updated to 0), which means that the data piece corresponding to this bit has not been snapshotted.

For example, in step S222, the latest snapshot version number of the file path is obtained. Regarding the latest snapshot version number of the file path, you can refer to the previous content and will not be repeated here.

For example, in step S223, the corresponding bit in the third bitmap of the snapshot item corresponding to the latest snapshot version number is updated to a valid value, thereby indicating that the corresponding data piece has been processed in the snapshot item corresponding to the latest snapshot version number. data migration. For example, in some examples, assuming that the latest snapshot version number is 5, the bit corresponding to the data piece in the third bitmap of the snapshot item of version 5 is updated to a valid value, thereby indicating that the data piece is in the third bitmap. Data migration has been done in the snapshot item of version 5.

Therefore, according to the steps described in Figures 3 to 7, when a data write request is received, it can be determined whether to perform data migration based on the first bitmap and the second bitmap. If data migration is performed, the third bitmap is updated after the migration. The bits corresponding to the first bitmap, the second bitmap, and the third bitmap. In this way, the data part of the snapshot can be separated from the object storage device and can be managed and maintained independently.

Figure 8 is a schematic flowchart of another data management method for a file system provided by some embodiments of the present disclosure. In some embodiments, as shown in Figure 8, based on the aforementioned steps S10 and S20, the data management method may further include steps S30-S70. Regarding steps S10-S20, you may refer to the above content and will not be described again here.

Step S30: In response to receiving the data write request, determine whether the data write request is an initial write request;

Step S40: In response to the data write request being an initial write request, synchronize the content of the first bitmap to the second bitmap, and set all bit positions of the third bitmap to invalid values;

Step S50: In response to the data write request being an initial write request, synchronize the content of the first bitmap to the fourth bitmap;

Step S60: After performing the data migration operation and updating the corresponding bits in the first bitmap, the second bitmap, and the third bitmap, perform a write operation on the source file item;

Step S70: After determining whether to perform data migration based on the first bitmap and the second bitmap, in response to determining not to perform data migration, perform a write operation on the source file item.

For example, in step S30, after receiving the data write request, it is necessary to determine whether the data write request is an initial write request. For example, in the embodiment of the present disclosure, since the global snapshot version number and the snapshot version number of the file path are not updated synchronously, the global snapshot version number may be delayed from the snapshot version number of the file path. The second bitmap and The third bitmap may not exist before the initial write operation. Therefore, the second bitmap and the third bitmap need to be copied during the initial write operation. Therefore, it is necessary to determine whether the data write request is the first write request, which is, for example, a first cover write.

For example, whether the data write request is an initial write request can be determined by determining whether the global snapshot version number and the snapshot version number of the file path are the same. If the global snapshot version number is different from the snapshot version number of the file path, it means that the data write request is the first write request; if the global snapshot version number is the same as the snapshot version number of the file path, it means that the data write request is not the first write request. ask.

For example, in step S40, if the data write request is an initial write request, the content of the first bitmap is synchronized to the second bitmap, and all bits of the third bitmap are set to invalid values. For example, you can copy the contents of the first bitmap to serve as the second bitmap, that is, make the second bitmap identical to the first bitmap. For example, all bits of the third bitmap are set to invalid values, which means that no data migration is performed.

For example, in some examples, in addition to the third bitmap, the snapshot item also includes a fourth bitmap. Multiple bits of the fourth bitmap correspond to multiple data slices one-to-one. The fourth bitmap is used to record creation. This snapshot item is the writing status of the corresponding source file item. The third bitmap and the fourth bitmap can be used for copying and deleting operations when reading and writing snapshot items.

For example, in step S50, if the data write request is an initial write request, the content of the first bit image is synchronized to the fourth bit image. For example, you can copy the contents of the first bitmap as the fourth bitmap, that is, make the fourth bitmap the same as the first bitmap. For example, when COW is triggered, the fourth bitmap does not change, and only the writing status of the source file when the snapshot item is created is recorded.

For example, in step S60, after performing the data migration operation and updating the corresponding bits in the first bitmap, the second bitmap, and the third bitmap (that is, after executing step S20), a write operation is performed on the source file item . For example, the write operation is triggered by a data write request, and the write operation can modify the data part of the source file item. Since before performing the operation of step S60, it has been judged whether data migration is required and each bitmap is updated when the data migration operation is performed. Therefore, the data slice can be written at this time without affecting the data that has been executed. The data portion of the data migration operation.

For example, in step S70, after determining whether to perform data migration based on the first bitmap and the second bitmap (that is, after executing step S10), if it is determined that data migration is not to be performed, then writing is performed on the data part of the source file item. operate. Since data migration is not required, the data portion of the source file item can be directly written.

In this way, data writing operations can be implemented, and data migration can be performed as needed to maintain the data part of the snapshot and separate it from the object storage device.

Figure 9 is a schematic flowchart of another data management method for a file system provided by some embodiments of the present disclosure. As shown in Figure 9, in some embodiments, the data management method may also include the following operations.

Step S80: In response to receiving the snapshot read request, determine the data location according to the third bitmap of each version of the snapshot item and read the data;

Step S91: In response to receiving the clone write request, determine the data location based on the snapshot item associated with the clone item, copy the required data to the storage location corresponding to the clone item, and perform a write operation;

Step S92: In response to receiving the clone read request, determine the data location based on the snapshot item associated with the clone item and read the required data.

For example, in step S80, after receiving the snapshot read request, the data location is determined according to the third bitmap of the snapshot item of each version and the data is read.

FIG. 10 is an exemplary flowchart of step S80 in FIG. 9 . In some examples, as shown in Figure 10, the above step S80 includes the following operations.

Step S81: Determine the current snapshot item according to the snapshot read request, where the current snapshot item is the snapshot item of the version that needs to be read;

Step S82: Obtain the corresponding bits in the third bitmap of the current snapshot item;

Step S83: In response to the corresponding bit in the third bitmap of the current snapshot item being a valid value, determine the data location according to the current snapshot item and read the data;

Step S84: In response to the corresponding bit in the third bitmap of the current snapshot item being an invalid value, sequentially obtain the corresponding bits in the third bitmap of the snapshot item with a larger version number until the acquired bit is a valid value. , determine the data location and read the data based on the snapshot item of the version corresponding to the valid value;

Step S85: In response to the fact that the corresponding bits in the third bitmap of all versions of the snapshot items are invalid values, determine the data location according to the source file item and read the data.

For example, in step S81, when a snapshot read request is received, the current snapshot item needs to be determined. For example, a snapshot read request can trigger a data read operation on the snapshot item, thereby reading the data portion of the snapshot item. For example, the current snapshot item is the snapshot item of the version that needs to be read. For example, in some examples, it is assumed that there are five versions of snapshot items in the file system. When receiving a snapshot read request, it is necessary to determine which version of the snapshot item among the five versions the snapshot read request is to read. For example, a snapshot read request will carry version information to know the version number of the snapshot item to be read.

For example, in step S82, after the current snapshot item is determined, the corresponding bits in the third bitmap of the current snapshot item are obtained. For example, the third bitmap records the migration status of each corresponding data piece. By obtaining the corresponding bit in the third bitmap of the current snapshot item, you can learn whether data migration has been performed on the corresponding data piece. For example, if the corresponding bit in the third bitmap is a valid value, it means that the corresponding data piece has undergone data migration; if the corresponding bit in the third bitmap is an invalid value, it means that the corresponding data piece has not done data migration. migrate.

For example, in step S83, if the corresponding bit in the third bitmap of the current snapshot item is a valid value, the data location is determined based on the current snapshot item and the data is read. For example, the corresponding bit in the third bitmap is a valid value, indicating that the corresponding data piece has undergone data migration, so the data location can be located based on the current snapshot item and the data can be read.

For example, in step S84, if the corresponding bit in the third bitmap of the current snapshot item is an invalid value, then the corresponding bits in the third bitmap of the snapshot item with a larger version number are obtained sequentially until the obtained bit The bit is a valid value, and the data location is determined based on the snapshot item of the version corresponding to the valid value and the data is read. For example, the corresponding bit in the third bitmap of the current snapshot item is an invalid value, which means that the corresponding data piece has not been migrated. Therefore, the current snapshot item does not have a corresponding data part, and the data location cannot be located based on the current snapshot item. Therefore, it is necessary to obtain the corresponding bit in the third bitmap of the snapshot item with a larger version number. If the obtained bit is an invalid value, continue to obtain the corresponding bit in the third bitmap of the snapshot item with a larger version number. bit until the acquired bit is a valid value.

For example, in some examples, assuming that the current snapshot item is a snapshot item of version 5, and the corresponding bit in its third bitmap is an invalid value, then continue to obtain the snapshot item of version 6, if the corresponding bit in its third bitmap is If the corresponding bit is still an invalid value, continue to obtain the snapshot item of version 7, and so on. Assuming that the corresponding bit in the third bitmap of the 7th version snapshot item is a valid value, it means that the 7th version snapshot item has performed data migration for this data piece. The data location can be determined based on the 7th version snapshot item and Read data.

For example, in step S85, if the corresponding bits in the third bitmap of all versions of the snapshot items are invalid values, the data location is determined according to the source file item and the data is read. For example, the corresponding bits in the third bitmap of all versions of the snapshot items are all invalid values, which means that no data migration has been performed on the snapshot items of all versions, so the data location can only be located and the data can be read based on the source file item.

Therefore, through the above operations, the snapshot read request can be received and the read operation of the data part of the corresponding snapshot item can be completed. It should be noted that any read operation triggered by a read request will directly find the data location and will not trigger data migration.

For example, in some examples, the file system also includes clones, which are copies of the snapshot entries. For example, snapshot items cannot be written, but clone items can be written. The clone item is an editable version of the corresponding snapshot item.

Returning to Figure 9, in step S91, if a clone write request is received, the data location is determined based on the snapshot item associated with the clone item, the required data is copied to the storage location corresponding to the clone item, and the write operation is performed. For example, a clone write request can trigger a data write operation on the clone item, thereby making changes to the data portion of the clone item. For example, a clone item corresponds to a certain snapshot item, and when the clone item is generated, the content of the corresponding snapshot item is used as the content of the clone item. Therefore, it is necessary to determine the data location based on the snapshot item associated with the clone item, then copy the required data to the storage location corresponding to the clone item, and then perform the write operation.

For example, in some examples, it is assumed that clone item 1 exists in the file system, and clone item 1 is a clone version corresponding to the version 1 snapshot item, that is, clone item 1 is associated with the version 1 snapshot item. When a write operation needs to be performed on clone item 1, the data location needs to be determined based on the version 1 snapshot item. For example, step S80 may be used to determine the data location corresponding to the first version snapshot item, which will not be described again here. After that, copy the data to the storage location corresponding to clone item 1, and then perform the write operation for clone item 1.

For example, in step S92, if a clone read request is received, the data location is determined based on the snapshot item associated with the clone item and the required data is read. For example, a clone read request can trigger a data read operation on the clone item, thereby reading the data portion of the clone item. For example, in some examples, it is assumed that clone item 1 exists in the file system, and clone item 1 is a clone version corresponding to the version 1 snapshot item, that is, clone item 1 is associated with the version 1 snapshot item. When a read operation needs to be performed on clone item 1, the data location needs to be determined based on the snapshot item of version 1. For example, step S80 may be used to determine the data location corresponding to the first version snapshot item, which will not be described again here. After that, just read the data according to the determined data location.

Figure 11A is one of the schematic diagrams of the data structure used by the data management method provided by some embodiments of the present disclosure. The data structure is, for example, the structure of file data. For example, Fh is the file handle of any open file, uint32_t last_snap_id represents the global maximum snapshot version number, and uint32_t max_peer_id represents the maximum snapshot version number of the file path. uint32_t snap_id represents the id of the snapshot item. If it is a file type data object, uint32_t snap_id is all 0. uint32_t peer_id represents the id of the snapshot item on the file path. For snapshot type data objects, uint32_t snap_id is the id of the snapshot item, and uint32_t peer_id is 0. In this data structure, Bitmap base_alloc represents the first bitmap, and Bitmap base_peer represents the second bitmap. Bitmap last_snap_alloc represents the third bitmap of the latest version of the snapshot item, and Bitmap last_snap_peer represents the fourth bitmap of the latest version of the snapshot item. For example, Bitmap last_snap_alloc and Bitmap last_snap_peer in this data structure are redundant information that not only exists in this data structure, but is also stored in other locations in the file system.

Figure 11B is a second schematic diagram of the data structure used in the data management method provided by some embodiments of the present disclosure. The data structure is, for example, a metadata structure. For example, uint32_t last_cow_snap_id represents the id of the snapshot item for the last data migration, uint32_t snap_id represents the id of the snapshot item, and uint32_t peer_id represents the id of the snapshot item on the file path. No bitmap information (bitmap information) is recorded in the metadata (such as inode). Each operation is temporarily recorded in Fh, and then persisted to the disk. The object (such as oid) dropped to the disk is determined by the metadata id (such as inodeid). , offset (such as offset), and bitmap (such as bitmap).

By adopting the above data structure, data management can be facilitated. It should be noted that the data structure used in the embodiments of the present disclosure is not limited to the data structure shown in FIG. 11A and FIG. 11B , and any other applicable data structure may also be used, and the embodiments of the present disclosure are not limited to this. The data structures shown in Figure 11A and Figure 11B are only illustrative and not restrictive. The actual data structure used can also include more or less fields, parameters, functions, etc., which can be determined according to actual needs. .

Figure 12A is a schematic diagram of a read operation in a data management method provided by some embodiments of the present disclosure. The read operation for the snapshot item and the read operation for the clone item will be briefly described below with reference to FIG. 12A.

For example, in this example, assume that three data pieces are written in the source file item basefs, namely w1, w2, and w3. After writing w1, the first version of the snapshot item (snapid:1) is generated, and the bit corresponding to w1 in the fourth bitmap of the snapshot item is 1. After writing w2, a version 2 snapshot item (snapid:2) is generated, and the bits corresponding to w1 and w2 in the fourth bitmap of the snapshot item are 1 respectively. After writing w3, a version 3 snapshot item (snapid:3) is generated, and the bits corresponding to w1, w2, and w3 in the fourth bitmap of the snapshot item are 1 respectively. In the second bitmap of the source file item basefs, the bits corresponding to w1, w2, and w3 are 1, and since three snapshots were taken, the latest snapshot version number peerid of the file path is 3.

When you need to read w2 in the version 2 snapshot item, check the bit corresponding to last_snap_alloc in the third bitmap of the Fh record of the version 2 snapshot item. If the corresponding bit in last_snap_alloc is 1, it means that it is for version 2. The snapshot item has been COWed, so just read the data object of the second version of the snapshot item. If the bit corresponding to last_snap_alloc in the third bitmap recorded by Fh is 0, it means that no COW is performed for the second version of the snapshot item, so the corresponding bit in the third bitmap of the snapshot item larger than the current snapid is obtained, and That is, the corresponding bit in the third bitmap of the version 3 snapshot item is obtained. If it is 1, the data object of the version 3 snapshot item is read. The data object (osd) consists of metadata id (inodeid), offset (offset), global snapshot version number (snapid) and file path snapshot version number (peerid). Data writing of snapshot items is triggered by the write process. If the corresponding bits in the third bitmap of each snapshot item larger than the current snapid are all 0, read the data object of the source file item basefs.

It should be noted that when receiving a read request, it is necessary to first determine whether it is a snapshot read request. For example, you can determine whether the snap_id in the metadata (inode) is TFS_NOSNAP. Here, TFS_NOSNAP is the identification information indicating the read source file item. If it is TFS_NOSNAP, it is a read request for the source file item. This does not require additional operations. The process of reading the source file item in the normal file system can be used. If it is not TFS_NOSNAP, it is a read request for a snapshot item, and the value of snap_id is the version number of the snapshot item that needs to be read. This requires the process described above.

For example, as shown in Figure 12A, assume that two clone items are generated for the version 2 snapshot item (snapid:2), namely clone item 2-1 (attachid:1) and clone item 2-2 (attachid:2) . When it is necessary to read w2 in clone item 2-1, first determine the associated snapshot item based on clone item 2-1, that is, determine the second version snapshot item, and then use the same method as reading the second version snapshot item. The process reads w2, thereby completing the read operation for clone item 2-1.

Figure 12B is a schematic diagram of a write operation in a data management method provided by some embodiments of the present disclosure. The writing operation for the source file item and the writing operation for the clone item will be briefly described below with reference to FIG. 12B.

For example, in this example, it is still assumed that three data pieces are written in the source file item basefs, namely w1, w2, and w3. After writing w1, the first version of the snapshot item (snapid:1) is generated, and the bit corresponding to w1 in the fourth bitmap of the snapshot item is 1. After writing w2, a version 2 snapshot item (snapid:2) is generated, and the bits corresponding to w1 and w2 in the fourth bitmap of the snapshot item are 1 respectively. After writing w3, a version 3 snapshot item (snapid:3) is generated, and the bits corresponding to w1, w2, and w3 in the fourth bitmap of the snapshot item are 1 respectively.

When it is necessary to rewrite a new file w2-1 at the data slice location of w2, first obtain the latest snapshot version number max_peer_id on the current file path. If max_peer_id changes, it means that the write is the first cover write. Therefore, the snapshot's bitmap needs to be initialized. For example, when initializing, copy the first bitmap of the source file item as the second bitmap of the source file item and the fourth bitmap of the snapshot item, and set all bits of the third bitmap of the snapshot item to 0. , that is, base_peer=last_snap_peer=base_alloc, last_snap_alloc=0. Then, since max_peer_id>last_cow_snap_id, the metadata (inode) information (for example, represented by attr) is read, the cow information of the corresponding last_snap_id is updated, and last_cow_snap_id=max_peer_id. Therefore, before writing the new file w2-1, record the snapshot version to be migrated and related information.

Then, compare the peerid of Fh with the max_peer_id on the current file path. If the two are equal, the corresponding data piece in the source file item is written directly. If the corresponding bit in the first bit of the source file item recorded by Fh is 0, that is, base_alloc=0, or the corresponding bit in the first bit of the image and the second bitmap are not equal, that is, base_alloc! =base_peer, the corresponding data piece in the source file item is also written directly. If the corresponding bits in the first bitmap and the second bitmap are both valid values, that is, base_alloc is equal to base_peer and both are 1, then data migration (COW) is required. At this time, the data corresponding to the source file item is read, and the oid of the snapshot type data object is constructed, which is composed of inodeid, offset, snapid and peerid. And, construct the data object oid to be newly written (that is, the data object of w2-1). The difference between these two data objects is that the snapid of the data object of w2-1 is 0, while the snapid of the oid of the snapshot object is the version number of the largest snapshot id on the current file path.

Therefore, before writing the new file w2-1, the data of the original file w2 is migrated, and the snapshot item for data migration is the version 3 snapshot item. After that, you can continue to generate the fourth version of the snapshot item (snapid:4). The bits corresponding to w1, w2-1, and w3 in the fourth bitmap of the snapshot item are 1 respectively. Since 4 snapshots were taken, the latest snapshot version number peerid of the file path is 4.

It should be noted that when receiving a write request, it is necessary to first determine whether it is a data write request. The data write request refers to a write request for a source file item. For example, the judgment can be made based on the snap_id, for example, whether the snap_id is TFS_NOSNAP. Here, TFS_NOSNAP represents the identification information of the write source file item. If it is TFS_NOSNAP, it is a write operation of the source file item and a data write request. If it is not TFS_NOSNAP, it is a write operation of the snapshot item, and it will prompt that it cannot be written and return directly.

For example, as shown in Figure 12B, assume that two clone items are generated for the version 2 snapshot item (snapid:2), namely clone item 2-1 (attachid:1) and clone item 2-2 (attachid:2) . When it is necessary to write w2-1 for clone item 2-1, first determine the associated snapshot item based on clone item 2-1, that is, determine the second version snapshot item, and then use the same method as reading the second version snapshot item. The process reads w2, copies w2 to the storage location corresponding to clone item 2-1, and then writes w2-1 based on w2. Thus, the writing operation for clone item 2-1 is completed.

It should be noted that the processes of read operations and write operations shown in FIG. 12A and FIG. 12B are only exemplary and not restrictive. According to actual needs, other applicable processes can be used to implement read operations and write operations according to the information of each bitmap, and the embodiments of the present disclosure do not limit this.

In the embodiment of the present disclosure, the data part is managed by introducing a bitmap (bitmap), and the data part in the snapshot of the distributed network attached storage (Network Attached Storage, NAS) file system is removed from the object storage device (OSD). Independently, the data part logic of the snapshot is abstractly described, which can be easily implemented in the file system without using pointers to manage the data part. For example, the data part of the snapshot is implemented in a unified manner according to the file system data interface. The data part of the snapshot and the data part of the file system are distinguished by the object name, for example, adding a snap field to the object oid. Through the above method, it is easy to manage and maintain the overall function of snapshots, improve system performance, reduce the dependence and performance pressure of file system snapshots on object storage devices, and facilitate developers to further optimize, reconstruct, and develop new features of object storage devices. Of course, the data management method provided by the embodiments of the present disclosure is not limited to application in NAS file systems, and can also be applied in file systems of any architecture. The embodiments of the present disclosure do not limit this.

It should be noted that in the embodiments of the present disclosure, the above-mentioned data management method for a file system may also include more or fewer steps, and the execution order of each step is not limited and may be determined according to actual needs.

At least one embodiment of the present disclosure also provides a data management device for a file system. The data management device separates the data part of the snapshot from the object storage device, reduces the dependence of the file system snapshot on the object storage device, facilitates management and maintenance of the overall function of the snapshot, and facilitates developers to make changes to the object storage device.

Figure 13 is a schematic block diagram of a data management device for a file system provided by some embodiments of the present disclosure. As shown in FIG. 13 , in some embodiments, the data management device 100 includes a judgment unit 110 and a processing unit 120 .

For example, a file system includes file data, and the file data is divided into multiple data pieces. The file system also includes a source file item and a snapshot item. The source file item includes the first bitmap and the second bitmap, and the snapshot item includes the third bitmap. Multiple bits in the first bit map correspond to multiple data slices one-to-one, and the first bit image is used to record the writing status of multiple data slices. Multiple bits of the second bitmap correspond to multiple data slices one-to-one, and the second bitmap is used to record the snapshot generation status of the multiple data slices. Multiple bits of the third bitmap correspond to multiple data slices one-to-one, and the third bitmap is used to record the transition status of the multiple data slices.

The determination unit 110 is configured to determine whether to perform data migration according to the first bitmap and the second bitmap in response to receiving the data write request. For example, the judgment unit 110 may execute step S10 of the data management method as shown in FIG. 3 .

The processing unit 120 is configured to perform data migration in response to the determination, perform a data migration operation and update corresponding bits in the first bitmap, the second bitmap, and the third bitmap. For example, the processing unit 120 may execute step S20 of the data management method as shown in FIG. 3 .

For example, the judgment unit 110 and the processing unit 120 may be hardware, software, firmware, or any feasible combination thereof. For example, the judgment unit 110 and the processing unit 120 may be dedicated or general-purpose circuits, chips or devices, or may be a combination of a processor and a memory. Regarding the specific implementation forms of the judgment unit 110 and the processing unit 120, the embodiments of the present disclosure do not limit this.

It should be noted that in the embodiment of the present disclosure, each unit of the data management device 100 corresponds to each step of the aforementioned data management method. Regarding the specific functions of the data management device 100, please refer to the relevant description of the data management method above. No further details will be given here. The components and structures of the data management device 100 shown in Figure 13 are only exemplary and not restrictive. The data management device 100 may also include other components and structures as needed.

At least one embodiment of the present disclosure also provides an electronic device, which includes the data management apparatus for a file system provided by any embodiment of the present disclosure. This electronic device separates the data part of the snapshot from the object storage device, reduces the dependence of the file system snapshot on the object storage device, facilitates management and maintenance of the overall function of the snapshot, and facilitates developers to make changes to the object storage device.

Figure 14 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure. As shown in FIG. 14 , the electronic device 200 includes a data management device 210 , which may be the data management device 100 shown in FIG. 13 . For example, the electronic device 200 can be a distributed NAS system or any other system that needs to implement snapshots, and the embodiments of the present disclosure are not limited thereto. For relevant descriptions of the electronic device 200, please refer to the above description of the data management device 100, and will not be described again here.

At least one embodiment of the present disclosure also provides an electronic device. The electronic device includes a processor and a memory. One or more computer program modules are stored in the memory and configured to be executed by the processor. One or more computers The program module includes a method for implementing the data management method for a file system provided by any embodiment of the present disclosure. This electronic device separates the data part of the snapshot from the object storage device, reduces the dependence of the file system snapshot on the object storage device, facilitates management and maintenance of the overall function of the snapshot, and facilitates developers to make changes to the object storage device.

Figure 15 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure. As shown in FIG. 15 , the electronic device 300 includes a processor 310 and a memory 320 . Memory 320 is used to store non-transitory computer-readable instructions (eg, one or more computer program modules). The processor 310 is configured to execute non-transitory computer readable instructions. When the non-transitory computer readable instructions are executed by the processor 310, they may perform one or more steps in the data management method for a file system described above. Memory 320 and processor 310 may be interconnected by a bus system and/or other forms of connection mechanisms (not shown).

For example, the processor 310 may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or other form of processing unit with data processing capabilities and/or program execution capabilities, such as field programmable Gate array (FPGA), etc.; for example, the central processing unit (CPU) can be X86 or ARM architecture, etc. The processor 310 may be a general-purpose processor or a special-purpose processor that may control other components in the electronic device 300 to perform desired functions.

For example, memory 320 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, and the like. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 310 may run the one or more computer program modules to implement various functions of the electronic device 300 . Various application programs and various data, as well as various data used and/or generated by the application programs, etc. can also be stored in the computer-readable storage medium.

It should be noted that in the embodiment of the present disclosure, for the specific functions and technical effects of the electronic device 300, reference can be made to the above description of the data management method, which will not be described again here.

At least one embodiment of the present disclosure also provides a storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the method for file provided by any embodiment of the present disclosure can be implemented. Systematic approach to data management. Using this storage medium, the data part of the snapshot can be separated from the object storage device, reducing the dependence of the file system snapshot on the object storage device, making it easier to manage and maintain the overall function of the snapshot, and making it easier for developers to make changes to the object storage device.

Figure 16 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure. As shown in FIG. 16 , storage medium 400 is used to store non-transitory computer-readable instructions 410 . For example, the non-transitory computer readable instructions 410 when executed by a computer may perform one or more steps in the data management method for a file system described above.

For example, the storage medium 400 can be applied to the above-mentioned electronic device. For example, the storage medium 400 may be the memory 320 in the electronic device 300 shown in FIG. 15 . For example, for relevant description of the storage medium 400, reference may be made to the corresponding description of the memory 320 in the electronic device 300 shown in FIG. 15, which will not be described again here.

The following points need to be explained:

(1) The drawings of the embodiments of this disclosure only relate to the structures involved in the embodiments of this disclosure, and other structures can refer to common designs.

(2) Without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (18)

1. A data management method for a file system, wherein the file system includes file data, and the file data is divided into multiple data slices;

   The file system also includes a source file item and a snapshot item, the source file item includes a first bitmap and a second bitmap, and the snapshot item includes a third bitmap;

   The plurality of bits of the first bit map correspond to the plurality of data slices one-to-one, and the first bit map is used to record the writing status of the plurality of data slices;

   The plurality of bits of the second bitmap correspond to the plurality of data slices one-to-one, and the second bitmap is used to record the snapshot generation status of the plurality of data slices;

   The plurality of bits of the third bitmap correspond to the plurality of data slices one-to-one, and the third bitmap is used to record the migration status of the plurality of data slices;

   The methods include:

   In response to receiving the data write request, determine whether to perform data migration based on the first bitmap and the second bitmap;

   In response to determining to perform data migration, a data migration operation is performed and corresponding bits in the first bitmap, the second bitmap, and the third bitmap are updated.
2. The method according to claim 1, wherein determining whether to perform data migration based on the first bitmap and the second bitmap includes:

   Determine the data piece to be operated according to the data write request;

   Read the bits corresponding to the data slice in the first bitmap, and read the bits corresponding to the data slice in the second bitmap;

   In response to the bits corresponding to the data slices in the first bitmap being valid values and the bits corresponding to the data slices being valid values in the second bitmap, it is determined to perform data migration.
3. The method of claim 2, wherein determining whether to perform data migration based on the first bitmap and the second bitmap further includes:

   In response to the bit in the first bitmap corresponding to the data slice being an invalid value or the bit in the second bitmap corresponding to the data slice being an invalid value, it is determined not to perform data migration.
4. The method according to any one of claims 1-3, wherein the data migration operation is performed and corresponding bits in the first bitmap, the second bitmap, and the third bitmap are updated. bits, including:

   Read the data slice corresponding to the data migration operation, construct the read data slice into a snapshot type data object and store it;

   Update the corresponding bits in the first bitmap to a valid value, update the corresponding bits in the second bitmap to an invalid value, and update the corresponding bits in the third bitmap to a valid value. .
5. The method according to claim 4, wherein reading the data slice corresponding to the data migration operation, constructing the read data slice into the snapshot type data object and storing it includes:

   Get the latest snapshot version number of the file path;

   Read the data slice corresponding to the data migration operation, and store the read data slice as a data object;

   Add the latest snapshot version number and identification information indicating the snapshot type to the information field of the data object;

   Associate a snapshot item corresponding to the latest snapshot version number with the data object.
6. The method according to claim 4, wherein the corresponding bits in the first bitmap are updated to valid values, the corresponding bits in the second bitmap are updated to invalid values, and the third bitmaps are updated to invalid values. The corresponding bits in the bitmap are updated to valid values, including:

   Update the corresponding bits in the first bitmap of the source file item to a valid value, and update the corresponding bits in the second bitmap of the source file item to an invalid value;

   Get the latest snapshot version number of the file path;

   Update the corresponding bit in the third bitmap of the snapshot item corresponding to the latest snapshot version number to a valid value.
7. The method according to any one of claims 1-6, further comprising:

   In response to receiving the data write request, determine whether the data write request is an initial write request;

   In response to the data write request being the initial write request, the content of the first bitmap is synchronized to the second bitmap, and all bit positions of the third bitmap are set to invalid values.
8. The method according to claim 7, wherein the snapshot item further includes a fourth bitmap, a plurality of bits of the fourth bitmap correspond to the plurality of data pieces, and the fourth bitmap Used to record the writing status of the corresponding source file item when the snapshot item is created;

   The method also includes:

   In response to the data write request being the initial write request, the content of the first bitmap is synchronized to the fourth bitmap.
9. The method according to any one of claims 1-8, further comprising:

   In response to receiving the snapshot read request, the data location is determined based on the third bitmap of the snapshot item of each version and the data is read.
10. The method of claim 9, wherein determining the data location and reading the data according to the third bitmap of each version of the snapshot item includes:

    Determine the current snapshot item according to the snapshot read request, where the current snapshot item is a snapshot item of the version that needs to be read;

    Obtain the corresponding bit in the third bitmap of the current snapshot item;

    In response to the corresponding bit in the third bitmap of the current snapshot item being a valid value, determine the data location according to the current snapshot item and read the data;

    In response to the corresponding bit in the third bitmap of the current snapshot item being an invalid value, sequentially acquiring the corresponding bits in the third bitmap of the snapshot item with a larger version number until the acquired bit is a valid value, Determine the data location and read the data according to the snapshot item of the version corresponding to the valid value;

    In response to the corresponding bits in the third bitmap of all versions of the snapshot items being invalid values, the data location is determined according to the source file item and the data is read.
11. The method according to any one of claims 1-10, wherein the file system further includes a clone item, and the clone item is a copy of the snapshot item;

    The method also includes:

    In response to receiving the clone write request, determine the data location based on the snapshot item associated with the clone item, copy the required data to the storage location corresponding to the clone item, and perform a write operation.
12. The method of claim 11, further comprising:

    In response to receiving the clone read request, a data location is determined based on the snapshot item associated with the clone item and the required data is read.
13. The method according to any one of claims 1-12, wherein when performing the data migration operation and updating corresponding ones of the first bitmap, the second bitmap and the third bitmap After bit, the method also includes:

    Perform a write operation on the source file item.
14. The method according to any one of claims 1-13, wherein after determining whether to perform data migration according to the first bitmap and the second bitmap, the method further includes:

    In response to determining that data migration is not to be performed, a write operation is performed on the source file item.
15. A data management device for a file system, wherein the file system includes file data, and the file data is divided into a plurality of data slices;

    The file system also includes a source file item and a snapshot item, the source file item includes a first bitmap and a second bitmap, and the snapshot item includes a third bitmap;

    The plurality of bits of the first bit map correspond to the plurality of data slices one-to-one, and the first bit map is used to record the writing status of the plurality of data slices;

    The plurality of bits of the second bitmap correspond to the plurality of data slices one-to-one, and the second bitmap is used to record the snapshot generation status of the plurality of data slices;

    The plurality of bits of the third bitmap correspond to the plurality of data slices one-to-one, and the third bitmap is used to record the migration status of the plurality of data slices;

    The device includes:

    A judgment unit configured to, in response to receiving a data write request, judge whether to perform data migration according to the first bitmap and the second bitmap;

    A processing unit configured to perform data migration in response to determining to perform data migration, and update corresponding bits in the first bitmap, the second bitmap, and the third bitmap.
16. An electronic device includes the data management device for a file system according to claim 15.
17. An electronic device including:

    processor;

    Memory, including one or more computer program modules;

    Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, the one or more computer program modules include a computer program module for implementing any one of claims 1-14 The data management method for a file system.
18. A storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the data management for the file system described in any one of claims 1-14 can be implemented. method.

Images